Tim Bray’s post on how enterprise systems are “Doing it Wrong” just triggered a realization for me. This whole agile vs. whatever debate has been a kind of red herring for a much larger argument about how businesses themselves ought to operate.
I’ve seen large companies adopt agile methodologies, and they don’t magically transform as a result. Something of the earlier culture persists, and the tendency to overplan, overbuild and generally waste tons and tons of money continues.
A single software project should never even hope to be a perfect solution.
The reason is that this whole thing isn’t about the methodologies. It’s not about the day-to-day operating procedures. It’s about the culture of control and hubris inherent in the corporate structure. Big companies, especially successful ones, think they know what’s right, and they think their success stands as proof of that skill. They underestimate the role of sheer luck in their past successes, and overestimate their ability to predict the future. So software teams in the enterprise make the mistake of believing that:
-
They know (or can know, with enough requirements gathering) exactly the nature of the problem their software should solve, and
-
The nature of that problem won’t change over time. Or, it won’t change fast enough to matter during the course of development to matter. Or, it will change but they’ll be capable of recognizing and adjusting accordingly.
The world is much more complex and unpredictable than this, and these beliefs have become the downfall of many projects. The economist-philosopher F.A. Hayek explained the difficulty of central planning (he’s writing about the economics of the entire economy, but I think this reasoning also applies within an individual organization, especially in enterprise software):
If we possess all the relevant information, if we can start out from a given system of preferences, and if we command complete knowledge of available means, the problem which remains is purely one of logic…
This, however, is emphatically not the economic problem which society faces… the “data” from which the economic calculus starts are never for the whole society “given” to a single mind which could work out the implications and can never be so given.
– from the article The Use of Knowledge in Society
It’s impossible for a single mind (or even a small group of minds) to encapsulate all the knowledge of complex enterprise operations. So a single software project should never even hope to be a perfect solution. Because of the fundamental information problem, even the requirements document could never be a perfect design for a single point in time. I believe this is the explanation for Galls Law.
Hayek goes on to explain that “emergent” order is superior (i.e. more efficient) to the order that central planners try to impose from above. We in western capitalist countries seem to know this in our bones in the post-communist era, but Hayek’s analysis is particularly conclusive.
Before we turn too sharply into classical economic thought, and before I start re-reading Adam Smith and Karl Marx, let me bring this back on topic.
When you view software development through this Hayekian lens, the benefits of agile methodologies really become apparent. By avoiding upfront design, the software is allowed to emerge from authentic, real-world interactions, the same way economic order emerges in capitalist economies. By shortening the feedback loop between conception, construction, testing and usage, you’re forced to start relying on empirical observations rather than predictions. By using dynamic languages and DSLs, you avoid the “information problem” (erg, more economics jargon) in translating human concepts into computing concepts.
So agile programming ideas make sense as potential solution in this whole enterprise software debacle. And Bray speaks of “bridging the gap” between the elephantoid enterprise software world and the grasshopper-like “web world”. He closes with this word of advice:
[Successful systems like Twitter] simply cannot be built if you start with large formal specifications and fixed-price contracts and change-control procedures and so on. So if your enterprise wants the sort of outcomes we’re seeing on the Web (and a lot more should), you’re going to have to adopt some of the cultures and technologies that got them built.
He’s right in the observation, but I think he’s under-prescribing on the solution. It’s going to take a lot more than adopting agile for in-house enterprise development at Citibank to start performing on par with Google’s programming teams, for example. It’s the corporate culture, not the development culture, that’s to blame for these problems. It’s the hubris of the organization that causes overspecification of documents, not the hubris of the software developer. In practical terms, it’s the CEO, not the CTO that’s got to lead the type of gap-bridging that Bray is speaking of.
Large companies (or “Enterprises”, I guess. I can’t seem to use that term without parentheses or quotes… it’s like the term itself is an ironic joke) won’t start making better use of IT until they realize that great software is a process not a project. And the process has to start outside the IT department if it has any chance of being successful.