Failure with Enterprise Systems: why we don’t learn, and what we can do about it

Tim Bray wrote a great article with the title: Doing It Wrong, in which he is sharing his insights on the perennial IT failures and how other’s manage to succeed despite ignoring the enterprise mantra. Tim made a lot of good points, the reactions to it where mostly great reading of their own. I thought my reaction warranted a blog post of my own, since otherwise it’d be a long comment lost in the jungle.

I won’t paraphrase Tim Bray here, his original posting is well worth reading: http://www.tbray.org/ongoing/When/201x/2010/01/02/Doing-It-Wrong

At this point, I must warn the reader that most of my parabolas are best understood when situated in the context of mathematical set theory, and loosely, in probabilistic terms: the chances of, and the lack thereof. Thinking in terms of sets, matrices, stochastic and probability, provide a great foundation for abstracting and considering IT complexity.

My take-away from the examples of successful web sites is that they have one great attribute: solving one problem at a time. The trick is that, as they solve one problem, they learn and discover new unanticipated solutions and they can leverage that immediately in the resolution of the next problem. The result is that these endeavours end up achieving success in lots of areas that they never imagined when they started, that is the second great attribute. There are certainly more great attributes, and no doubt plenty of shortcomings, but I live in “glass half full” land.

Examples?: Google started with search, Facebook was about college students, Twitter wanted people to catch up (in their breaks) on what their friends where doing. On a micro level, the examples are innumerable and would make this posting longer than necessary.

In my view, the attitude of, solving one problem at a time, is applied to all aspects of the endeavour, not just technical.

The combined effect of the solving one problem at a time attitude, and the two great attributes cited earlier, embody the following tacit admissions:

1. nobody is perfect,
2. an organisation, just like a person, will never be perfect,
3. it is wasteful to attempt to control everything from the get go, change occurs at a faster pace than our inventory and learning process
4. we know what we want, but we never know where the events might lead to.

This type of discovery and progressive enhancement is not witnessed in enterprise systems environments, simply because it’s being preempted by stubborn short-sightedness and often times by analysis-paralysis.

With enterprise systems we insist in having a complete oversight before building things, but without knowing it we keep missing important pieces all the way and all the time, from start to finish.

There’s a paradox here: in the frantic attempt to achieve complete and total oversight, we become short-sighted and fail to notice numerous key signals. The signals take an inordinate amount of forms and they are well grounded in the context: business context, organisational complexity (human nature, entrenched interests, etc), technical complexity (how we fail to design and code what we are really thinking/imagining, how we fail to grasp the technology for what it really is), our span of control, our span of attention, the target solution lifespan, the limits of collective wisdom, etc, the list is endless. In my interpretation, a huge array of overlooked problems stem from human organisation shortcomings, by far the most insidious and  difficult to master: things like feeling, fear and uncertainty about one’s position or career, attention deficit, unexposed incompetence and mediocrity in the ranks, etc.

If we learned anything about architecture, we should know that it’s impossible to make an exhaustive list of every single factor that could become a part of the problem or the solution. If we can’t identify all potential problems and potential solutions, how can we possibly build something that caters for everything (which is what Enterprise System is supposed to be, a system that addresses all)?

We might be able to make a very good list at the start, things change as we list them. Say we overcome that hurdle, we usually don’t have sufficient control over everything in that list to be successful. If we manage to get sufficiently extensive control, there is still the hurdle of exerting such immense control for the length of time that it takes to build everything that is necessary. Say that miracles exist and that we manage to cross that last hurdle also successfully, then unless every stakeholder (internal and external) could be robotically piloted to perfection some things will go amiss.

At this point you might think I’m exaggerating the challenges, I probably am for many cases for sure. But if I were to underplay the hurdles, that also would not be the honest true because any one of the cited area of challenge can take a life of its own unbeknownst to or beyond the control of IT management. When such devastating change occurs, we enter another phase: denial for the sake of our own survival (the lies that cover the mother of all lies). That is a key aspect I want to point out. Christopher Alexander‘s seminal books provide a good background on the complexity of getting designs right (design in an extremely wide sense of the term).

There’s one more paradox: we are all quick to admit that “nobody is perfect”, we also know that an organisation is also a “body” and is a reflection of the humans that comprise it, yet we still think that we can build a “perfect organisation” (be it a project, an IT, a business, a program, a solution, or whatever structure or impersonation we give to our organisations).

The short-sightedness with Enterprise Systems is the result of insufficiently applying learnings, which is unfortunate since we really have plenty of opportunity to learn. Take any industry or knowledge domain that we love to paraphrase in our enterprise system endeavors, they all learn and apply knowledge incrementally. We’re the only ones, the IT industry, who think that we can learn everything and apply it all in one go. Our mistakes start to accrue in the problem and solution scope inventory phase: a lot of unaccounted for biases are introduced as we seek to segregate the invariants from the variable elements of the enterprise systems. From there we just keep compounding blind spots, until someone eventually pulls the plugs or that the project collapse under the sheer weight of the endeavour.

If we can’t get control over everything that matters before building successful enterprise systems, should we give up trying and forego all central control?

I’m not suggesting that we forego control and planning, that would imply that I know the magical answer, which I don’t and won’t ever claim to. I too, like many in my profession, am in my quest for the nirvana and I’m under no illusion that it exist or is achievable. And that last point is a suggested take-away: keep dreaming and reaching for the nirvana, but be aware that you might never achieve it.

At the beginning of this post, I cited one behaviour and two attributes, as expressing my take-away from what makes some web sites successful. With this I will be first to admit my own limitations: I am interpreting things with my own background and influences, someone else could have other equally plausible biases. The reader is entitled to his/her own bias. The important message is that we should learn to read signals and adjust our actions accordingly, achieving total control is rarely feasible or even desirable.

The odd saying goes like this: ” a fool with a tool is still a fool”. I’d fancy a positive version of it, glass half full type, like this: “a smart with a tool is a super smart”. In my experience, aptitude should be matched with an equal measure of attitude, they’re both in limited supply and should be consumed with moderation. Whenever the combination lacks of balance, failure looms. There’s a lot of preaching the good message, there’s far fewer demonstrable examples of doing it successfully. The range of tools and methods are wildly varied and have evolved tremendously over time. What’s remained virtually constant is the involvement of human organisations. Success still remains by and large elusive. Anyone claiming to have all the answers is definitely deluding. Awareness is the starting point, then actively doing something with the learning is the only essential positive ingredient.

I’d appreciate any learning from you, if you care to share.