Certified IT Architect, an attempt by The Open Group to raise the profile of this profession

The Open Group Certified Architect Program offers a career map for folks specialised in architecting Information Systems. The name “IT Architect” suggests that the focus is only on Information Technology, which I think is a bit of a narrow scoping. But this is certainly an opening for folks looking to map their career on an institutionalised structure.

The Open Group released a Certified Architect program, I learned about it on a recent blog post by Mike Walker. Since this is work from The Open Group, it’s going to spark reactions from friends and foes alike, depending on the side of the proverbial fence where people sit, it is either going to be dubbed “nice job!” or just “a completely skewed piece”. I don’t know, everyone’s entitled to his/her tastes.

By calling it IT Architect however, I am wondering if The Open Group intentionally limited the scope of what this entails. Professor Guido Dedene might have said that it should be more about Information Systems Architect, which is broader and more hollistic than just IT Architect. And I would agree with Pr. Dedene, because I think that mastering the technology alone isn’t sufficient to help solve business problems. If the mastery goes beyond Technology, then I would argue that IT is a reductive qualifer.

IT Certification may have gained some undue bad reputation because of some people misusing (abusing) the term. I have come across people walking around with all sorts of certifications yet you’d find that their knowledge and actual experience wouldn’t stand any proper scrutiny. I’ve even known people calling themselves architects when they didn’t have the faintest clue as to what the terms may signify. Those are certainly a few bad cases that shouldn’t be allowed to ruin the reputation of this profession. This is also not to say that certification isn’t useful, it clearly is a plus when walking to job interviews where the paper is what people value first and foremost.

Mike Walker’s blog gives a nice overview of The Open Group’s Certified Architect program, people starting up in the field would do well to get informed in this manner then chose their path diligently. Efforts of this kind are definitely a welcome contribution to the profession of Architect in Information Technology (IT) – I shall say Information Systems (IS) field.

Information Systems Architect should write solid code, Part 2, how?

Code is the most efficient way to convey ideas to programmers. Architecture code should be embryonic, a starting point for a solution. Architecture principles help when they define some ground rules for composing solutions out of components, the boundaries and contexts of these compositions and the special conditions that should be guarded. All this can be expressed much faster in wireframe code, than in complicated text. If you can express architecture principles in diagrams and text, be sure to also express them in solid code. I call this say it in code.

In an earlier post, I stated that an Architect should write solid code. I’ve also tweeted about this, pointing out that an IT architect who doesn’t code cannot be trusted. To illustrate my point, I chose a fictitious test case organisation, Trotters, where Kermit works as an Architect. I described some of the consequences when someone like Kermit is far removed from code. If you are interested in this discussion but haven’t read my earlier post, I invite you to do so here.

In this post I will discuss some of the ways that Kermit could write code to make the architecture practice more effective in his organisation.

To begin with, here are some more (bold) statements – I could always prove the points at a later stage.

  • If you can draw diagrams, you could also provide framing code base that realise the diagram
  • If you cannot create code representative of the diagrams that you are drawing, then either:
    • you are unsure how to do this (no further comment on that), or
    • your ideas are not fully fleshed out yet (so you are not done), or
    • you are reluctant to get into the details, for some reason
    • In any case, this is a problem.

Writing wireframe code goes much faster than writing long documents, and it harmonises interpretation much more efficiently. Text and diagrams leave a lot of room for interpretation, every programmer is likely to interpret things his/her own singular way.

Code is the most efficient way to convey ideas to programmers. Architecture code should be embryonic, a starting point for a solution. Architecture principles help when they define some ground rules for composing solutions out of components, the boundaries and contexts of these compositions and the special conditions that should be guarded. All this can be expressed much faster in wireframe code, than in complicated text.

How should Kermit go about this?

To simplify I will focus on web application as the solution domain for Trotters. Kermit and the team are working with object-oriented (OO) design and development concepts, they standardise on UML notation. They could be programming on Java or .NET platform, that doesn’t matter here.

Here is a simple guidance for Kermit:

  • UML supports drawing component diagrams. Kermit has probably designed a layered architecture diagram. Therefore, Kermit can create equivalent UML artefacts in code by designing classes for relevant components using the most appropriate realisation path.
  • Components in a layered architecture will typically interact via interfaces. Kermit can create (mostly one-to-one) interface classes between each connection points in the architecture. OO design skills are necessary to do this right. This is a first test of Kermit’s ability to say it in code. This can be further refined when for example using a concept such as design by contract, of which ample free resources can be found on the Internet
  • Boundary conditions and constraints: modern programming languages like C# or Java offer an array of features to deal with constraints. (similar concepts are fairly trivial in other languages) In addition, boundary conditions and constraints can be expressed in test harnesses, in turn included in the codebase as part of a continuous integration setup. Such productivity measures are invaluable and they often teach teams aspects of the solutions that might not be trivial, let alone described in text documents.
  • Enforcing architecture: this aspect is made trivial when considering the development environments (IDEs) available to programmers these days. These IDEs ship (or can be complemented) with code analysis and validation tools. The most recurrent patterns and errors can typically be verified with little or no effort, thanks to the richness of the validation rules that are available for these tools. An organisation like Trotters, as discussed in part 1, is typically weak in this area, and that is a shame.

Once expressed in code, the following become enabled:

  • Efficient two-way feedback on the architecture: programmers can immediately spot any issues and raise them for resolution with the architect. Kermit’s ability to communicate in code will come in handy, as he is able to grasp implementation issues quickly and can improve his design in the process
  • Enforcing architecture principles: nearly every modern programming environment offer tools for validating code and testing boundary conditions.
  • Platform constraints are immediately brought to light: with architecture code, architects and programmers are immediately confronted with infrastructure constraints. This makes it possible to analyse the situation, convey feedback quickly to all stakeholders before anything is really built. This helps in reducing gaps in expectations across all groups.
  • Guarding against basic mistakes or repetitions: there are elements that are typically common to every solution in a given business domain. There is no value in painstakingly recreating such elements, as that would only expose Trotters to unnecessary mistakes (typing, errors or omissions due to human factors, learning the hard way).

An easy argument could be to claim that Kermit, as an architect, does not have enough time to delve into programming details. Another easy argument would be to claim that should Kermit delve in code, he would be overstepping his role (micro-managing?), or that the programmers’ creativity could be hampered. Such arguments are easy to bring up, it is very tempting when one knows nothing better to do. But such approach is misguided, in most cases. First of all, I argue that an architect can write solid code much faster than he/she can create useful documents with legible text and diagrams. A second reason why I think this is misguided, is that the tools and techniques available today are amazingly powerful and simple to use, not using them to their true potential equates to disservice to the organisation. As a way to illustrate this last point, I’ll take a totally unrelated example: as I look over the shoulder of some people using Microsoft Word, I’m often amazed to see how they go through the pain of manually formatting documents by inserting spaces and page breaks. These people don’t seem to be aware of features that were in the product since version 2.0! (the first I’ve used). That is the point. And this example is actually nothing compared to the inefficiencies caused by programmers and architects enforcing early 90’s work processes with 2010 generation of tools and techniques.

To summarise, if you can express architecture principles in diagrams and text, be sure to also express them in solid code to reduce communication gaps. I call this: say it in code. Users typically don’t relate to design documents (architecture is part of design in this context). Often case, design documents would pass approval gates without users noticing faults or shortcomings in them. And this should be expected, precisely because users hire architects for their solution skills and experience. Architects shouldn’t expect that users could validate solution architecture, which is what normally happens a lot – again I’m using the term users in a broad context here. On the other hand, once solutions are shown to users, they can more effectively provide meaningful feedback. So, Architects should express technical solutions in solution, and code is the closest form that can be less immune to human interpretation induce deviations.

So far, I’ve focused on the application architecture aspect of the Architect role, in my simplified view. Looking at the broader architect role, including business analysis and infrastructure design, saying it in code will involve other aspects that I won’t analyse too much here for a number of reasons. This posting is already quite long, I’m still trying to keep it short enough yet complete in its illustration. In future postings, I will tackle these other architect roles.

In the next instalment I will explore the limits of this concept. If I get more time, I will develop the concept further for Trotters by showing specific examples of the type of code that Kermit would be writing to help his organisation (eat your own dog food idea).

Information Systems Architects should write solid code. Part 1, Why?

I believe that an architect, in the Information Systems world, should be first and foremost a very good programmer. The architect should write code on occasions, it would be better if writing code is actually a habit. When this is the case, the Architect is able to bring a positive influence to the organisation. When this is not the case, the Architect would most probably be less efficient and will find himself/herself regularly in unproductive discussions.

I believe that, in Information Systems environments, an architect should be first and foremost a very good programmer. The architect should write code on occasions, it would be better if writing code is actually a habit. When this is the case, the Architect is able to bring a positive influence to the organisation. When this is not the case, the Architect would most probably be less efficient and will find himself/herself regularly stuck in unproductive discussions. Unfortunately, a non-negligible amount of inefficiencies stem from the Architect’s lack of awareness of the application platforms and code and the engineering activities surrounding them.

This can be proven ab absurdo, but I will provide a simple case to illustrate the point.

In this and subsequent postings, I plan a multi-part posting, I will be using the term Architect to refer to anyone who has responsibility in capturing user requirements and translating them into instructions for programmers. This broad definition includes business analysts and application designers. Though infrastructure architects and designers play a key role in shaping up the IT environment, I will classify them as programmers in this discussion because they are often at the receiving end of something specified by someone else.

Those who have needs and ask for something to be done with the IS/IT systems are users.
Those who translate user needs into work instructions are architects.
Those who actually execute the tasks of creating or modifying IS/IT systems are programmers.
The expression write solid code designate the act of creating an artefact that can be unambiguously understood as machine code, when used in a modelling context these are formal model artefacts, when used in a programming context these are program source code.

I know there might be a million other definitions out there. I think, at this point, someone could take offence and quit reading. That would be too bad. I am not interested in definitions and titles, I want to make a point and I need these specific working definitions. They work fine in this context, take my word for it.

Let’s give a simple persona to our IS/IT Architect,  Kermit. Kermit is a man in his mid-thirties, he works for a company called Trotters Fresh Dairy Products  Ltd (known as Trotters).

Say that Kermit does not write code, whatever the reason. This means that Kermit is not familiar with coding issues because he doesn’t deal with them regularly (so Kermit actually knows a programming language? good for him, but he is not involved in coding at Trotters and that that’s the point here). Kermit only talks in concepts, not related to code.

Kermit communicates his concepts to the programmers Robert and Marie-Louise using diagrams and explanatory notes. Kermit also communicates with business stakeholders using diagrams and explanatory notes.

As software is written, there will inevitably be mismatches between the application release(s) and user expectations – malfunctions are particular kinds of mismatches, changes in user expectation are also forms of mismatches. Resolving any such issues involves representatives of users, programmers and architects. At the very least, Trotters IS/IT will be facing a number of challenges due to the combined effect of the following factors:

  • Kermit made his architecture decisions based on his extensive knowledge of widely used architecture frameworks and standards, papers from industry analysts and discussions in architecture communities, his own perception of what the business is about and what matters most to Trotters business and organisation
  • Kermit’s architecture was interpreted by the business according to their knowledge, their perception of IT and software (often shaped by external influences and people background), and their evolving needs
  • Kermit’s architecture was interpreted by the programmers according to their knowledge, the platform and programming tools they are using and the resources that were available to do the work.
  • Coding was constrained by a number of factors including technology platform, programming tools, resource availability, Robert and Marie-Louise are not aware of everything that might be possible with the tools in their possession. Some of Kermit’s suggested principles were not feasible due to these constraints, choices had to be made on-the-fly.
  • Changes constantly occur in Trotters’ business environment, with them people adjust their perceptions and their expectations, it is very hard to convey all those changes in a usable and timely manner to all stakeholders. So people make assumptions that they feel comfortable with and won’t necessarily validate such assumptions.
  • Business is always pressed for time, competition is fierce in Trotters’ industry, financial constraints are ever present, decision makers are dealing with compromises all the time

Arguably there is more to say about the factors that influence complexity in the organisation, the above seem sufficient to make the point here. Let’s analyse the impact of Kermit’s lack of awareness of the code:

  • Industry architecture best practices, analyst white papers, none of this make sense until they are cast into the reality of a specific organisation. This cannot be done efficiently without intimate knowledge of the platform and the way the application is being (or supposed to be) coded. With Kermit, Trotters organisation will be learning this reality casting process the hard way
  • Even if perfectly drafted, the architecture can quickly become stale because the business reality would have moved after Kermit issued his blueprints but before the applications could be fully built and released.
  • Kermit couldn’t coach Robert and Marie-Louise efficiently, because he wasn’t involved in the coding. Even if Kermit would work closely with the programmers, since Kermit can’t write solid code, then a significant amount of the programmers’ time would go into educating Kermit on coding issues and practices
  • The numerous micro-decisions made during the software construction contribute to a non-negligible change in the initial promises of the architecture, such changes will not surface until an application release is available to the users.
  • Kermit misses the chance of bringing any lessons learned back into his architecture vision, because the feedback loop is not present or is poorly executed. Conversely, Kermit has little or no way to ensure, thus guarantee, that his vision is actually been executed upon. If the vision wasn’t correct, that fact is not properly reflected everywhere. If the vision was correct but was poorly executed, there was nothing to prevent that from happening
  • Vendor platforms evolve, so any knowledge built on a particular set of vendor solutions is likely to become obsolete and require refreshing before it is put to use after a certain amount of time. This constant knowledge renewal typically doesn’t happen, sometimes because it cannot be afforded, but also partly because people tend to get stuck in their routines and fail to notice that their knowledge is becoming obsolete. There is nothing in place to validate that knowledge at Trotters’ is adequate and aligned between the architecture practice and the programming practice
  • Kermit probably know some technology platforms very well, but there is no telling that he knows the particular platform that is part of the solution at Trotters, or perhaps the platform has evolved since the last time that Kermit did any hands-on work on it.
  • Kermit cannot assess the experience of Robert and Marie-Louise on the platform either. Conversely, Robert and Marie-Louise have no effective way to find out about each other’s strengths and ensure they are complementary. Trotters’ organisation has to trust that things would work out somehow
  • Sometimes people do notice that their knowledge might no longer be sufficient, some could then become defensive and would fight to keep any blind-spots from the open. This is human, it can be fought but it should mostly be managed. Users, architects and programmers at Trotters are facing the same dilemmas, they might not even be aware that this could be motivating some of their actions.

We can go on and on about the factors that could be causing mismatches between user expectation and application releases, but the above is sufficient to derive a pattern and draw some preliminary conclusions.

When there is any significant distance between the architects and the actual solution building and delivery, inefficiency tends to creep in very quickly and organisations don’t get the most of their assets. It does not help when those who specify what needs to be done and outline how it should be done, are removed from the actual construction of the solution. When the architect has no idea what code was (or is being) written, he could be talking about concepts that only exist on his slides and documents but actually never materialised. What would be the point of that? The above is the rationale behind my statement that An IT Architect should write solid code.

In next settlements I will elaborate on how an architect could be writing solid code, increase his effectiveness in the organisation and ensure that his vision of the solution is as close as possible to the actual solution. I will call this Say it in Code.

PS:

I am an IS/IT Architect, I could be subject to the same shortcomings described above. This is one of the lessons that I have learned over time. I’ve always felt this to be an issue in my job, I try to minimise any gaps between the vision and the realisation that was made of it by staying close to the code.

Accidental IT infrastructure tsars

Some of the most successful innovations in IT infrastructure have come about by accident, by groups who couldn’t even be taken seriously when they started their venture. Is this the beginning of a bigger movement? Is this the edge of what an Agile culture could nurture?

In the past decade, some of the most successful IT infrastructure innovations have been stumbled upon, after the originators found themselves in unanticipated conundrum! In some instances, this kind of anarchic R&D is reaping rewards of several orders of magnitude higher than walled-in research efforts.

Think about it for a moment, we are enjoying an amazing array of technology that were created after some dudes’ fiddles and ramblings forced them to rethink their infrastructure. In their travails, clever & bohemian as they are, they’ve come up with solutions that many of us can enjoy. Witnessing such success, deep pocketed vendors joined in the fun, and the field is being redefined in the process.

Once frowned upon, FaceBook is now a leading contributor to prominent open source infrastructure projects. Google’s track record is longer and deeper, their data centre innovations are being copied around the world, their latest announcement in the U.S. is just illustrating how far reaching their ambitions are. The whole No SQL movement has spawned a category of data manipulation techniques forcing a rethink of the way we manage information. Google, Yahoo! and FaceBook are all strong leaders in this space. The best example is probably the open source Unix/Linux ecosystem (counting in the OpenBSD and FreeBSD variants, which are not Linuxes). What is happening with the uptake of Cloud computing is simply amazing, would make many a veteran chuckle (isn’t this what was supposed to be uncool?). Now that heavyweights such as Oracle and Microsoft are trumpeting the Cloud, we are probably entering a new era where mix-and-match will become a norm.

These infrastructure innovations are making it easier and faster for us to do more with our computing resources. Incidentally, this is also making us more vulnerable and fragile from a security point of view. Security breaches can do much more damage much faster than ever before. This could mean that our security infrastructure is simply being thrown out, forcing us to rethink the very notion of security.

Marketing mantra aside, here looking at you Google, these infrastructure innovations have had a profound impact on the way we perform computing these days. A substantial amount of these innovations were actually not planned, they seem to have been following a kind of Darwinian evolution theory, the good ones quickly gain traction and everyone adjust their positions.

Is this the beginning of a bigger and truly world changing movement? Is this a vindication of the open source phenomenon? Is this the edge of what an Agile (Californian?) culture could nurture?

Time will tell.

Code frameworks and ghettos, where creativity gets a lock-in

An acute issue in programmer productivity lies in framework learning curve. More often than not, no framework can be more productive than having to learn a new one. A code framework can be a force for good when it solves infrastructural issues and allow developers to concentrate on the specific job at hand. When the job is about addressing a user need, then it is well worth investigating the benefits of any framework before diving in head first. But when the job itself is about infrastructure, then the story could get more complicated.

I once read a rant by an open source developer called Zed Shaw, where he was saying that Rails is a ghetto. Zed was venting frustration about some members of the Rails (of Ruby on Rails) community, apparently he couldn’t monetise his skills properly due to politics in the community. Hence it felt like a ghetto to him. An interesting point I took from his rant was that you could get stuck in a framework and not achieve your goals, easy to guess but also easily overlooked. As I write this post, I checked the rant again for reference and I found that Zed withdrew it and replaced it with a much more toned down text, you can read that here.

A code framework can be a force for good when it solves infrastructural issues and allow developers to concentrate on the specific job at hand. When the job is about addressing a user need, then it is well worth investigating the benefits of any framework before diving in head first. But when the job itself is about infrastructure, then the story could get more complicated. In this post, I am only addressing situations when the job is about addressing user needs.

Many code frameworks are useful for beginners, useful for creating an initial solution framework. That’s the first phase of bringing ideas to life, this is where you’ve realised your Hello World and you’re all pumped up about how easy that was.

A second phase kicks in as you start learning more about the problem domain and potential solutions, you usually iterate through your design and code to reflect changes. This is a good test of the chosen framework’s flexibility, can you improve your design easily? can you improve your code easily? The further in the development, the more acute the questions become. If you’re lucky to think like the framework designers do, then progress should be swift. If not then a tough reality would start to dawn on you. Either way, your creativity start to be shaped by the frameworks that you’re using. This is where the potential pains with frameworks start to surface, it is a good learning stage. I liken this phase to betting on horse race: it always seems that the next bet will be a hit, people keep on betting. Invariably, there will come a time when the developer could feel that he was cheated somehow, he wasn’t told something or that he didn’t realise something.

The third phase in this framework journey is when the coding is done, and the application need to be put into production use. In this phase, performance and reliability issues usually start to pop up. The inevitable questions arise: is there anything wrong in my code? was the infrastructure correctly setup? If you don’t think the way your framework designers want you to think, then you could make missteps and issues would compound quickly. And maybe you do, but you just can’t help much since the framework ties you down. In this stage, only an elite few is able to efficiently address such issues. In the unfortunate case where the troubles call for an audit or project rescue by another party, the business owners tend to feel a bigger pain as their purse is severely tapped because an entirely different way of thinking and seeing the world is brought in.

Code frameworks are like giant bags of assumptions about technology, people, problem and solution domains. They help if the assumptions largely apply in the most significant aspects. When choosing frameworks, do people realise the extent of the assumptions they are about to make? How often do people bother to (or are equipped to) check such assumptions? These are interesting questions for a survey.

Creativity lock-in is not a bad thing if it helps people focus on specific and relevant problem and solution domains. When that is not the case, creativity lock-in can have devastating consequences. Ghetto can be a bit of a hyperbole, but it is a powerful paradigm for reminding people to consider a broader picture when selecting frameworks. A code framework should help with project technical hygiene, open up more opportunities. When the selection is mostly based on product marketing, then the assumptions are not being checked, who’s at fault then? – I wouldn’t blame the product vendors.

Buzzing, tweeting, now everybody can get a taste of being stalked

There is something sneaky about all this social media activity, people are turning into stalkers in their droves. Google Buzz just made this dawn on me. It can be quite uncomfortable to think that you are unwitting turning yourself into a “self strip view broadcaster”.

I’ve started buzzing, stumbling upon it’s activation at a browser restart. After a couple of days, I’m reconsidering the whole thing: why am I getting involved in all this?

Having been involved in innovations in social networking and online communities for a number of years, I was always going to try the latest stuff to see if it could help in my job. Little did I realise the slippery nature of all this. It is somewhat like “zapping” with your TV remote control, pointless but you somehow keep doing it until something urgent drag you away from the sofa. Before you know it you’ve spent a couple of hours watching the TV but seeing absolutely nothing. What a waste that is!

I started tweeting because I was too lazy to blog, it was just an excuse for me and I hoped to learn something in the process.

My idea of blogging has always been measured, I don’t like all the self broadcasting that goes with it, I didn’t want to write “hello world” program codes or regurgitate what others have been saying somewhere else.

I tried Google Wave and I never really found a reason to stick to it. I could imagine tons of use though, I just wasn’t ready for any of those – still am not.

Now I’ve joined Google Buzz, it’s giving me the goose bumps for it’s eerie stalker feeling. I never felt stalked like this before, it’s not due to my contact list or my friends activities, but rather it is the fact that I kept seeing a mirror reflection of my activities in many places of my Google account. Then you start to think: if I click on any link, it’s likely being broadcast. For someone  who is a private person by nature, this freaks me out.

A few years ago, I’d already taken a giant step by going out there and putting pictures, comments and various things online. I could always do it again, when I get a chance, and it felt ok. But, the idea that something follows me and tells the world about what I’m doing doesn’t feel right, I have no control over it. It is bad enough that this is happening, street CCTV cameras are an example, it’s much worse when you can actually see realtime the footages. That is what is troubling, and Google seemed to have thought none of that. Of course, when you’ve not known anything else you would just adapt to it.

There you go, the bucket must stop somewhere. I’ve got nothing to hide, but I don’t want to broadcast everything either. This is a case of a feature’s default option in user’s disadvantage. That is a hindrance to user acceptance from my point of view, and Google failed in this case. This case inspires me to write an architecture principle: design for acceptance, the default features and options must cater for user’s natural preferences and behaviour.

Failure with Enterprise Systems: why we don’t learn, and what we can do about it

With enterprise systems we insist in having a complete oversight before building things, but then we keep cutting corners all the way and all the time. There’s a paradox there: in the frantic attempt to achieve complete and total foresight, we become short-sighted and fail to notice numerous signals.
This short-sightedness is the result of insufficient learning, which is unfortunate since we really have plenty of opportunity to learn. Take any industry or knowledge domain that we love to paraphrase in our enterprise system endeavors, they all learn and apply knowledge incrementally. We’re the only ones, the IT industry, who think that we can learn everything and apply it all in one go. We start our mistakes in our inventory phase: as we seek to segregate the invariants from the variable elements of the enterprise systems. From there we just keep compounding blind spots, until someone eventually pulls the plugs or that we collapse.

Tim Bray wrote a great article with the title: Doing It Wrong, in which he is sharing his insights on the perennial IT failures and how other’s manage to succeed despite ignoring the enterprise mantra. Tim made a lot of good points, the reactions to it where mostly great reading of their own. I thought my reaction warranted a blog post of my own, since otherwise it’d be a long comment lost in the jungle.

I won’t paraphrase Tim Bray here, his original posting is well worth reading: http://www.tbray.org/ongoing/When/201x/2010/01/02/Doing-It-Wrong

At this point, I must warn the reader that most of my parabolas are best understood when situated in the context of mathematical set theory, and loosely, in probabilistic terms: the chances of, and the lack thereof. Thinking in terms of sets, matrices, stochastic and probability, provide a great foundation for abstracting and considering IT complexity.

My take-away from the examples of successful web sites is that they have one great attribute: solving one problem at a time. The trick is that, as they solve one problem, they learn and discover new unanticipated solutions and they can leverage that immediately in the resolution of the next problem. The result is that these endeavours end up achieving success in lots of areas that they never imagined when they started, that is the second great attribute. There are certainly more great attributes, and no doubt plenty of shortcomings, but I live in “glass half full” land.

Examples?: Google started with search, Facebook was about college students, Twitter wanted people to catch up (in their breaks) on what their friends where doing. On a micro level, the examples are innumerable and would make this posting longer than necessary.

In my view, the attitude of, solving one problem at a time, is applied to all aspects of the endeavour, not just technical.

The combined effect of the solving one problem at a time attitude, and the two great attributes cited earlier, embody the following tacit admissions:

  1. nobody is perfect,
  2. an organisation, just like a person, will never be perfect,
  3. it is wasteful to attempt to control everything from the get go, change occurs at a faster pace than our inventory and learning process
  4. we know what we want, but we never know where the events might lead to.

This type of discovery and progressive enhancement is not witnessed in enterprise systems environments, simply because it’s being preempted by stubborn short-sightedness and often times by analysis-paralysis.

With enterprise systems we insist in having a complete oversight before building things, but without knowing it we keep missing important pieces all the way and all the time, from start to finish.

There’s a paradox here: in the frantic attempt to achieve complete and total oversight, we become short-sighted and fail to notice numerous key signals. The signals take an inordinate amount of forms and they are well grounded in the context: business context, organisational complexity (human nature, entrenched interests, etc), technical complexity (how we fail to design and code what we are really thinking/imagining, how we fail to grasp the technology for what it really is), our span of control, our span of attention, the target solution lifespan, the limits of collective wisdom, etc, the list is endless. In my interpretation, a huge array of overlooked problems stem from human organisation shortcomings, by far the most insidious and  difficult to master: things like feeling, fear and uncertainty about one’s position or career, attention deficit, unexposed incompetence and mediocrity in the ranks, etc.

If we learned anything about architecture, we should know that it’s impossible to make an exhaustive list of every single factor that could become a part of the problem or the solution. If we can’t identify all potential problems and potential solutions, how can we possibly build something that caters for everything (which is what Enterprise System is supposed to be, a system that addresses all)?

We might be able to make a very good list at the start, things change as we list them. Say we overcome that hurdle, we usually don’t have sufficient control over everything in that list to be successful. If we manage to get sufficiently extensive control, there is still the hurdle of exerting such immense control for the length of time that it takes to build everything that is necessary. Say that miracles exist and that we manage to cross that last hurdle also successfully, then unless every stakeholder (internal and external) could be robotically piloted to perfection some things will go amiss.

At this point you might think I’m exaggerating the challenges, I probably am for many cases for sure. But if I were to underplay the hurdles, that also would not be the honest true because any one of the cited area of challenge can take a life of its own unbeknownst to or beyond the control of IT management. When such devastating change occurs, we enter another phase: denial for the sake of our own survival (the lies that cover the mother of all lies). That is a key aspect I want to point out. Christopher Alexander‘s seminal books provide a good background on the complexity of getting designs right (design in an extremely wide sense of the term).

There’s one more paradox: we are all quick to admit that “nobody is perfect”, we also know that an organisation is also a “body” and is a reflection of the humans that comprise it, yet we still think that we can build a “perfect organisation” (be it a project, an IT, a business, a program, a solution, or whatever structure or impersonation we give to our organisations).

The short-sightedness with Enterprise Systems is the result of insufficiently applying learnings, which is unfortunate since we really have plenty of opportunity to learn. Take any industry or knowledge domain that we love to paraphrase in our enterprise system endeavors, they all learn and apply knowledge incrementally. We’re the only ones, the IT industry, who think that we can learn everything and apply it all in one go. Our mistakes start to accrue in the problem and solution scope inventory phase: a lot of unaccounted for biases are introduced as we seek to segregate the invariants from the variable elements of the enterprise systems. From there we just keep compounding blind spots, until someone eventually pulls the plugs or that the project collapse under the sheer weight of the endeavour.

If we can’t get control over everything that matters before building successful enterprise systems, should we give up trying and forego all central control?

I’m not suggesting that we forego control and planning, that would imply that I know the magical answer, which I don’t and won’t ever claim to. I too, like many in my profession, am in my quest for the nirvana and I’m under no illusion that it exist or is achievable. And that last point is a suggested take-away: keep dreaming and reaching for the nirvana, but be aware that you might never achieve it.

At the beginning of this post, I cited one behaviour and two attributes, as expressing my take-away from what makes some web sites successful. With this I will be first to admit my own limitations: I am interpreting things with my own background and influences, someone else could have other equally plausible biases. The reader is entitled to his/her own bias. The important message is that we should learn to read signals and adjust our actions accordingly, achieving total control is rarely feasible or even desirable.

The odd saying goes like this: ” a fool with a tool is still a fool”. I’d fancy a positive version of it, glass half full type, like this: “a smart with a tool is a super smart”. In my experience, aptitude should be matched with an equal measure of attitude, they’re both in limited supply and should be consumed with moderation. Whenever the combination lacks of balance, failure looms. There’s a lot of preaching the good message, there’s far fewer demonstrable examples of doing it successfully. The range of tools and methods are wildly varied and have evolved tremendously over time. What’s remained virtually constant is the involvement of human organisations. Success still remains by and large elusive. Anyone claiming to have all the answers is definitely deluding. Awareness is the starting point, then actively doing something with the learning is the only essential positive ingredient.

I’d appreciate any learning from you, if you care to share.

iPhone 3G battery drain and potential code smell

When your iPhone is setup to use Push email update, it can drain your battery dramatically whenever the server becomes unavailable. This is clearly a design problem that should have been prevented. Applications should degrade gracefully, and similarly, they should enhance gracefully.

Recently I’ve been annoyed by an issue with my iPhone 3G, a phone I’ve had since August 2008: it’s battery was draining super fast and I didn’t see why this was happening. The battery life has never been stellar, but it also has never been this bad.

As I tweeted about this problem here, a friend of mine pointed out that this could be due to Push update when the server is not available, tweet here. He was right, as soon as I disabled the faulty email account, the battery issue stopped.

My problem is solved, but I’m not happy about what I think might be the root cause of this issue. I think I’ve bumped into a code smell, the like that architects ought to prevent for a pleasurable user experience.

When you think about it, the main reason that Push would drain battery life is that the phone would be continuously probing a non-responsive server, trying to reconnect. This would be mindlessly going on for as long as the problem persists, and that is precisely what I find troublesome.

A mobile service that is likely to experience connection problems, as a matter of fact any connected application, should have intelligence to space out the retries and give up after a reasonable amount of failed attempts.

When a server becomes unreachable, the device should stop trying and provide a feedback, visual or not. This enables the user to decide on an appropriate action or none. But I don’t see why the application would persist and keep trying again and again, until your device runs out of battery.

Looking at it from a coding perspective, there are two possible scenario: (1) someone decided to approve this functionality, or (2) it was overlooked.

In the second case, (2), this would qualify as a “code smell”. Such code smell could easily be remedied by simply introducing a variable and increasing retry interval, aiming to both reduce the number of attempts and giving it sufficient time to fail gracefully. An example such algorithm could be based on Fibonacci numbers, see Wikipedia, reducing the chances of mindless retries.

Naturally, for a company as large and reputable as Apple, this can hardly ever be a case of technical competence issue. They probably won’t even heard of this posting, or if they did, they probably wouldn’t care.

A big take away is that, it’s the tiny deficiencies of this kind that can accrue and potentially result in an overall poor user experience. At the end of the day, even if you have a nice architecture, you would be failing your users if you didn’t pay attention to tiny code smells with huge potential impact.

For a good user experience with connected applications, an architecture principle could be: degrade gracefully; after a reasonable amount of failed retries, stop and notify user when a server becomes unreachable. As richer features become available, gracefully enrich the user experience.

Effective architecture documentation practice on a budget

In documenting Software Architecture, boxes and lines are the norm and they are nice, they can even be aesthetic. But unless they shed light on otherwise unclear semantics, boxes and lines can be wasteful. Architecture documentation is more than the diagrams, it concerns everything that bears knowledge of the system and can inform decisions taken.

Think of good practices in waste reduction, since you have scarce resources.

Use architectural frameworks as guidance to good practice, don’t bother creating all kinds of artefacts that nobody in your teams will miss.

Regular architecture reviews and code reviews will do much to ensure quality and share knowledge within your teams. Run them effectively.

Boxes and lines are nice, can even be aesthetic. But unless they shed light on otherwise unclear semantics, boxes and lines can be wasteful.

To be effective, your architecture models must capture and maintain essential knowledge about your system, this includes user stories and issue and feature databases, and source code!

When you’re clear on what your team should produce and maintain and the scheduling, then and only then look for cheap tools that can help.

Here is a take:

You can get away with a few simple documents:

  • screen flows: use a simple drawing tool, or scan sketches into image files
  • user stories (or use cases), it’s your software promise to its users
  • design principles, essential Do’s and Dont’s for your technical folks
  • design trade-offs, technical decisions and their limits
  • contracts (internal and external APIs for your modules)
  • business rules, the knowledge-bearing decision points in your system

In the list above, one can argue whether business rules should really be included in an architecture documentation. The same goes for screenflows and user stories. But when running on tight budget, decisions tend to be made on the flight and built into the code, people rarely go back and update otherwise stale diagrams.

User stories are created at the initiation stages, but over time they are superseded by the issues logged against iterations of the product. The features could change shape as a result of the learning process, to a point where the initial architecture might become significantly altered.

What happens is that, over time, knowledge increasingly migrate to the issue database and the source code, and it stays there. That is the rationale for considering the issue database as part of the architecture documentation.

Some tips:

  • Avoid waste: any document that doesn’t hold unique information is a waste
  • Be economical with the boxes and lines, focus on contracts and boundaries
  • Use spreadsheets to capture your architecture principles, trade-offs, business rules
  • Maintain an active issue and feature database. You would probably be doing this anyway, but I’m saying that consider it an architecture documentation artefact
  • Encourage in-code documentation, enforce through standard quality checking and code reviews
  • If your tool-kit permit it, (thinking JavaDoc) generate documentation from source code and include them in your code review sessions

There are more tactics for capturing and maintaining knowledge within your organisation, but that is perhaps a topic for another post.

Leverage the best of community and agile approaches for results

It is a good idea to leverage the best of communities and agile approaches to product development in Software Architecture practices. Last semester product releases from Microsoft are indication that they are applying these concepts, Architects should learn from that.

It seems to me that Microsoft is applying some valuable lessons from open-source and community driven software processes: get user buy-in as early as possible, deliver early and frequently incomplete or partially flawed products. People will happily participate, you would get a more accurate feel for the reception when the product finally ships. I don’t know what the pundits are saying, but to me (as a software architect and user), this strategy pays off handsomely.

Every new Microsoft product I laid my hands on over last half-year has been nicely thought out and clearly user-oriented by design. You can almost feel that product development received a lot of attention, and that is good news for the 90% of us who use Microsoft software.

I’ve started using Microsoft Office 2010 Beta on my Windows 7 box. You immediately see the impact of some of the lessons they’ve learned with the previous major release, the ribbon concept has been tweaked, the experience feels more natural yet innovative. I’ve not had to spend any time reading documentation (though that would be a mistake for a final product, reading the documentation is always for the best), I had no troubles working just like before.

Similarly, when I first learned about the early Microsoft Azure visions, I was having a strange feeling of “lipstick on a pig” treatment. After PDC2009, I saw lots of improvements and change of heart on some early ideas that seem to be, without a doubt, the result of extensive community participation.

The early concepts and beta releases of SharePoint 2010 and Office 2010 have stunned me in their clarity of vision, for the first time I’m getting excited about Microsoft’s web-based software. Having spent the best part of last decade delivering non-Microsoft solutions, albeit I’ve never lost sight of what they were doing, I am seeing a lot of good vibes coming from Redmont these days.

Another potential idea that can be read here, would be that Microsoft is directly engaging users and hence doing away with their former approach where their partners supplied the bulk of feature requirements (I’ve read a lot of Michael Cusumano and Mary Jo Foley throughout the years, any misreading would be mine).

To me, these are all signs that Microsoft’s products are improving, they are increasingly addressing unmet user needs. This would be a software delivery equivalent of a “growth business”, I buy it.

I see a parallel with the practice of software architecture, whether its Enterprise Architecture or Solution Architecture on a smaller scoped project. Software Architects can achieve much success by adopting some of the same recipes hinted at earlier, by no means a complete list:

  • don’t seek completeness on any significant topic before getting stakeholder communities fully engaged (no, they won’t think you’re daft)
  • don’t think you have all the answers (many thought leaders are saying this), actively seek and incorporate ideas from the receiving parties – they’d have a hard-time to reject their own input, wouldn’t they?
  • delegate major chunks to smaller and dedicated groups, see to it that the inter-group communication is fluid and sustained (I don’t know if Microsoft does this, it seems many of their products are still large silos).

With this type of approach, the outcome tend to feel much more natural and the acceptance will probably be easier. You see it for example when, using the product, you guess how something might work and could verify that the feature was implemented almost exactly as you guessed. This happens a lot when I use Apple products. I used to think that Microsoft would never be able to match such feat, but I now see that they are changing their approach for the better.