Ahmad’s Muse

Posted by ahmadster at January 12th, 2007

I believe this is one of the most destructive and under estimated bugs that inhibit devs (not code, but developers).

The outcome of over-engineering is heartbreaking on so many levels. Not only is the product destroyed, but resources are wasted, careers are ruined, and egos get crushed, in this case needlessly. But worst of all, this disease goes on undiagnosed decimating project after project.

I know I will get lambasted for saying this, but I honestly see this most (not all, most) with devs coming from a specific discipline. One that mainly uses a technology that starts with J and ends with AVA, referred to as J*** from here on.

Consider the following diagram to better understand what I’m noticing.

As I said, this behavior appears to manifest itself mainly within the J*** community. It is already an established fact (if it isn’t, I’m establishing it now) that computer languages and the characters of the devs using them are tightly woven and intertwined. To know a dev is to know his languages.

Case and point, I indirectly got involved in a project that had been in the works for about 8 months already. The requirement was the classical multi-data-sources that needed a single merged view. The solution that was proposed and worked on by your friendly neighborhood J*** dev was a central data warehouse that collects data from many sources into a single Mega-DB. Then that DB would provide that data by merging it chronologically.

In 8 months, work on plug-ins that connect to many different kinds of sources and protocols was about 50% done (to my knowledge). The client was very unhappy, after all that time there wan nothing to show yet.

I cannot take credit for this one, so I must give credit where credit is due. Danny (our CEO) was asked to look at the problem and suggest a solution. Three (3) days later, the site was up and the client was pleased to see data flowing.

So how did he do it? the answer was RSS!

It was very easy for any data source to provide the data in XML. Not just any XML, but RSS XML. Then using any plain RSS aggregator, data would be merged chronologically. No Mega-DB to build and maintain, no periodical data dumps, no plug-ins, and almost no code involved at all.

We did spend some time fixing up a PHP RSS aggregator and adding a search facility, but the solution was dripping of simplicity and elegance, it was art.

I have no idea why all those protocols were used before, I don’t know why they didn’t ask for a standard form of data from all sources, or if a Mega-DB was actually part of the requirement. Maybe there was more to it than what I was told, but it remains an example in my mind of how devastating over-engineering is.

I would advise that you at least be aware of the possibility that you may be committing over-engineering at any time. And be brave enough to re-evaluate your decision (another Agile Software Development principle).

Posted by ahmadster at January 7th, 2007

Update: Upon request, I have changed the post to protect the innocent. It’s hard trying to discuss issues like these without stepping on people’s toes and pointing fingers. So forgive me if I leave out some major details in the spirit of conveying knowledge without hurting feelings.

Back in the year 2000, we had a field trip to one of our customer’s agent offices to see how they were using our flagship product. We had recently updated their site so we were excited to see how they react to the cool new chrome.

I sat down with an agent that proceeded to show me her work day on our tool. Midway through the process we hit a page that stopped her in her tracks. We tried over and over and we hit the same bug and it kept taking more and more parts of the site down. Obviously the site was broken, and productivity level was at 0.

My first reaction was to call the office and have that update rolled back. But you see at that time I wasn’t CTO, I was Lead Dev and thus considered to be the pure embodiment of evil as far as the Network Admin’s were concerned. The servers were under the Network Admin’s jurisdiction, so no dang roll back was ever going to happen that day no matter what.

So what was the objection all about? any changes to the site had to be reviewed by the Network Admins before it got pushed out to ensure uptime. Isn’t that just swell. The site is broken! it can’t get any broken-er than that.

I have a big smile on my face every time I think of that moment. You will smile too when I tell you that our Network Admin’s didn’t know any programming language at all (ok batch files are scripts so,, you got me there). The absurdity of the moment just made me giggle. I wasn’t even supposed to converse with the sole tester (QA) because that would “taint the process”. If that process was so effective, how did that bug make it all the way to production?

I know it took more than a day to finally do the roll back (maybe two, not sure). Needless to say, we lost the client. Another incident I remember was the “question mark” incident. A client’s front page had one simple typo, an extra question mark. How long do you think it should take to edit an ASP page and fix the HTML part to remove a dang question mark? it took 3 full days. Did we keep that customer for long?

This brings us to my point. When I took over as VP of IT (eventually rename CTO), the first thing I did was eliminate these ridicules shenanigans. The people involved in any of our clients projects were less that 5 per project. That’s including the contact points at the client’s side. Clients that came in with a laundry list of minor fixes were dealt with as we spoke to them. Bigger more urgent problems were assigned to a SWAT team. Highly skilled and trusted devs that go in, get the job done, and get out.

If you notice how a SWAT team works, they burn lots of calories real quick in a flash. They can’t keep it up for ever, but they are precise and effective. Every dev team needs to assign at least a couple of devs to be their SWAT. Most noob Project Managers keep dumping everything on their best devs because “they do the best job”. That brings in those dreaded words that should keep every PM up at night, Developer Burnout.

A burnt out developer is like an over-heated pulley. She/He will lockup on you will be left in a tangled mess. Other terms that comes to mind are Spaghetti Code and “What the heck is this?!”. If you start hearing those, then my friend, you have a common case of Developer Burnout.

All these rants I’ve mentioned here are part of a bigger story. Much like AJAX, something that has been there since the oceans were dug but now its New, Cool, & Correct™, so is Agile Software Development. It’s nice to see the world finally semi-wake up. But man was it a painful trip.

The stories I mentioned earlier are all symptoms (indirect as in, not the direct cause) of what is known as the Waterfall Model. If you went to Management school right about when I went to it too, all information system software projects were based on this model. It was/is the official model of software development (think IBM in the 50’s). It’s not that I think its totally wrong, its simply a matter of using common sense to decide when to use what method. In a small startup trying to carve it’s mark on the trunk of the hightech history tree, it is simply reckless not to be as Agile as possible.

So here is my advise to all the noob VP’s, PM’s, and CTO’s out there. Respect your devs and trust them. Work organically, drop all those hydrogenated, bleached, enriched, unnatural methods you were told were the truth. Your focus is the customer. Use the Force, Luke!

BTW, 7 years later, Etelos Systems, Inc. is a bastion of Agility, check it out, its brisk baby.