Should software really be built like Model Ts?

Is the assembly-line structure of your enterprise IT organizations threatening your job? It may be.  Here’s why:

IT organizations are structured more for the benefits of accountants than for customers or IT teams themselves.

For example, here’s a diagram of a typical data warehouse (or business intelligence) project that I managed for a financial services firm:

Software Assembly Line
Most enterprise software today is built like Model Ts

The division of labor seems logical. How else could you approach it? After all, the data tier must be built first, so that the services can reference and consume them. The services must exist before the reporting tier can build any reports. And the whole thing must be operational before the testers can poke holes in it, right?

When you lay it out sequentially, it looks like this:

BI Waterfall Delivery
Assembly line IT translates to waterfall delivery

Recognize that? It’s the infamous waterfall of software development. Waterfall has been widely discredited since the 1990s. Yet it is the dominant approach used by IT organizations, in no small part because of the accounting department’s desire to “maximize resource utilization” by loading programmers with 4-5 projects at once. The implicit assumption here is that knowledge workers can easily task switch, carefully dividing their days into 20% increments, and shift from one to another with no loss of productivity. Heck, most of us can’t even get “work/life” balance right, much less balance five detailed projects at once!

Stockholm Syndrome

When IT managers spend their whole day looking at spreadsheet summaries of their projects rather than being in the weeds with the team, they begin to see their world as the accountants do. Managing an IT budget is much easier when you have steady burn. Since most corporate IT is not revenue generating, one of the few options for managers is to do more with less: more projects per person.

In an assembly-line model, this makes sense. Simply turn up the conveyor belt speed, and have workers build more widgets per hour. But in enterprise software, this usually turns into the chocolate factory episode of the old I Love Lucy show.

Now, here’s the rub: CIOs are an endangered species. The latest numbers from industry sources like CIO Magazine and Gartner show that CIOs are being squeezed out of the C-suite, not replaced when they move on, and viewed more and more as purely operational directors, reporting up to the COO, or, in cases where IT spending is out of hand and delivery is particularly egregious, to the CFO. You may not feel the continents shifting beneath your feet, but dedication to traditional “assembly line” delivery methods may be shifting you right out of a job.

Transference

Many IT leaders insist that their technology domain, whether distributed development, middleware services, data warehousing, or service-oriented architectures, is too complex for modern agile development methods. They are wrong. Many IT leaders transfer their failures onto these new-fangled (25 year old) agile methods. Instead, they should be looking to themselves, and their own failure of proper execution.

The conceit comes from the fact that scrum and similar lean methods often refer to user stories. “But we don’t have users, per se, so we can’t possibly have any user stories, so we must use waterfall,” say the team leads. There are three things wrong with this view.

First, waterfall is a proven loser in new software development in 99% of cases. And no, it’s not “better than nothing.” It’s often considerably worse. Waterfall tends to pit business and IT against each other, spawns costly and often useless documentation work, and generally destroys morale inside a team, since they will be beaten to achieve arbitrary dates and feature sets. This last one often results in high rates of attrition, or ‘brain drain.’

Second, focus on business value, not “user stories.” A user story is an abstraction. Call it a use case, a requirement, a feature, or an oompa-loompa. What matters is that it represents a piece of business value. When I employ agile for data services projects, for example, I help the team get away from the “user story” mindset. The business customer has asked for some items of great business value, such as new or better reports. These reports may require you to build a new infrastructure; create a very complex data model; marshal data between systems; and build visually rich report UIs. Your project team and leader must break these items down into elements of business value, either for the end customer, or a companion team, or both. This takes as much creativity as logic.

Finally, you do have users. In fact, you have three different kinds of users:
• Inter-company teams who will help you build and maintain the software;
• Staff members who will use the software daily;
• Business customers or executives who paid for the project.

Here are some techniques I use on so-called “back-end” projects to focus on customer value, and get out of the assembly line mentality:

Flip the Assembly Line 90° with a Delivery Stack

Instead of an assembly line of development tiers, with further silos (role divisions) inside those tiers, I prefer to focus on a feature that the Business Customer values, and then reconfigure the team to form what I call a Delivery Stack. Like a technology stack, a Delivery Stack is the minimum set of people I need to deliver the first major feature. I say “people” rather than “roles” because some people are capable of wearing multiple hats (ideally, all team members are multitalented). The Delivery Stack for our business intelligence project above looked like this:

Curtis' IT Delivery Stack
Flipping the software assembly line on it’s side gives us a Delivery Stack

Fence off the team

To reduce a project’s loaded cost, start with the smallest team possible, and dedicate them to the project 100%. This will help compensate for the resource maximization pressure from Accounting. Even on very complex projects, three to five of your best and brightest can push the train a mile uphill in the time it would take a typical assembly-line project to get out of the “Project Kick-Off Phase.” As the team gels, you can add more resources slowly to help widen the delivery pipeline.

Commit

No matter which flavor you choose (scrum, XP, lean, kanban, etc.), iterative development is far more effective than traditional waterfall. Some methods are more appropriate than others for certain situations. For example, if you are running a support team rather than a new development team, you’ll want to focus more on bandwidth, much like a call center team employees a “flying geese” strategy for maximum utilization without overload. In these cases, kanban is probably more appropriate than, say, XP.

The other aspect of commitment is to stand behind your team. This entails three things:

1. Provide your team the tools and coaching needed to succeed.
I say “coaching” because mere books or classroom training is inadequate. The first few iterations are always problematic, and without experienced, embedded coaches to assist, your teams will fail, and gravitate back to the comfy blanket of waterfall, where they can blame the business for not giving them all the requirements up front.

2. Don’t blame the methods for failure.
The methods are fine. Iterative knowledge development has been established for millennia, since Socrates led his students through inductive proofs. If it’s not working, don’t let the team off the hook. Make them look inward for improvements in execution (again, a coach can help here).

3. Educate the Customer.
The Business Customers will want to know “When will it be done and how much will it cost?” Period. They will want your teams to commit to dates and set milestones along the way. They will beat you with a stick when you miss those dates, because they, too, are getting beat with sticks by their superiors to whom they conveyed your promises. The “cone of uncertainty” is not a nitpick; it is an epistemological problem of the highest order: how do we know the nature of a thing which does not yet exist, and has many independent variables? You can answer Business Customer questions, but you’ll need context, like degrees of confidence, and the difference between effort and duration. Of all the skills needed for successful delivery, the ability to help Customers understand this is the most vital.

Clothe that Emperor

Change takes courage. Though you may sometimes feel like a voice in the wilderness at the executive staff meetings, rest assured you are not wrong: that emperor is indeed buck-naked. Use the techniques described here, and your own team’s innovations, to rapidly improve your software delivery track record.  If you’re not sure where to start, call me.  I’m happy to help.

Share your thoughts

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s