Story Granularity, Complexity and Planning

Obie Fernandez wrote about some problems he's been running into with respect to story granularity, and Bill de hÓra picked up the thread.  This is a subject I've struggled with a number of times, so I'm going to give myself a little free therapy and try and work through some of it here.

Story Granularity

Whenever possible, it's good to break down the desired features of an agile project into stories that are pretty granular.  What "pretty granular" means to your team and company varies somewhat, but something in the range from a day to a week would be relatively common in many of the projects I've worked on.  

Once the story gets significantly beyond that threshold, most teams will do their best to try and break it down into smaller pieces.  But there are times where that's difficult, or possibly even undesirable.

Minimal Marketable Feature

One of the ways to look at story granularity is that you want to assemble the minimal marketable feature.  You want the story to represent exactly the minimum amount of work that you could do and still "market" or "sell" the feature to the desired userbase.  If anything were removed from the story, the story could no longer be sold, it won't be credible or cohesive.  I find this perspective is often useful to help me "right-size" stories.  Stories that are larger than the minimal marketable feature are gold-plated, and stories that are smaller than the minimal marketable feature don't deliver meaningful user value.

Unfortunately, it often seems to be true that the minimal marketable feature is larger than your desired story size, and that creates a subtle tension between project and product management in an agile environment.

There are, of course, ways to try and work around this problem, and de hÓra talks about some of these.  Fundamentally, what tends to happen is that you subdivide the minimal marketable feature.  When you do this, it tends to create some other interesting problems.

How can you subdivide the minimal marketable feature?

Sometimes, this isn't very hard.  There are natural lines in the feature that allow you to implement it in several phases, each of which makes a cohesive piece of functionality, just not functionality that is individually marketable.

Often, however, the functionality cannot be broken into cohesive chunks.  If you end up having to deliver it in pieces that might individually break parts of the system, it's difficult to find the right division lines.  It gets even more complicated when in order to subdivide the feature and estimate the pieces, you need to pick an implementation approach.  Selecting a detailed implementation approach is a decision that many agile teams would like to defer past the initial planning stage, but if you can't easily finish your release and iteration planning without dividing the story and you can't divide the story without talking about implementation, then you're somewhat stuck.  

In these cases, you could take a 'theme' or an 'epic' past your planning stage, knowing that it's a risk and an unknown, but preferring that to talking about implementation concerns too soon.  Alternately, you could buckle down and do some up-front design and hope that whatever plan you come up with still makes sense when it's time to do the detailed implementation.  There are tradeoffs to be made here, and I can't make them for you.

What happens when the minimal marketable feature spans several iterations or releases?
If the subdivision is necessary for tracking progress, but all the work will be done within an iteration, then you've dodged another bullet.  If, on the other hand, the pieces that make up the minimal marketable feature will be split across two or more iterations, you're facing a new problem.  Even if those iterations are within the same release right now you may be reducing your ability to release, to adapt, to change.  

This is only a small problem if the subdivisions each deliver cohesive functionality that are simply too small to be individually valuable.  It's a much bigger problem if the elements aren't individually cohesive, where the result of a particular story might leave your project in an inconsistent state.  In these cases, you may need approaches that allow you to deliver code without linking it directly to the application, or the ability to deliver features that are disabled.  If, like some organizations, these approaches are already necessary (e.g. to work with multi-tenant SaaS solutions), this may not be incredibly painful.

Are large stories a cause of problems or a symptom?

Sometimes, the presence of large stories isn't the real source of the problems you're facing.  The problem is that the story shouldn't really be as large as it is, and has become so for a few reasons.

First, you want to be sure that the minimal marketable feature really is minimal.  This can happen when the product owner / manager is relatively new to agile planning and hasn't yet experienced the joy of discovering that your users can get by with a lot less than they think they need.  It can also happen when clients are unreasonable, when features are designed by committees, or filtered through a decision-making structure where each layer needs to 'innovate'.  It can happen when someone is trying to avoid past mistakes by ensuring that new features are robust and flexible, a sort of anti-YAGNI tendency.  If this is happening, you need to understand why it's happening so that you can try and find a solution that doesn't involve building extra-large stories where a medium-sized story will do.

You also need to come to terms with overall system complexity.  I've often seen systems that grow and evolve, add special cases and new features that interact with each other to create a surprising amount of complexity.  When this happens, adding even simple features can require a great deal of work.  "Sure, we can add that 'customer profile' box on the home page, but we'll need to create a customer profile display that is aware of the multi-tenant additional fields and meets the role-based security constraints.  We'll need to add several new service methods to customer web service API, and a new component to the user interface layer.  Unfortunately, three of those fields aren't available to this system, so we're going to need to make a JMS call to the CMS in order to get those pieces. We'll need to be able to turn the component on and off on a per-client and per-interface basis, and possibly on a per-role basis in order to work with existing clients, not all of whom are going to want to display this kind of information, and we may also  need to add field-specific display configurations."  

I might be exaggerating a little, but as systems grow, they tend to accumulate complexity, and many organizations don't do a strong job of keeping complexity down, either because they don't understand the heavy price they're paying for the complexity, or because they simply feel that the complexity is necessary.

In Summary

It's easy to say stories should be small.  It's not always easy to make it happen.  The devil's in the details.