Story Granularity, Complexity and Planning

Obie Fernandez wrote about some problems he's been running into with respect to story granularity, and Bill de hÓra picked up the thread.  This is a subject I've struggled with a number of times, so I'm going to give myself a little free therapy and try and work through some of it here.

Story Granularity

Whenever possible, it's good to break down the desired features of an agile project into stories that are pretty granular.  What "pretty granular" means to your team and company varies somewhat, but something in the range from a day to a week would be relatively common in many of the projects I've worked on.  

Once the story gets significantly beyond that threshold, most teams will do their best to try and break it down into smaller pieces.  But there are times where that's difficult, or possibly even undesirable.

Minimal Marketable Feature

One of the ways to look at story granularity is that you want to assemble the minimal marketable feature.  You want the story to represent exactly the minimum amount of work that you could do and still "market" or "sell" the feature to the desired userbase.  If anything were removed from the story, the story could no longer be sold, it won't be credible or cohesive.  I find this perspective is often useful to help me "right-size" stories.  Stories that are larger than the minimal marketable feature are gold-plated, and stories that are smaller than the minimal marketable feature don't deliver meaningful user value.

Unfortunately, it often seems to be true that the minimal marketable feature is larger than your desired story size, and that creates a subtle tension between project and product management in an agile environment.

There are, of course, ways to try and work around this problem, and de hÓra talks about some of these.  Fundamentally, what tends to happen is that you subdivide the minimal marketable feature.  When you do this, it tends to create some other interesting problems.

How can you subdivide the minimal marketable feature?

Sometimes, this isn't very hard.  There are natural lines in the feature that allow you to implement it in several phases, each of which makes a cohesive piece of functionality, just not functionality that is individually marketable.

Often, however, the functionality cannot be broken into cohesive chunks.  If you end up having to deliver it in pieces that might individually break parts of the system, it's difficult to find the right division lines.  It gets even more complicated when in order to subdivide the feature and estimate the pieces, you need to pick an implementation approach.  Selecting a detailed implementation approach is a decision that many agile teams would like to defer past the initial planning stage, but if you can't easily finish your release and iteration planning without dividing the story and you can't divide the story without talking about implementation, then you're somewhat stuck.  

In these cases, you could take a 'theme' or an 'epic' past your planning stage, knowing that it's a risk and an unknown, but preferring that to talking about implementation concerns too soon.  Alternately, you could buckle down and do some up-front design and hope that whatever plan you come up with still makes sense when it's time to do the detailed implementation.  There are tradeoffs to be made here, and I can't make them for you.

What happens when the minimal marketable feature spans several iterations or releases?
If the subdivision is necessary for tracking progress, but all the work will be done within an iteration, then you've dodged another bullet.  If, on the other hand, the pieces that make up the minimal marketable feature will be split across two or more iterations, you're facing a new problem.  Even if those iterations are within the same release right now you may be reducing your ability to release, to adapt, to change.  

This is only a small problem if the subdivisions each deliver cohesive functionality that are simply too small to be individually valuable.  It's a much bigger problem if the elements aren't individually cohesive, where the result of a particular story might leave your project in an inconsistent state.  In these cases, you may need approaches that allow you to deliver code without linking it directly to the application, or the ability to deliver features that are disabled.  If, like some organizations, these approaches are already necessary (e.g. to work with multi-tenant SaaS solutions), this may not be incredibly painful.

Are large stories a cause of problems or a symptom?

Sometimes, the presence of large stories isn't the real source of the problems you're facing.  The problem is that the story shouldn't really be as large as it is, and has become so for a few reasons.

First, you want to be sure that the minimal marketable feature really is minimal.  This can happen when the product owner / manager is relatively new to agile planning and hasn't yet experienced the joy of discovering that your users can get by with a lot less than they think they need.  It can also happen when clients are unreasonable, when features are designed by committees, or filtered through a decision-making structure where each layer needs to 'innovate'.  It can happen when someone is trying to avoid past mistakes by ensuring that new features are robust and flexible, a sort of anti-YAGNI tendency.  If this is happening, you need to understand why it's happening so that you can try and find a solution that doesn't involve building extra-large stories where a medium-sized story will do.

You also need to come to terms with overall system complexity.  I've often seen systems that grow and evolve, add special cases and new features that interact with each other to create a surprising amount of complexity.  When this happens, adding even simple features can require a great deal of work.  "Sure, we can add that 'customer profile' box on the home page, but we'll need to create a customer profile display that is aware of the multi-tenant additional fields and meets the role-based security constraints.  We'll need to add several new service methods to customer web service API, and a new component to the user interface layer.  Unfortunately, three of those fields aren't available to this system, so we're going to need to make a JMS call to the CMS in order to get those pieces. We'll need to be able to turn the component on and off on a per-client and per-interface basis, and possibly on a per-role basis in order to work with existing clients, not all of whom are going to want to display this kind of information, and we may also  need to add field-specific display configurations."  

I might be exaggerating a little, but as systems grow, they tend to accumulate complexity, and many organizations don't do a strong job of keeping complexity down, either because they don't understand the heavy price they're paying for the complexity, or because they simply feel that the complexity is necessary.

In Summary

It's easy to say stories should be small.  It's not always easy to make it happen.  The devil's in the details.

Agile Office Furniture

When looking at systems furniture or other corporate furniture, are some better-suited to agile techniques than others, including pairing and pair-swapping? I was just looking at the Kimball Hum desk, which is appealing in some ways, although it still doesn't match the truly mobile solutions where you could have a workspace but move it around easily (Herman Miller used to have a desk I really liked in that respect, although I haven't seen it recently).

I also have fond memories of using a Huddleboard.

I'm curious if some of you have recommendations with respect to furniture that's worked well for you.

Project Wiki or Project Blog?

I've seen a lot of technology development projects adopt wikis as a way of sharing information. Often, unless you appoint someone the job of keeping the information both up-to-date and organized, these often get out of hand pretty quickly. Without a common kind of organization to keep things in check, it gets difficult to find information, and without someone constantly reviewing the documentation for relevance, some information gets out of date quickly while others remain relevant.

I'm starting to wonder if projects should probably seek a platform that emphasizes a blog style rather than a wiki style. The blog entry metaphor is more suited to "writing information about a point in time." When you do a search for something in your project blog and find information from May 2006, you're much more likely to think to yourself, "Hey, I wonder if this is still relevant."

That's not to say that a Wiki isn't a good idea, just that wiki content should be saved for information that is both long-term relevant and worth keeping up-to-date, and then someone should, in fact, regularly review it for relevance and make any necessary updates.

In keeping with my agile mindset, this kind of regularly updated documentation should be kept to that information that is very valuable, so that you don't distract a team whose job is creating value through software and end up spending a lot of their time reviewing and updating documentation that isn't necessary.

Toronto Summer: Agile 2008, RubyFringe

Wow - Summer 2008 is going to be a killer Toronto summer for Conferences. We've got:

• Agile 2008: August 4th-8th, the best-known conference on Agile.
• RubyFringe: July 18th-20th. A new conference dedicated to the fringe of Ruby, everything but Ruby on Rails, really. This presumably covers 'waves' and 'merb' and many other things. Just announced.
  

Individual and Collective Responsibility and Standup Meetings

Although I believe that stand-up meetings can be a powerful tool, I think they often go astray: they turn into status meetings. Others have covered the patterns and anti-patterns of standup meetings before, but I'd like to delve into this one point in more detail.

When standup meetings turn into status meetings, I'm inclined to think the problem often traces back to a sense of where the responsibility lies: with the individual or with the team.

Individual Responsiblity
Does each member of the team think that he or she is responsible for a task and must let the rest of the team (or a subset of the team) know where he or she is at? That's a status meeting, and it stems from a sense of individual responsibility.

With this mindset, individual team members may feel as if he or she is personally late to bring a task to completion mid-iteration, that the tasks others are on is interesting, but basically unrelated to the task at hand, and that a standup meeting is simply a way of keeping in touch with the rest of the team, and more importantly, making sure that some "important" members of the team know what's going on. The danger here is that each individual focuses on their task to the exclusion of the team's goals.

Collective Responsibility
What you should be striving for is a sense of collective responsibility. The team, as a whole, should understand where they're trying to go, what's important and what isn't, whose tasks may be on the critical path, and what those tasks mean. Without this collective responsibility, a team will often not act like a team, but rather like a collection of individuals.

When a team member on an important task is stymied, a team with a sense of collective responsibility will react naturally to that, attempting to route around damage. Rather than worrying about their own task they'll react to the needs of the team, trying to accomplish the goals set for the team, rather than worrying about a task that may or may not have been assigned to (or, better, selected by) them.

It's from this collective responsibility that "self-organizing" arises, and without it, an agile team is simply a set of very small teams managed by a central command structure.

Lazy Web
Have you experienced both mindsets? Have you managed to move a team from individual to collective responsibility? Does this make sense to some of you?

Couples Therapy for Business and Agile Software Development

Agile software development practices and businesses have been trying to get along for a few years now. In some environments, they're able to find common ground, work together to solve each other's problems, and otherwise form a healthy relationship. In others, they can't even seem to have a conversation without squabbling.

After considering their commonalities and their differences, I propose a form of Couple's Therapy to resolve the differences.

Different Perspectives
It's important to start by recognizing that both the agile developer and the business manager have different perspectives, but that neither party is wrong.

Business managers need to plan: to write budgets, make business plans, and even put out release schedules. These are natural parts of running a business, and they're valuable in their own right. A business that doesn't make budgets and plans isn't likely to have the necessary resources, to meet their client's or market's timelines and is a business that's likely to fail.

On the other hand, an agile developer knows that requirements are almost always unclear, always changing, and that, in fact, the users don't really know what they want. They know the cost of implementation is, at best, a guess, and that the best way to manage this is through some kind of process that balances the constants needed to accomplish goals with the dynamism and need for change that makes agile processes so successful. To manage through constant course-correction rather than trying to predict the future with up-front detailed planning.

These perspectives can seem at odds. An agile developer is wont to say things like, "We don't know when it'll be done" and "I can't tell you what features will make it into that release". These statements can be disturbing to a business manager who is unfamiliar with agile processes or with software development in general.

On the other hand, business managers might say things like, "I need a committed release schedule" or "If you don't tell me how long this will take, I can't make a budget or get resources for this project!" These statements can be disturbing to an agile developer who hasn't learned how to interact with business.

Common Ground
Although these viewpoints focus on different things, there's far more common ground than some people realize. Most business managers have seen or heard enough about software development to know that it's unpredictable, and that software projects are hard to deliver "on time, on budget". A good manager also knows that plans are meant to be changed, that priorities shift constantly, and that nothing can be set in stone.

On the other hand, Agile teams know that they need to have resources, and that resources tend to come from budgets. They have planning metaphors, even if those aren't always the metaphors to which businesses have been accustomed.

Both businesses and agile teams desire successful project completion. Both need to react to change on a regular basis. Both thrive on up-to-date information. Both desire to understand the pace of development, and to make sure that the right features are built in the software at a reasonable cost. These commonalities are more fundamental and more important than the differences.

Working Together
So, having realized that both parties have different, valid perspectives, how can they find ways to work together?

Agile teams can help business to plan, to budget by estimating desired features, working on release plans and otherwise helping the business to understand how long it will take to develop software. As long as both parties understand that plans are subject to change, this planning activity can be valuable.

Agile developers and coaches can also help businesses to understand the positive aspects of agile development: the support of change, the emphasis on delivering value early and frequently and on delivering value through software rather than other artifacts.

Businesses, on the other hand, can accept that up-front detailed requirements and planning for software development is rarely as effective as everyone might desire. That many projects fail, and the ones that do are the ones that refuse to deal with the realities inherent in software development. Business managers can encourage 'up to date' information instead of encouraging an up-front plan that never changes.

Both businesses and developers need to keep the other up-to-date regularly on the factors that affect planning: team velocity, changes in scope, timeline and priorities. Keeping the flow of information open, honest, and frequent is just as important in a business relationship as it is in a personal relationship.

The two teams can then work together on adjusting plans, estimates and schedules based on updated information. This kind of collaboration is possible and normal when agile software development and businesses form and sustain a lasting relationship.