Okay… let’s set a little context here. In my last post we talked about two different types of projects. The ones that are knowable and the ones that aren’t knowable. Projects where it makes sense to estimate and projects that are more like R&D investments where we are spending money to learn and discover. Today, I want to talk more about the first kind. The ones where we do have some idea of what we are building and the technical challenges that might be involved.
Getting clarity on what we are going to build and how we are going to build it isn’t easy. This is especially true when we have multiple competing stakeholders, no clear way to resolve priority conflicts, more to do that we can possibly get done, and the technology while understandable, certainly isn’t trivial. In the face of this kind of ambiguity, I think that many of us have thrown up our hands and concluded that software isn’t estimateable and everything should be treated like R&D.
I think this is a mistake. If we keep pushing to treat everything like R&D, without understanding the delivery context we’re working within, the whole agile movement risks loosing credibility with our executives. If you remember from our earlier conversations, most companies have predictive-convergent business models. We may want them to be adaptive-emergent, but they aren’t there yet. We’ll can talk about how to move these folks, but for now we have to figure out how to commit.
Back in the day when I was just learning about agile, and immersed in the early thinking of Kent Beck, Alistair Cockburn, Mary Poppendieck, and Ken Schwaber… I came across this idea that the team had to commit to a goal or an increment of the product at the end of the iteration. There were some preconditions of course, the story had to be clear and understandable, we needed to have access to an onsite customer, and the team had to have everything it needed to deliver.
If the story wasn’t clear and understandable, there was this idea of a spike. I’ve always understood a spike to be some snippet of product we’d build to go learn something about what we wanted to do or how we were going to do it. This idea has expanded some over the past few years to include any work we bring into the sprint to do discovery or learn something we didn’t know before. It’s basically an investment the product owner makes to get clarity into the backlog.
Like most of you guys I’m sure, my planning approach was heavily influenced by Mike Cohn’s Agile Estimating and Planning book. Mike turned me on to Bill Wake’s INVEST model and I’ve used that as a tool for understanding and teaching good user stories ever since. The longer I use the INVEST model the more profound I think it is, but as widely accepted as this idea seems to be, I think there are a few under-appreciated letters in the model. The one most relevant to this discussion is the ‘E’.
E is for Estimateable
The idea behind INVEST is that these six attributes of a user story set the preconditions the Product Owner must meet before the story can be brought to the team. If they are not INVEST ready, they get deferred and we schedule a spike. The idea behind the ‘E’ specifically is that the user story must be well enough understood by the team that they know how to build it. The team has to be able to break down the user story enough to put detailed estimates on the tasks and be willing to commit.
Remember when I said that almost every disfunction on an agile team tracks back to the backlog? We’ll here is the problem… most teams are accepting user stories into sprints that are not INVEST ready and are certainly not estimateable. If the team doesn’t have enough information to make a commitment they shouldn’t make a commitment. So… do we conclude from this that software isn’t estimateable or do we conclude that we collectively do a poor job backlog grooming.
Because most of us work in dysfunctional organizations that don’t manage a roadmap, don’t stick to their vision, thrash around at the executive level, and generally can’t make up their mind… many of us have come to the conclusion that creating a backlog is waste. Why spend all that time writing up user stories when things are going to constantly change? This is indeed a problem, but giving up on planning is not the answer. Making up your backlog as you go isn’t the answer either.
When we make up the backlog as we go, everything the team encounters is going to be new to them. Everything they do is going to require a spike. The whole notion of Sprint Planning and Release Planning is predicated on the notion that we generally know the size of the backlog, we learn the velocity of the team, and based on those two variables we can begin to predict when we’ll be able to get done. If everything requires a spike, the backlog is unstable and indeterminate.
For every user story we attempt to build, we create at least one spike, and probably two to three more user stories. Even if we have a stable velocity, the scope of the release is increasing faster than we can burn down the backlog. We never end up with stable view of what’s happening on the project. The team feels like it’s thrashing, the product owner feels like their thrashing, and the organization gets frustrated because the product isn’t getting out the door.
Dealing with Backlog Uncertainty
To solve this problem… to get better at making and meeting commitments… even at just the sprint level… we have to plan a sprint or two ahead of the increment we intend to commit. That means if we want to get better at nailing sprint commitments, we need to have backlog properly groomed BEFORE the sprint planning meeting. If there is spike work to be done, that needs to be identified and done BEFORE the sprint where the user story is going to be built.
The implication here is that the team (in any given sprint) is going to have some capacity allocated toward the work of the sprint and some capacity reserved for preparing for the upcoming sprints. This could be as simple as having a meeting or two every sprint to help the PO groom the backlog, do look ahead on the backlog, to ask questions and to give guidance. Sometimes it’s more ad-hoc and sometimes it’s more formal. Some teams track this work, some just allow it to lower velocity.
Either way, allowing some slice of capacity for preparing for the upcoming sprints is an essential element to begin stabilizing delivery, agree? If you are with me so far… I’d suggest that stabilizing a release follows the same rules we just applied for stabilizing sprints. Don’t commit to anything in a release that the team doesn’t understand or know how to build. If there is a ton of risk and uncertainty coming out of release planning, Scrum isn’t going to necessarily help us delivery reliably against the release objectives.
So how do we get this level of clarity on the release level backlog? If we apply the same concept at the release level we applied at the sprint level, we’d have to suggest that the team has to do spike work BEFORE they get to the release planning event. That means that we need to have some idea of what’s in the upcoming release, and the unknowns associated with that release, BEFORE the current release is even done. If we are doing 3 month releases, I’m looking for high-level planning somewhere between 3 to 6 months out.
That’s NOT Agile!?