Organizational health for Agile adoption: Key traits of Customer Champions

This post is a continuation of my series on adopting healthy practices that enable an organization to make the agile transformation. You can read the first five parts of this series here:

Part I: Introduction
Part II: Vision and Risk
Part III: Backlog Management
Part IV: Key Players
Part V: Sprint Execution

In Part IV (Key Players) we discussed that an important role that someone on projects at your organization must play is that of an advocate for the customer, or a Customer Champion. People staffed in this role, whether the sole proprietor of a startup or several individuals in a hierarchically-organized corporation have some initial traits that make them effective in this role regardless of the situation, as well as a difficult set of opposing concerns to contend with.

First of all, it’s important that a Customer Champion who has experience in the market or job that the agile project is targeting realizes that once they become an advocate for the customer, they are no longer the customer. Take for instance a fictional project that is building software to make it easier to track the quality of manufacturing automotive parts. A person in this role may have had 20 years experience building cars, and consider themselves experts in the field. But for the past 6 months, they have been working behind a desk, and when they were on the job, their idea of quality concerns were those which made it hard for them to get quality to sign off on parts so they could continue building more. Someone responsible for quality metrics themselves may have had an entirely different set of problems that are more important to them. Regardless, a good Customer Champion has to realize that any experience they bring to the table in a field related to the target customer of a project is from a single blind perspective, and they will have a tendency to rely on their experience to guide their interpretation of the customer’s concerns. Instead, they should focus on gathering information about disciplines other than those that they are familiar with to make the most educated decisions on what to build. In other words, work hard learning what you don’t know and what other related job role’s problems are – you’re already well in tune with those closest to the vest.

Secondly, it’s common that a Customer Champion is helping to design a “new version” of an existing software solution, or to automate something that used to be a manual process. It’s rare that Customer Champions have a background in software user interface design, business analysis, and studying flow-of-work, but many organizations hand over the reigns for making these decisions to exactly these individuals. Instead, the Customer Champion should realize that the one chance the organization gets to truly make a significant impact on the way the target customer’s problem is solved is at the beginning – and the only way to make a radical change to existing problems is to think completely outside of the box and not use any existing program as a blueprint. This can’t happen if the only identified solutions are those that solve problems with the existing version of the solution.

The opposing concerns I mentioned before that are hard to deal with are the pull of what a customer says are their primary problems against what their primary problems really are. A good Customer Champion will setup discussion forums, make phone calls, send polls, and gather questionnaires to determine how their customer feels about the relative importance of the problems they are trying to solve. Not doing these things is a recipe for disaster as it is. But even if a mountain of data is gathered directly from the mouth of the users that will buy your product or service, the real work hasn’t really started. Next this data must be analyzed and patterns must be recognized. And even after these patterns are identified, you must dig deeper to find the root cause, which is many times not obvious.

The important thing I’m getting at here is that identifying the biggest problems of your target user is a tough job. So many people in this role I encounter at organizations spend their time driving the look and feel of the user interface, coming up with ways to cram up-and-coming industry technologies into the product line or service, and frankly, designing a clone in functionality of the existing system when all of these things are the least value they bring to the table. Top tier software developers are much better at making technology tradeoff decisions, coming up with novel solutions through technology to problems, and designing effective and usable interfaces. And graphic designers are much better at making things look pretty than customer advocates or software developers in most cases. The one thing developers absolutely cannot do however is determine with certainty what the customers’ problems are in the first place. It’s this information that really determines whether you have a chance of being successful. A project may ship fast, look good, scream quality, and have all the right market positioning – but without making the target customer base more successful in their work, it’s just a pretty solution built on an expensive, volatile foundation waiting for a more agile competitor to knock it down.

Organizational health for Agile adoption: Sprint Execution

This post is a continuation of my series on adopting healthy practices that enable an organization to make the agile transformation. You can read the first four parts of this series here:

Part I: Introduction
Part II: Vision and Risk
Part III: Backlog Management
Part IV: Key Players

In Part III (Backlog Management), we talked about how features are placed on a backlog to set the vision for a project prior to starting the first sprint. Now that we know the key players that need to be represented by one or more people as described in the previous post, let’s describe the flow of executing a sprint in detail. Recall from the Backlog Management post that a sprint is a fixed period of time during which the development team works on features before “coming up for air” (review and re-prioritization by the business). Below is a diagram depicting the steps in execution of a typical sprint. I’ll describe each step below the diagram.

Agile Sprint Planning and Execution Workflow

1. Analyze market, set vision, scope and/or mockup initial features

During this step, the Customer Champions and/or Competitive Analysts, along with Visionaries and major stakeholders on the project (or maybe one guy in a startup!) collaborate to set the vision for the project. The vision may be “create the best recipe management software on the web”, “solve world hunger”, “solve the problem of sharing documents between manufacturers and suppliers”, or “create a culture for development of web 2.0 products in the pharmaceutical industry”. There are hundreds of business books out there to help you craft mission statements or culture mantras for your business or project so I’ll leave that as an exercise for the reader. At some point however, a critical decision must be made as to what level of analysis will be done prior to involving development at this step. On a very agile project with pressure to aggressively be first to market, it is easy at this step to come up with only a list of features and add these to the backlog as described in Business Backlog Prioritization in the Backlog Management post. Then the highest prioritized features are assigned to developers (as I’ll describe soon in more detail) and they collaborate with the customer representatives to come up with user interface or use case designs.

In my experience however, I find you will actually get to market faster with less refactoring churn if you make a “throwaway mockup” of the entire project/product at this point. The goal here is to select a single developer or designer (the Usability Expert from the Key Players post) and work with the Customer Champions to create mockups for every screen in the system before the first sprint begins. Doing this up front has several benefits. First, it forces the customer representatives to think through all the features they want to build in more detail than the simplified list a backlog provides. This will aid in getting a feel for the total size of the project during planning. Secondly, it gives developers a general roadmap to look at when designing features in a sprint where they can identify patterns earlier to reduce refactoring. For example, lets say that your application has a feature that allows users to attach photos to their profile through a web page. The developers implement this feature in sprint 1 and the business is happy. In sprint 5, the feature “add ingredients” is now designed and it comes to light that photos should also be attached to ingredients. The developers may have implemented the photo attachment feature in a fashion where it is “hardcoded” to only support a profile, and they now need to refactor the code from sprint 1 to handle multiple types of objects to which an image can be attached. Had these photo attachment features been available through a throwaway mockup prior to starting the project, the developers could have analyzed the throwaways to identify the pattern of photo attachments when designing the first sprint’s feature and avoided the refactoring.

The important thing to note if doing a throwaway mockup however is that it is simply that – a throwaway. The final designed screens do not need to have the same style, layout, and even navigation. Think of it as a user interface brainstorming and business analysis scoping tool. It’s purpose is to guide the design of features in a sprint without complete blindness as to opportunities for reuse between features in future sprints, without spending a ton of time doing “real” user interface mockups. In some projects this may not be feasible because the business doesn’t want to spend the time it needs to really scope out everything up front. Another common concern is how to utilize developers while this is going on. I’d suggest they spend their time researching, prototyping, and learning about the technologies that will be used regardless of the features that will be implemented. The time needed to decide on exception handling, data access, security, and other approaches at this point is well spent. A Development Lead can also analyze the throwaway mockup when available to help them make the best technology selections.

2. Reprioritize

Here the Customer Champions shuffle the backlog as described in the Backlog Management post and put the most important features to them on top.

3. Highest Priority Features

Next the Development Lead and Customer Champions agree on the features that will be done next and select a number based on the available developers in the team (I suggest one per developer) and take them off the Product Backlog, adding them to the current Sprint Backlog. At this point, the items in the product backlog are “free” so to speak, and available for the Customer Champions to reprioritize, add to, and remove from while the sprint backlog is being worked on by the development team. I’d suggest managing the sprint backlog in a tool or spreadsheet that can be filled out with the following information about each feature being worked on in the sprint as it progresses:

• Feature name
• Assigned developer
• Status (“Designing, Implementing, Testing, Complete” at a minimum – more if you need to track at a finer granularity)
• Estimate

4. Collaborate to Produce Mockups, Business Rules & Ubiquitous Language AND 5. Assign and Estimate Features

During this step the goal is to have collaboration occur between developers and Customer Champions resulting in agreement on terminology that describes the domain of the problems the software is trying to solve. This happens through discussions, meetings, and online collaboration as necessary. The resulting terminology is the Ubiquitous Language as described in Domain Driven Design.

the Development Lead and other developers meet to assign one feature from the sprint backlog to each developer. At this point, there are two approaches I’ve seen taken when doing the design and each have their positives and negatives.

The first approach to agile design executes step 5 before step 4 and involves each developer taking ownership for the entire process of building that feature from analysis all the way until it is handed off to QA. This requires developers who are very talented, self starters, have solid experience in usability, object oriented design, and an ability to identify opportunities for reuse. Be realistic about your confidence in the equal talents of all of your resources before taking this route. In this approach, each single developer collaborates with the Customer Champions to come up with user interface mockups (in a system with a UI) or system use descriptions (in a service or headless system) for the feature they are responsible for. These collaborations can result in mockups, napkin drawings, use case documents, prototype APIs, eXtreme Programming story cards, or any other method of tracking. It’s important however that no matter what approach is used, the following artifacts of each case are at initially captured as part of the design:

1. How does the user (or system) get to this feature – this only has to describe what’s known in this sprint, and not all possible navigation opportunities at this point. Use the throwaway mockup as a guide if you made one. At least think about it!
2. What validation is performed when the user (or system) interacts with this feature – Take a crack at this before coding, even if you will (inevitably) uncover more during implementation.
3. What are the “default values” that fields or API parameters are set to when the user (or system) interacts with this feature
4. What job role (or system role) do the people (or systems) who interact with this feature belong to – Again, take a crack at this up front, it’s many times easier to design for security permissions at the beginning than later.
5. What actions that the user (or system) can take when using this feature change the state of other aspects of the feature – For example, “when you select this combo box, it filters this other one” or “if you pass true to this parameter, the data returned is filtered in this way”.

Remember that even though most agile books will tell you that use cases are outdated, much of the same information captured by them is still important to identify at this step when using other methods. If you choose to use another methodology to gather the needed requirements, great – the better job you do of capturing details, the better estimate you will have.

The benefit to this first approach (letting each developer own the entire design process for their assigned features) is that there is a potential for increased productivity if multiple Customer Champions are available. The downside is that developers may not identify opportunities for reuse and learn each other’s subset of the Ubiquitous Language at the same time.

The second approach to agile design executes step 4 before step 5 and involves the best developers on the team being the primary analysts and designers of every feature. This has the benefit of letting the most talented developers shine in creating the user interface and primary models of the software. In most teams, you’ll find a higher quality of code and increased consistency from feature to feature if you go this route. The downside to this approach is that the junior developers not involved in design of the features have to find something else to do. In reality this is rarely an issue unless your developers write 100% bug-free code as they will be fixing bugs from their prior features while the next highest priority features on the backlog are being designed. Businesses can get paranoid about having these junior developers in “feature limbo” during this period, however the pain incurred by having a junior developer take on the design of a feature that is over their head (and in isolation from more senior developers) and resulting in a bad design is enough to eradicate this after a couple occurrences.

You may have read about YAGNI – “don’t build it until you need it” is the mantra here. And while I both love this article and agree with it’s premise completely, don’t mistake YAGNI for something you are going to need, but is just coming in a future sprint. If it’s on the backlog but just not assigned to the current sprint, there’s still a good chance “you’re gonna need it”. If it’s not, don’t spend time at all on it. When in doubt, work with the business to determine the rank of a future feature that may have an impact on your design of the current one – sometimes it may be determined that it’s a “nice to have” feature that is far off in the distance and won’t even be implemented in the current release.

As soon as developers have had initial collaborations with the business and have a scope for the feature, they should update the estimate in the sprint backlog with a rough estimate. In a future post I’ll describe the sprint planning and management process by which a Development Lead or Schedule Facilitator (described in the Backlog Management post) can shift features to future sprints to have things line up more easily. As soon as the developer designing the feature has produced all artifacts other than the code (mockups, story cards, use cases etc.) they should update the estimate with a more detailed (and more accurate) value. This estimate should include the time they think will be needed to write unit tests to the level of coverage agreed upon by the team but not include the time needed for any additional testing by QA personnel. I’ll talk more about agile estimation in a future post.

6. Implement Features

During this step, developers do what they do best – design and write code. The activities that take place here are outside of the scope of this discussion, but an important aspect of managing the sprint is to track the status of each feature as it progresses through being built. As the sprint progresses, developers can update the estimate or provide a “time remaining” on regular intervals, but this is up to your organization to decide. Remember that a big part of the agile process is to eliminate waste. If status is being gathered just to put pressure on developers, you probably don’t have the right people to do agile development in the first place. If you are going to enable agile development at an organization, you should be using professional resources that do not need to be told what to do.

7. Test and Validate Implemented Features

Whenever a feature is done being coded and unit tested by a developer, it’s status should be set to “Testing” and QA personnel should be notified that this feature is ready to test. At this point the developer who implemented the feature should organize a collaboration with QA to communicate what’s been implemented in detail, as there is a good chance that what was designed going into the sprint has had some changes since the beginning. Don’t bother updating the design documents – the tests written here are your documentation for what is currently implemented. Once this feature handoff occurs, that developer picks the next highest item off the product backlog to work on. The goal is for each developer to finish at least one feature per sprint to have something to show at the end, but depending on the length of sprints and complexity of features, there may be several features implemented per developer per sprint.

The QA testing process is very much specific to your technologies and organization. Regardless, developers should be prepared to have to balance fixing bugs found in previously completed features with developing newly assigned ones. This underlines the importance of developers not handing features off to QA until they are really well unit tested and complete.

8. Mark Features as Accepted for Sprint Review

When features have been tested by QA, and no bugs that must be fixed before that feature can be shipped exist, the sprint backlog’s status for that feature should be updated to “Accepted”.

9. Present and Gather Feedback on Tested Features

Near the end of a sprint, developers should coordinate with one another to try to make sure that any new features being worked on do not break the functionality that was already “Accepted” in prior features for that sprint.  There will be cases where to move forward existing features must be broken, but if these changes cannot be made in a way that allows the original features to still be presented to the business, that feature’s status should be rolled back from “Accepted” to “Implementing”.

After developers have coordinated their efforts and the last day of the sprint concludes, a meeting should occur in which developers demonstrate the implemented features to the business. At this meeting any changes or enhancements that are requested should be added to the product backlog and prioritized along with whatever else exists at the top of the backlog.

10. Combine Sprint Feedback with Customer Feedback and Market Evaluation

Here is where the agile process really shines. At this point, we should have some shippable features, a list of changes or enhancements (potentially) to those features gathered in the sprint presentation meeting, and an updated evaluation of the market (our competitors and users’ needs). The business can then decide again what’s most important. Has the market or users’ needs changed such that existing planned features are lower priority or no longer necessary? Is there a new market need that is more important? Are changes found during the sprint meeting the most important ones to build next? Whatever the answer, the business is armed now with assets and options for re-aligning the efforts of the development team, perhaps radically if needed, without disruption to the quality of their work or the schedule. When the Customer Representatives arrive at their updated prioritization of the backlog, the process continues again at “scope and/or mockup initial features” from step 1. In future posts I’ll discuss planning, estimation, important traits to look for in the key roles to execute this process smoothly, tools to use, and tips to successful sprint execution.

Experiment Driven Design

Much like we write tests to assert that the code is really working the way it should, Nathaniel Talbott thinks we should be able to write experiments to provide us with facts that assert the usage of a feature of software is really valid. This is a novel concept, and he discusses it briefly in this interview over at InfoQ. It’s a common occurrence that customer representatives in a software project sometimes state with absolute certainty the importance of a feature when they really have no facts to back those statements up. EDD is meant to give customer reps and developers a common set of tools to measure the importance of usage of features before taking the time to fully build them. Assaf Arkin over at labnotes.org created a prototype framework for creating EDD experiments. You can check it out here. This implementation is currently in ruby, but I can see the .NET community adopting it (as they do with so many other open source frameworks) in the near future. However you can start using the framework to validate business assertions today even if the implementation of your project is in .NET, the experiments will just be in ruby.

Organizational health for Agile adoption: Key Players

This post is a continuation of my series on adopting healthy practices that enable an organization to make the agile transformation. You can read the first three parts of this series here:

Part I: Introduction
Part II: Vision and Risk
Part III: Backlog Management

Although I’m trying to emphasize in these posts that companies well staffed for agile software projects don’t require any exact rules to follow about how people are organized, in practice there are some key skills that make it easier and some of these work best when they are not the same person.

First of all, you absolutely must have at least one resource I’ll refer to as the Customer Champion. This person does not have to know how to build software, can have a horrible eye for usability and aesthetics, and have no concept whatsoever for technology. In fact, it can be easier to work with them the less they try to aid in the design of the software itself. The Customer Champion is the person who understands (hopefully) what problems the customer has and of those, which impact them the most as far as time wasted, wasted money, wasted efficiency, or perhaps missed opportunities. Note that the Customer Champion does not have to know how to solve these problems! Sure they may have some ideas, and those ideas may be very helpful in coming up with the eventual solution, but their value is in communicating in the language of the business what the current processes are in the domain that the software is trying to solve and what areas for improvement exist.

Secondly, and this is important both at a product company who is trying to lead a market with their product as well as an organization that is trying to be the best in its line of work (for which software is only an internal part of the business’ value offering overall) we need someone who is a Competition Analyst. This can be the same person as the Customer Champion, but in many cases, the person asked to do both jobs is really only good at one of these skills and fails horribly at the other. If this is the case at your organization, I’d recommend making time for this resource to invest in training to get the knowledge needed to do both jobs effectively or hiring a second resource who can do it. Also do this before you start your project – it sounds ridiculous to say it but there, I have. You can solve a current set of customer problems with software but be ignorant of the competition and fail spectacularly. You can also position yourself against the market leader but fail to enable the minimal functionality needed for the customer to do their work and suffer the same fate. The Competition Analyst must find and evaluate upcoming and existing companies that either do or have the opportunity to start doing work that interferes with the success of the company, and aid in coming up with solutions to the problems identified by the Customer Champion to position the company as the market leader.

I’m not going to dive into all the approaches and processes the above two roles can use to be effective, that’s an entire topic in and of itself. The important thing is that during the design and implementation of each feature on an agile software project, the organization must be driven by the reality of the customer’s problems as well as the strategy that will produce results keeping the company at the forefront of offering value to its users. Again, this doesn’t matter if the project is for a consulting engagement for which the client’s users are the “market” or the customers of a product company. I’ve also found that while in a startup environment a brilliant person who knows a problem domain from experience and is also a great software designer can do the jobs of these roles and the ones I’m about to describe, in any other organization that wishes to respond with agility to shifts in customer need and market competition it’s much more effective with the business and development represented by at least two separate people.

Inevitably, we also will not produce working software of much value without a skilled Development Lead. This resource can have a title like Solution Architect, Project Lead, Technical Lead, or Lead Developer but if this person is truly fulfilling this role, they must have several skills. First, this person needs to be able to talk in the language of the business resources. The Development Lead will be the primary person with responsibility for the overall technical solution, and so to design an appropriate implementation, it’s critical that they are able to talk to business resources in their words and avoiding “programmer lingo” as much as possible. The Development Lead also needs to understand domain modeling as described by Eric Evans in his book, Domain Driven Design. Now I’m not saying they need to use the exact patterns in his book, but they need to understand how to go about modeling software that represents objects in the context of the domain, using the language of business users, and following industry practices that take more research and dedication than watching videos on how to use their development tools of choice. This book happens to be, in my opinion, one of the most effective pieces of current literature in conveying how to go about this – though it will lead to research in other topics (Martin Fowler, etc.) and by no means negate the need to still tediously research technologies and tools. The Development Lead does not need to be the resource most skilled at math or writing algorithms, but they do need to be able to select technologies to use and be versed enough in their use that they can lead developers in translating the models of the domain into real working software. If they are also the best programmer on the team this is a bonus from a resource cost standpoint, but will require a delicate balance to produce both solid designs and working code since their efficiency in both disciplines is effectively halved.

If the software will have a user interface of any sort, it’s important also that your organization or project structure has access to a resource who is a Usability Expert. I’ve found that many companies leave the design of the user interface up to a business person, and this is often a mistake. First of all, user interface patterns and control/widget availability is changing all the time, and business resources often don’t have time in their schedule to be keeping up on all of the latest trends. There’s no denying that an effective user interface that is also modern reflects confidence in software. I also find that business resources sometimes design the user interface out of fear that the developers will leave something out. In reality, if business resources are doing their job effectively at conveying the domain and its problems to the developer, there are no grounds for this paranoia. Secondly, designing an effective user interface usually requires looking at all the features of the software as a whole, and then “zooming in” to specific features to identify patterns for common navigation, controls, and other capabilities and this is a skill more natural to a Software Architect type of resource than a business user. Though a business resource may understand the “big picture” of the business better than a development one, software inherently only models a subset of this big picture of the overall domain, and this subset is where the patterns and opportunities for consistency need to be focused. The person who does this job could be the Development Lead, a graphic designer with usability experience, or a software developer – whoever is most skilled at designing user interfaces (and not just pretty pictures – usable pretty pictures). While it is possible to have each developer on a team design the user interface for each feature assigned to them (I’ll discuss this more when we get to posts about UI design) to enforce consistency and enable the highest effectiveness it still makes sense to have one resource that will have input on setting the standards for and reviewing all user interface mockups.

On any project of a non-trivial size, it’s essential to have a dedicated Quality Assurance (QA) Lead. Even on projects where every developer is great at writing unit tests, and the quality of the code appears to be high, the benefits of a resource that is responsible for championing the overall quality of deliverables and providing a “second set of eyes” is another recipe needed for sustainable delivery of successful projects. This person should, just like the Development Lead, be very effective at talking in the language of the business resources. A resource that can use software to create automated specific case, integration, performance, and user acceptance tests is helpful as well, but even more important is the ability to validate that the software satisfies agreed upon functionality and to identify paths through the software that are not being adequately tested already by developers.

When the number of business and development resources on a project reach a certain scale, it becomes useful to have a Schedule Facilitator. This resource might have a title like Project Manager, Software Development Manager, or SCRUM master – or it’s duties may be fulfilled by the Development Lead. On a small to medium-size project (don’t ask me for a number), if agile processes are working, the development and business resources should be communicating and executing autonomously and some of the benefits introduced by a Schedule Facilitator will already be in place. However at a certain scale, the need to coordinate the management of the backlog, availability of resources, and deal with the pressure to meet milestone deliverable dates requires time that may detract from a solid design if these duties are dually burdened by an existing critical development resource. The Schedule Facilitator should understand the agile processes agreed upon at the organization, and be an advocate for both business resources and development resources equally to enable transparency and fairness in the organization. When this resource begins to favor (or shield) one discipline of the organization from another, it leads to miscommunication, bad feelings, and ultimately unsustainably low morale. It is critical that this resource respect the value of business and development resources equally, and do whatever they can to get the right people in the room to continue making progress on projects. It’s also important for them to yield business and technical feature negotiation to those best equipped to make the decisions. Remember, the goal of this resource is to facilitate, not “design” or “manage”.

One thing you may be thinking after reading this post is how it’s “not the agile I’ve read about”. You may have been led to believe an agile project starts with a napkin, continues with code, adds some tests, and iterates with refactoring. Though I will get to all of these very useful practices, in reality the “off the shelf” agile community-driven practices we use to arrive at implementable code still exist within the larger framework of the business and its need to sustain delivery of value. A healthy organization is one that has an environment in which the problems that are solved with software are being driven by those with the problems, and developers are not the right people to identify those. Conversely, a healthy organization needs skilled development resources who are free to use their skills to enable modern solutions to the problems without a design being dictated to them by business resources. It’s this explicit boundary, and a need to continually reinforce the roles and their importance in this mutually beneficial relationship, that enables organizations to continually, predictably, and smoothly deliver value over several releases of projects. If you are going to invest the time and money to build something, it’s important to lay a solid foundation not just for the software’s technical architecture, but for the dynamics of the people who will go about making it happen.

Organizational health for Agile adoption: Backlog Management

This post is a continuation of my series on adopting healthy practices that enable an organization to make the agile transformation. You can read the first two parts of this series here:

Part I: Introduction
Part II: Vision and Risk

Depending on which statistic you believe, many researchers in the field of software development agree that the majority of projects that fail to make it to market are those with insufficiently detailed requirements. Though I agree completely with this, many companies also fail to avoid an even more troublesome, but often overlooked issue – that of building either too many or the wrong features. Traditional waterfall based development processes spend a large amount of time creating requirement specifications and detailed design documents before any code is written. In the time between the start of a project and when coding begins, those responsible for determining the functionality of features often understand the requirements better than they did at the beginning of the project. This results in a last minute rush to cram in these requirements after the design has been done, which results in an impact on the design for which time is usually not given to developers to adequately account for them. On top of all of this, the needs of the customer (whether internal or external) may have changed or the market against which features are targeted may have shifted focus that now requires a different priority in order to remain in step with the customer or client’s needs.

These problems which I find especially prevalent in large projects with many features, are a big reason why the backlog concept for agile projects is so effective. A backlog is essentially a queue in which features are placed and then re-ordered that drives the software development team’s activities. To kickoff a new project, the visionaries of the project and those most in tune with the customer’s needs come up with an initial set of features. The goal here is not to design anything in detail, but to try to break functionality into logical features that could be stated to the customer as well as software developers (you could read these on a list of “things it can do” about your product or project in a brochure). It will be a challenge when starting out to determine what is too granular to be a feature versus too broad. If building a dashboard application of some sort for example, if all we are building are the widgets that go on the dashboard themselves, each feature of the dashboard could be a single widget. If building the framework of the dashboard itself (managing which widgets are on the dashboard, and where they are located for example), this would be another feature. Too broad of a feature would probably be “dashboard page” if this includes the dashboard and widgets together. Don’t worry however if you have some features defined too broadly at first – the process provides a means to divide up the features easily into smaller features as the project progresses.

Below is an example of a backlog created by a fictional CEO, two field market analysts, a customer representative, and a product manager during some initial meetings or via collaboration on a document. At a smaller company this may be the marketing rep and the product manager only. In a consulting engagement it may be the client project sponsor, consultant, and possibly an internal end user. It’s not important for this topic to dictate what tool is used to come up with this list or how it is stored. The important thing is to get the right people together make this initial list. If anyone is far removed from the customer, it may make sense to have them review the resulting initial list instead of participating in its creation. It is crucial that those involved do not concern themselves with time to market, cost to develop, or priority at all during step. Prioritization will be done later, this is strictly a brainstorming activity.

Recipe Manager Product – Initial Backlog

Here we’ve got a backlog with CRUD (create, read, update, delete) features around a few objects; namely recipes, ingredients, and meals. We’ve also got the concept of a dashboard, with a couple of widgets, and a search. It’s a good idea at this point to circulate the list to the original attendees of the meeting and let it stew for a while. Even though it’s important not to go into more detail at this phase, it is very important to establish high level scope. Identifying the major pieces up front makes planning much easier.

Once the backlog has been tossed around in the minds of the attendees for a while, and any new items have been added to it, it’s time to have our initial prioritization of the backlog. This should be done in a group, as it’s easier to get out reasons for prioritization in a group and avoids having to merge the reprioritized lists. Below is the reprioritized list. The features that are most important are at the top of the list, with those that are less important at the bottom. On a real project, this list may be over 50 items long. We’ll refer to this process as Business Backlog Prioritization.

Business Prioritized Backlog

You’ll note that the planners of the vision of the project determined that the dashboard features were most important to them. Often the most visible or sexy features will be determined as most important at this point. In a real project, the top 10-20 features coming from the visionaries help the development team to see what is most important to the business when they perform their own prioritization.

Now the development manager or team lead should meet with whoever else from their team makes sense to review the initial backlog. This could be the entire development team, or just an architect depending on time constraints and confidence in resources. We’ll call this the Development Backlog Prioritization. The goal of this prioritization and review is twofold. First, the team should try to identify any features that will obviously take longer than a sprint, and break them up into smaller features. A sprint is a fixed time period during which development of a subset of the features on the backlog are implemented, I will talk much more about them in future posts. Each development team determines what the length of their sprints should be, typically this can be anywhere from a week to 30 days. The second goal of this prioritization is to determine how realistic implementing the top items on the list are based on whether it is possible to identify any dependencies of those features further down the list that should be bumped up.

Upon looking at the backlog, the developers determine that implementing the dashboard and accompanying widgets without authentication will be difficult since much of the code tied to laying out widgets on the screen requires identifying the currently logged in user. They also feel that implementing the recipes and ingredients features first will make it possible to create widgets that pull real data. It is in this meeting as well, or a set of meetings, where the development team may be asked to create WAGs (wild-ass-guesses) for all of the features on the backlog to get a vague feel for the overall cost of the project. I’ll reserve further discussion of WAGs of the backlog for a future post on planning. Anyway, they reprioritize the list again and arrive at this backlog:

Development Prioritized Backlog

At this point the backlog is presented back to the visionaries, and they look for ways to still get the features they want earlier, hopefully understanding the technical constraints as explained by the development team representative (software development manager, team lead etc.). Finally, a compromise is reached. User authentication will be implemented but without adding and editing users initially, listing of ingredients will be implemented but without adding, editing, and deleting, and the dashboard will then be created with the initial ingredients widget. The Approved Backlog Pending Assignment is below:

Approved Backlog Pending Assignment for Sprint #1

At this point the software development manager or development lead meets with the team. Each developer available selects a feature from the top items in the backlog that is either most interesting to them (a challenge) or leverages a specific skill they may be advanced in until everyone on the team has been assigned a single feature. In this case, our fictional team has 4 developers, so the first four features are scheduled for implementation in sprint #1. The remaining backlog, after the features for sprint 1 have been assigned, looks like this:

Free Backlog

At this point the visionaries are free to add, remove, and reprioritize items in the backlog until sprint #1 is complete. Prior to starting sprint #2, the visionaries must finalize their priorities and collaborate once again with the development team as described in Business Initial Backlog Prioritization. It’s important to note that a backlog should not be looked at as a list to complete. It may have items on it for the entire life of your company or product. The backlog is simply a queue that steers the direction of the project and allows development to respond with agility to shifts in the marketplace, initially missed dependencies, or changed needs of the customer. It also becomes a vital brainstorming and planning tool for capturing future ideas before they can be completely designed.

Future posts will discuss how the development team goes about analyzing, tracking, and estimating the features assigned within a sprint. The important takeaway here is that visionaries may not change the direction of the development team’s current activities until those features are complete. Using a backlog gives those responsible for steering the ship regular intervals at which to control the direction of progress, while at the same time encouraging collaboration with developers in creating a realistic, deliverable set of tasks that can be revisited and measured at regular intervals.

Organizational health for Agile adoption: Vision and Risk

I’d like to continue the discussion about creating a healthy organizational environment for adopting agile practices in my prior post by starting at the top. Somewhere at the top of most organizations, including startups, are the executives or leaders responsible for steering the ship. At some older companies, this can be someone who worked hard for years and finally was promoted simply due to hard work, dedication, and frankly, seniority. To be blunt, however, for a company to truly create an agile culture takes leaders who are willing to do a lot more work than they’ve already done.The folks (or person) at the top that I see run companies perfectly aligned with the right elements for agile success are those who are truly visionary leaders with a penchant for semi-calculated risk. A great set of leaders is looking out into the future, and not just looking at what successful companies are doing, but seeing the things they are not doing and sometimes, betting a good part of their future on an unproven idea.

A good recent example of two companies that exemplify this are Apple and Microsoft. Microsoft’s Steve Ballmer recently announced that the company would be going “all in” on cloud computing. Put simply, this means that Microsoft will shift towards applications that use the internet for storage and processing power. Microsoft dismissed the importance of the Internet early in its inception, and came around with leading the way through many technologies including the first (albeit standards-flaccid) really powerful web browser with Internet Explorer version 4. However the company still spent most of its mindshare, research, and cultural energy on the world of PCs and a world with market dominance around Windows, and only began to really start building some true web applications recently, and under pressure. Steve Ballmer is the correct and perfect person, a leader at the top, to communicate this message of change to his company and users of their products. It’s his responsibility (though it will funnel down to and be reinforced by his reports) to be crystal clear about the direction of the company and what employees need to put their energy into to make his vision become reality. Though this represents a big shift (if they really do it) for a company like Microsoft, it’s still a calculated one. Google has been Steve’s #1 target for years and for good reason – they make a killing off apps hosted in the cloud. Microsoft wants a big part of that pie. The problem is, they are late to the game. They are also doing something innovative in the company’s eyes, but not in the rest of the world’s (other than those who turn a blind eye to all things non-Microsoft). Sure there are Microsoft developers that will be elated to pick up new skills that let them build the same types of apps available to other developers with better tool support and Redmond’s message of market viability of the shift behind them. But for end users it represents more of the same – a saturation of already existing, successful alternatives, with no real risk or innovation. This announcement will not be met with much resistance, surprise, or pushback. It really doesn’t mean every Microsoft technology-driven project will have to be “cloud enabled” to move forward, and there’s already a history of prior art in existing apps that show how to get there.

On the other hand, we have the Apple iPad announcement. Here is an idea that was tried years ago and failed, and is being attempted again with no recent success. The market is untapped, the potential may or may not exist, and if you do some even casual searching to see what folks think about it, you see lovers and haters. This represents a disruptive announcement that could change the game in computing, or fail horrendously. The difference between this and Microsoft’s announcement is that the iPad represents innovation that involves only semi-calculated risk. No one has been successful in a tablet-specific hardware market yet (don’t count the small number of windows-enabled tablet PCs, which are full PCs). It’s obvious that Apple is going after the netbook market (everyday users who don’t need the full capabilities of a computer but for which a phone is too restrictive), however there are no statistics about existing users that have already bought tablets that prove it will be a success. When someone like Steve Jobs makes an announcement like this, you can bet that Apple as a company is energized. They see themselves as risk takers and visionaries themselves, and will carry the message of change forward through the company with fire and conviction. This is how “buzz” is truly created in an industry.

You may think by reading this that I’m an Apple fanboy or hate Microsoft, but nothing could be further from the truth (we have Macs and PCs in my home, and I write Microsoft and Ruby for a living). I simply used these examples because they are high profile companies with announcements that support the topic. The important takeaway here is that choosing to go agile at a company takes courage. And that doesn’t mean courage by the project manager for the “pilot project” that will “try out” agile at a company. It means that the executive leadership at a company must foster a culture that encourages semi-calculated risk to fulfill vision. I have had the privilege of having a development manager who was a mentor to me in business for the first 12 years of my career at several companies, and he taught me something I’ll never forget. He said that with great risks come great rewards. Just like someone who plays the stock market, if you go conservative, all you can expect is mediocre results. And if you track the entire market, you will go down with the entire market. But if you have the courage to take risks, not only will you create the opportunity for great reward, but you shine a light down to the rest of your employees, partners, and customers that you are an important company to watch that will lead, and not simply ride the wave of prior successes to its eventual crash.

It’s unfortunate that due to how difficult the economy is right now, so many companies have let fear cripple their ability to innovate. Most don’t know it, but the majority of the economic growth in the United States and many developing countries is from small business, not established and large corporations. The ironic, true risk of Microsoft’s decision (or any company that chooses “change through emulation”), is that by the time the company has demonstrated leadership in cloud, the game may well have been changed. Rather energy could have been put into changing the game themselves. Even if a choice to innovate and take a risk results in failure, it almost never fails to generate buzz. And buzz is what gets employees excited, end users passionate, and creates the next market leader. If you have the courage, and are willing to see minor failures through to their eventual big success, your company can also lead the market by adopting agile practices.

Organizational health for Agile adoption: Introduction

I’ve been doing non-waterfall development for the past 10 years of my career in one form or another, and though Agile/SCRUM/XP practices are almost assumed at this point, I still see many organizations that need help with making it work. The agile transformation usually starts with a well-intentioned individual in a middle management or lead developer position who tries to set it forth upon an organization with promises of faster time-to-market, less documentation, and golden parachutes for every collaborator within a small project as a pilot. In reality however, really making agile methods work requires adjustment and a willingness for self improvement of every role within an organization. There are many companies out there that offer agile coaching, but I see too many of them working primarily with a single project (or even limited to specific roles on that project) and not really get buy-in from everyone who needs to take part in the change.

I’d like to use this post to begin a series on healthy practices to establish an environment for taking advantage of agile methods within your company. I’ll do a post on each role within most companies, and discuss some important considerations for that role and how their actions should (and should not!) effect others in the process. This is based on my real-world experience and the practices I’ve found to work through applying many of the books out there on the subject. In this, my advice is far from the “de facto standard” of which I don’t think exists at this point. To start however, I’d like to state some underlying realities I’ve encountered that any organization must be willing to face before even considering going down this route.

A common problem I see is touched on by the following post from Nick Malik for Inside Architecture on MSDN’s blogs. In Waterscrum vs. Scrummerfall, Nick (and the links within that I encourage you to read) addresses the fact that successful adoption of agile is very difficult to do without leadership from someone who’s done it before. I’ve bootstrapped agile into at least 3 companies over the years and every one of these was a learning experience for me. The hurdles along the way were enormous and had I known what I do now, it would have been easier for me to hire an expert. Unfortunately these can get pricey, and it’s difficult to evaluate how well of a job they will do when you don’t even know what you’re looking for! If you are thinking about doing agile, be honest with yourself about your level of experience. Remember, you are the guy they will all point fingers at if things go south…

Another problem with these “half-agile” adoptions is the cherry-picking of practices. Agile is supposed to be about empowering a talented development team to be autonomous, and flexible enough to respond to change that wasn’t anticipated. However this doesn’t mean the marketing, business analysis, or product management folks are off the hook for solid requirements. If the functional stakeholder in a project isn’t available to the team as much as needed and all the requirements aren’t known up front, this is a recipe for disaster (I’ve seen it happen). Similarly, if the requirements of a product are emerging as being developed but the development team doesn’t have appropriate tools to react to radical change in the codebase, things will fall apart. The tenets of agility have to be put into practice by everyone up or downstream of the deliverables of the project(s).

Lastly, taking on agile means agreement that there will be new skills that need to be learned by everyone in the organization. Though many of these are small, I sometimes see folks try to continue business as usual even after nodding their heads in agreement to the proposed changes in process. If you are going to truly transform the agility of a business, it takes willingness from every member to be diligent in spending time on these new activities and being professional about feedback, and honest with each other about what is and isn’t working. One of the most frustrating things is to see a key stakeholder throw up their hands to say “this process sucks, let’s abandon our agile efforts” who has been frustrated about a single aspect of the new process but didn’t speak up earlier. Agile methodologies are for people, and are supposed to encourage eliminating waste. If a step in your process isn’t adding value – don’t do it! If you’ve tried something and it’s not working, get in a room with stakeholders and formulate an alternative!

And this leads to my final note about overall agile hurdles – professional talent. In a company of mixed skill where some folks have personality conflicts with others and politics are prevalent, making agile methodologies work is very, very difficult. In these types of organizations it is much more sane to use a traditional waterall approach that keeps all the players honest about what is changing, who is responsible for what, and who is and is not delivering value. Micromanagement is a must-have for resources that are not self-starters, prone to defensiveness, and unwilling to work in teams. For companies that have really mature people however, even if these folks are not as technically adept as might be desired, the potential for success and smooth operation of projects is exponentially higher with agile methods. These teams will find it natural to get together and just “make it work” and will be energized by change instead of frightened by it. It’s these teams that are really ready to use agile methods. If you work on a project with resources of the first type however, your first priority should be creating a professional environment with solid resources before throwing a collaborative, group-driven process on them.

We’ve all worked with the super smart guy who can’t get along with anyone, and even the most personable folks have their bad days, projects, and situations – but in general project structures comprised primarily of people who see their work in a positive light and are enthusiastic about change are the ones that will see you clearly down the road towards agile transformation.

Results vs. rhetoric: upcoming books

I’ve thought for several years now about writing a book about real world experience versus classic management methodology and it looks like there are two new ones coming out that will be great to read before taking that step.

The first one is Getting Results the Agile Way, a book being written by J.D. Meier, a Microsoft employee. To shamelessly steal from his blog, he lists several things the book helps with:

• How to find work / life balance
• How to shift from tasks and activities to meaningful results and outcomes
• How to use stories and scenario-driven results to carve out value in your life
• How to overwhelm your challenges with fierce results
• How to defeat perfectionism
• How to avoid analysis paralysis and take action a simple story at a time
• How to find your flow state for more engaging work
• How to find your passion and purpose
• How to play to your strengths for more energy and better results
• How to conquer fear and avoid learned helplessness
• How to motivate yourself in ways that make you feel you can move mountains
• How to focus on what really counts
• How to prioritize more effectively
• How to create more value for yourself and others
• How to spend more time on what you want, and less time on what you don’t

 

Even more exciting (to me at least) is the upcoming book Rework by Jason Fried and David Heinemeier Hansson, founders of 37 signals, the startup that spurned ruby on rails. The book is all about challenging existing market-think. These two can be pretty radical but I always find some nuggets of gold in each speech I’ve seen online or blog post that comes out of their company. Here’s a few of their tenets:

• ASAP is poison
• Underdo the competition
• Meetings are toxic
• Fire the workaholics
• Emulate drug dealers
• Pick a fight
• Planning is guessing
• Inspiration is perishable