Keep It Lean And Agile ("KILAA")
PHP Magazin Print-Edition: 6.2008:
Agile Development mit Scrum
Keep It Lean And Agile ("KILAA")
More and more companies who until recently developed using more conservative methods such as Waterfall or RUP software, are now switching to lean and agile methods. Based on best practices agile processes offer so-called process frameworks, which can be customized individually, according to the respective needs of teams and projects. Sounds good so far but let’s examine the pros and cons, areas of usage, implementation, long-term application and, lastly, the differences to other methods such as CMMI. And when it’s all said and done: the final answer.
Software Development – A Historical Overview
The first software was developed a long, long time ago and it’s almost as long ago when the first project did NOT meet the planned deadline or budget requirements. (… “Wait, don’t I know this from somewhere?”...). Process models such as the Waterfall model or RUP were created in the hope that one could apply engineering practices to the development of software products. These are structured document-based processes that attempt to unify software-development processes. As is the case with industrial products, much time and energy is poured into the previous planning phase. Requirements are documented, long-term planning will ensure safety and extensive documents will guarantee that the entire process is transparent at all times. All of these steps appear to make sense, which is why they are implemented in many a company.
But where do we know that line from, the one that says that 50% of all projects were not concluded within the planned time or budget, in other words how did they fail? Oh, right: the Chaos Report by the Standish Group. We often like to use this one to demonstrate how the dilemma can be fixed. To be sure, tools can be helpful in simplifying and supporting you with daily tasks. But it is also true that tools do not improve processes.
But why do Projects Fail in spite of Structured Processes and Supportive Tools?
That’s easy: successful software development means successfully facing changes. This is because in contrast to many industrial products software products undergo frequent changes all the time. There is no such thing as software products without changes such as new requirements. That’s why an approach focusing on processes alone and attempting to create regular and repeatable actions is inadequate for software development.
One possible solution to this problem is an agile method. And although rules such as the Manifesto for Agile Software Development have been devised and other priorities identified, this does not necessarily mean you must choose between either conservative or agile approaches. Being agile means being flexible, putting people first and trusting the fact that an autonomous team will achieve more than a team that blindly adheres to rigid and stiff processes. Being agile does not mean chaotic processes or “anything goes”. We’ll see shortly how traditional and agile practices can be combined harmoniously.
Agile or lean processes for projects first originated in the automobile industry. Toyota was the first to implement such an agile process model and to this day they work according to this principle. This agile Toyota model was the basis for Scrum a.o. (see also below). In 1995 Chrysler’s own Kent Beck first introduced agile software-development processes in an internal pilot project and so created Extreme Programming. This pure XP process model for software technology is tailored according to individual needs.
The Scrum framework can easily be combined with XP but in contrast to popular opinion, it does not require you to use XP practices. In the Agile Manifesto advocates of both agile movements have summarized and documented the similarities of Scrum and XP:
- People and cooperation before processes and tools
- Functional software before extensive documentation
- Cooperation with clients before contractual negotiation
- Reaction to change before adherence to plans (source: http://agilemanifesto.org).
Figure 1: Established Agile Methods: An Overview
Scrum - An agile Method##
Scrum is an iterative and incremental process for product development and team organization. The so-called Scrum framework defines a collection of work techniques, structures, roles and methods, which support you in getting tasks done faster and at higher quality. Firstly, this is accomplished by a high degree of self-motivation because with Scrum the team decides autonomously which tasks are performed in which way Secondly, continuous evaluation generates optimal work practices, which in turn enable you to prioritize client requirements and to put them into iterative and fast action.
The term Scrum was first used in Rugby and means something like “keeping the ball in the game”, Andrew Scotland, Head of Development at BBC New Media Division gave a very astute description of Scrum at the JAOO 2005 conference in Aarhus, Denmark: Scrum is a simple "inspect and adapt" framework that has three roles, three ceremonies, and three artefacts designed to deliver working software in Sprints, usually 30-day iterations.
- Roles: Product Owner, Scrum Master, Team
- Ceremonies: Sprint Planning, Sprint Review and Daily Scrum Meeting
- Artefacts: Product Backlog, Sprint Backlog and Burn-down Chart
Scrum - In a Nutshell?
As mentioned earlier, Scrum contains 3 immediate, participatory roles. 1. The Product Owner identifies the specific requirements, prioritizes them and introduces them to the team. 2. The so-called Scrum Master manages the course of action and makes sure the team can work without interruption during an iteration (Sprint). 3. The team responsible for product development completes this triumvirate.
Unlike with other methods the team is not merely the entity that performs the task. By asking the Product Owner targeted questions the team first has to find out what exactly lies beneath the specific requirements. The team is not instructed how to i.e. build features but instead is responsible for the right implementation after understanding the specific problem. Besides these 3 there are other stakeholders, which act as observer and consultant for example.
The requirements are noted, maintained and expanded and especially prioritized in the Product Backlog. The requirements that received the highest priority are discussed with the team during a Sprint Planning Meeting, which takes place at the beginning of the iteration (Sprint). This is where the team finds out what the specific requirement really means and then it estimates the necessary time for the development. All of those requirements that can be taken care of in one Sprint are removed from the Product Backlog and put into action by the development team during the Sprint. This work package is not modified during the running iteration so as to not risk its completion. That’s because in Scrum a completed item is one that has been tested and documented and the Product Owner can only approve completed items during the so-called Review Meeting. Familiar statements like “I am 80% finished” are no longer valid. All other Product-Backlog requirements are taken care of by the Product Owner and prepared, changed or given a new priority for the subsequent Sprint.
The tasks broken down in the work package are registered in the Sprint Backlog and updated according in terms of remaining workload every day. That’s how the always up-to-date Burn-down Chart is created.
Figure 2: A Sprint Burn-down Chart in Agilo for Scrum. The Open Source tool can be downloaded for free at www.agile42.com.
During the Sprint the team can focus on processing the Sprint-Backlog tasks without interruption. Each day the team meets for a 15-minute time-boxed meeting: the Daily Scrum Meeting. This is where emerging issues are addressed and results discussed. The Scrum Master ensures that rules are adhered to throughout the entire process and that there are no external disturbances. At the conclusion of the Sprint the team presents the implemented functionalities to the Product Owner and stakeholders during the Sprint Review Meeting.
Scrum – It’s Easy Right?
As easy as Scrum sounds, this fast and easy-to-implement framework still comes with a few prerequisites. Responsibilities have to be defined, roles clarified and techniques acquired. It is by far not enough to visit a training workshop or to obtain a Scrum -Master certificate. As is the case with any other method, if you want to use Scrum successfully, you have to analyze processes, evaluate the strengths and weaknesses of the team and, if applicable, break open existing structures to re-organize them. It’s important to know that a framework like Scrum is not an out-of-the-box solution. Individual consulting, training and coaching by well-versed advisers is not just sensible but advisable. But don’t worry, you’ll see results after just one Sprint. And the success–as seen in countless projects–is extraordinary.
Scrum – The Road to Success?
If a company’s marketing strategy included claims like: 300% faster development times, 100% more customer satisfaction or similar statements, we wouldn’t believe it. And yet in the case of Scrum these and similar results are often heard. But how did that happen? The success of Scrum is based on its fast implementation capabilities but also on one simple fact: the teams have fun. And when you’re having fun, you enjoy your job and obtain better results. After all, let’s not forget one thing: developing software is a creative vocation. Fulfilling excruciatingly detailed requirements that also dictate the technical implementation hardly leave any room for creativity. Responsibility and team spirit generated by Scrum will trigger a motivational increase unlike anything else.
Advantages of Agile Processes such as Scrum
- High flexibility for the development process
- Fast reaction to change requests
- Early and thorough client involvement
- Simplicity and an intense focus on team work
- Collaboration beyond team membership
- Improved communication
- Visible and sellable results after each iteration
- Higher software quality through testing
- Greater process transparency
- Outstanding productivity within project teams
- Communication between geographically separated teams requires more effort
- Flexibility at the cost of formality (possibly a concern for applications with higher safety requirements)
- No focus on documents
Of Apples and Pears
To compare an agile framework like SCRUM with CMMI (Capability Maturity Model Integration) is like comparing apples and pears. Scrum is based on best practices and makes the following assumption: a team with responsibility and decision-making power can achieve far better and faster results than a team that works according to rigidly imposed processes. In contrast CMMI is a complex structure consisting of a process model, a quality standard and a validation model. Process models such as CMMI or other process models such as the V-Modell attempt to define the course of action as closely as possible, so that repeated actions are possible at any time and can be performed by any team member. They may be tailored individually to organization-specific concerns but otherwise the process models are rigid structures whose focus lies with planning, processes and documentation. Agile on the other hand signifies “mobility, flexibility” and it signals the capability to react flexibly to new requirements or changes. The practical approach of Scrum focuses on result oriented and team oriented work methods.
Figure 3 Methods and their Key Aspects
Whereas critics of agile methods often assail Scrum for being what they call chaos management, CMMI advocates are often faced with criticism regarding the model’s rigidity, lack of creativity, high cost and inertia.
But now for the comparability: there simply is none. As mentioned before, this would be like comparing apples with pears. What you can do, however, is combine and connect both models. After all, why shouldn’t you use Scrum as the method for developing while at the same time implementing CMMI and so optimize the software-development processes in your company.
Martin Fritzsche and Patrick Keil in 2007 at the University Munich created a detailed analysis on how Scrum and CMMI process areas are compatible with each other. The result is clear. With few exceptions the two opposite process models can be successfully combined.
Figure 4: Covering the CMMI Process Areas with XP and Scrum (XX= widely supported, X=supported)
From Theory to Practice
There we have it: even if apples and pears cannot be directly compared, they can be combined rather well with each other. But let’s take this a step further. Regardless of which method or model a company wants to use, the question whether a model is right or wrong, successful or lacking cannot be answered fully in theory. For one, every process change also brings about structural changes, whose full impact can only be measured once implementation has begun. Secondly, methods are usually not put into practice in their original or optimal form. Therefore, it doesn’t suffice that i.e. only the developer team work with Scrum if the sales, marketing or executive department continues to infuse changes or new requests at any time, without respecting the Sprint phase. Using methods such as Scrum, developer teams can work more efficiently and faster but only when everybody involved in the development process understands what Scrum is and the circumstances under which it thrives. Each participant has a role with rights and duties. Understanding and respecting these rules is a must in order to employ Scrum or any other method successfully.
This leads to the inevitable fact that methods cannot be applied one-to-one and remain unchanged in every single company. Let’s take a Sprint for example. Usually a Sprint starts with a Sprint Planning Meeting and takes 4 weeks during which the team works on the task without interruption. In most cases this scenario is a realistic one and works without any problems. What happens, however, with support questions or bugs? Does a client have to wait a whole month before an error is fixed? Or maybe Scrum should not be implemented in this company? None of the above! Even if at first one must adhere to the theoretical basics when first implementing a new method so as to ensure they are understood and have become second nature. There still must be room for business specific modifications. Who is to say you can’t process support issues during the Sprint phase? Or why shouldn’t you be able to shorten the Sprint phase to two weeks? That’s why the term framework is more suitable for describing Scrum than the term method is. After all, ultimately the successful and adequate components of the framework determine the implementation in the respective company. To decide on when and where adjustments and deviations from the standard methods make sense, it is a good idea to bring in a consultant such as Dr. Andrea Tomasini from agile42 GmbH: ”Changes or adjustments often rest on variables such as time to market, team constellations and specific client requests.”
Fact is, if you decide to go with agile methods, you have to pay attention to the following things-step by step:
- All participants must be involved and shoulder the responsibility during the introductory period
- Management must back up consultants and pilot teams 100% as changes may also “hurt” a little
- All teams and roles required special training and individual coaching until the new structures have become second nature. Reading a book or just visiting a workshop won’t be enough.
- At the beginning, the processes should adhere to the theory of the method. Only when the principles have been understood, should the changes and adjustments be put into place
Waterfalls and Other Natural Phenomena
Many, “Non-Techies” as I like to call them think software can be changed quickly, easily and without any major effort. This simple assumption surely stems from working with software and having experienced the way it simplifies many daily work processes. How should software users know about all the problems connected to software development? How often have you heard this sentence: “Building software is like building a house.” or “Imagine a supply chain...” And how can you expect a “regular” user to know terms such as traceability or change management? These popular descriptions of how software development works also clarify another point however. They illustrate an aspect that has long been accepted as true but that is not backed up by the reality of things: the Waterfall approach and other rigid process models are a lot less useful than agile processes are. Our many years of experience with countless projects and a large number of new methods have taught us that software indeed cannot be developed by following rigid and planned steps. Approaches like the Waterfall model are based on engineering techniques and assume that all requirements are known at the beginning and planned. They do not allow for dynamic development and cannot meet the requirements for iterative and incremental aspects. Software development is, however, defined by frequent new requirements, changing market strategies and countless changes. And when we’re talking changes, it’s not just changes in the software itself. It’s also changes within the team, work methods and processes. Software development is based on evolutionary courses of action and, therefore, is not comparable with building houses or assembly lines. Even if these examples may well continue to serve as an explanation for laypersons.
Agile Methods and their Limitations
Agile methods often meet with limitations in “major” projects. Obviously the project size (project scope, duration, cost, risks) is relative but the necessary communication in bigger projects must be coordinated more pointedly and cannot always happen synchronously. Also projects with a fixed price tag may prove difficult because creating a complete and specific requirements catalog definitely contradicts the spirit of agility. Nevertheless, a solution is possible. One could, for example, agree on fixed prices for milestones and releases instead of for the entire project. And lastly, there are projects that demand extensive documentation. Whether this condition is due to safety regulations or other reasons is irrelevant. Such projects are less suitable for agile methods.
To be sure, being agile is more than just hype and if you think agile methods such as Scrum are only applicable to areas of Web development and less critical software development, you’re wrong. Scrum can be implemented everywhere where new value systems do not pose a threat, where results count for more than hierarchies and where cost savings is an issue. Being agile, after all, has nothing to do with unstructured or chaotic approaches. Au contraire, agile teams work more constructively and efficiently. And even though agile methods such as Scrum do have their limitations, i.e. with projects subject to strict legislative regulations, analyzing possible areas of usage is definitely worth your while. Experience has shown that many issues with software-development projects are not of a technical but rather a social nature. In agile projects, therefore, it is not only the technical qualifications that count but more prominently the soft skills. If you want to operate with agility, you have to be:
- A team player
- A good communicator
- Self critical
That said: Keep it Lean and Agile (KILAA)
PHP Magazin Print-Editon: 6.2008: Agile Development mit Scrum
References on XP and Scrum:
- Mike Beedle, Ken Schwaber, Agile Software development with Scrum, Prentice Hall, 2002
- Martin Fowler, The New Methodology, http://martinfowler.com/articles/newMethodology.html, 2000-2005
- Martin Fowler, Is Design Dead, http://martinfowler.com/articles/designDead.html,2000-2004
- Ken Schwaber, Agile Project Management with Scrum, Microsoft, 2004
- Henning Wolf et. al., Extreme Programming, dPunkt.verlag, 2005