Agile software development

Agile software development is a conceptual framework for undertaking software engineering projects. There are a number of agile software development methods, such as those espoused by the Agile Alliance, a non-profit organization.

Most agile methods attempt to minimize risk by developing software in short timeboxes, called iterations, which typically last one to four weeks. Each iteration is like a miniature software project of its own, and includes all of the tasks necessary to release the mini-increment of new functionality: planning, requirements analysis, design, coding, testing, and documentation. While an iteration may not add enough functionality to warrant releasing the product, an agile software project intends to be capable of releasing new software at the end of every iteration. At the end of each iteration, the team reevaluates project priorities.

Agile methods emphasize realtime communication, preferably face-to-face, over written documents. Most agile teams are located in a bullpen and include all the people necessary to finish software. At a minimum, this includes programmers and their "customers." (Customers are the people who define the product. They may be product managers, business analysts, or actual customers.)  The bullpen may also include testers, interaction designers, technical writers, and managers.

Agile methods also emphasize working software as the primary measure of progress. Combined with the preference for face-to-face communication, agile methods produce very little written documentation relative to other methods. This has resulted in criticism of agile methods as being undisciplined hacking (aka Cowboy coding).

The Agile Manifesto
Agile methods are a family of development processes, not a single approach to software development. In 2001, 17 prominent figures in the field of agile development (then called "light-weight methodologies") came together at the Snowbird ski resort in Utah to discuss the unifying theme of their methodologies. They created the Agile Manifesto, widely regarded as the canonical definition of agile development, and accompanying agile principles.

Comparison with other methods
See [1] Appendix A, pages 165-194

Agile methods are often characterized as being at the opposite end of the spectrum from "plan-driven" or "disciplined" methodologies. This distinction is misleading, as it implies that agile methods are "unplanned" or "undisciplined." A more accurate distinction is to say that methods exist on a continuum from "adaptive" to "predictive." Agile methods exist on the "adaptive" side of this continuum.

<--Agile--> <--Iterative--> <--Waterfall--> <|-||-> Adaptive                        Predictive

Adaptive methods focus on adapting quickly to changing realities. When the needs of a project change, an adaptive team changes as well. An adaptive team will have difficulty describing exactly what will happen in the future. The further away a date is, the more vague an adaptive method will be about what will happen on that date. An adaptive team can report exactly what tasks are being done next week, but only which features are planned for next month. When asked about a release six months from now, an adaptive team may only be able to report the mission statement for the release, or a statement of expected value vs. cost.

Predictive methods, in contrast, focus on planning the future in detail. A predictive team can report exactly what features and tasks are planned for the entire length of the development process. Predictive teams have difficulty changing direction. The plan is typically optimized for the original destination and changing direction can cause completed work to be thrown away and done over differently. Predictive teams will often institute a change control board to ensure that only the most valuable changes are considered.

Contrasted with iterative development
Most agile methods share iterative development's emphasis on building releasable software in short time periods. Agile methods differ from iterative methods in that their time period is measured in weeks rather than months and work is performed in a highly collaborative manner. Most agile methods also differ by treating their time period as a strict timebox.

Contrasted with the waterfall model
Agile development has less in common with the waterfall model. In some eyes the waterfall is discredited, but circa 2005, this model is in common use. The waterfall model is the most predictive of the methodologies, stepping through requirements capture, analysis, design, coding, and testing in a strict, pre-planned sequence. Progress is generally measured in terms of deliverable artifacts - requirement specifications, design documents, test plans, code reviews and the like. The waterfall model can result in a substantial integration and testing effort toward the end of the cycle, a time period typically extending from several months to several years. The size and difficulty of this integration and testing effort is one cause of waterfall project failure. Agile methods, in contrast, produce completely developed and tested features (but a very small subset of the whole) every few weeks or months. The emphasis is on obtaining a crude but executable system early, and continually improving it.

Some agile teams use the waterfall model on a small scale, repeating the entire waterfall cycle in every iteration. Other teams, most notably Extreme Programming teams, work on activities simultaneously.

Cowboy coding is not Agile
Another "method" in common use is cowboy coding. Cowboy coding is the absence of a defined method: team members do whatever they feel is right. Agile development's frequent reevaluation of plans, emphasis on face-to-face communication, and relatively sparse use of documents sometimes causes people to confuse it with cowboy coding. Agile teams, however, do follow defined (and often very disciplined and rigorous) processes, distinguishing agile approaches from cowboy coding.

As with all methodologies, the skill and experience of the user(s) define the degree of success and/or abuse of such activity. The greater the controls and/or checks and balances systematically embedded within a process, offer stronger levels of accountability of the user(s). It is the degradation of well intended procedures which lead to activities often defined as Cowboy coding.

Both lightweight and heavy-weight methodologies may still lead to this breakdown as the user(s) attempts to facilitate within social/political environments within organizations. The probability of this breakdown can be directly correlated to the degree of processes inhibiting the risk of user(s) deviating from the organization standard, however at the cost of efficiency opportunities.

The ideology of Software Development Best Practices has established a fervor of religious debates, all of which the tendency has been to express superiority in effectiveness to deliver.

When to use agile methods
Agile development has been documented (see Experience Reports, below, as well as Beck pg. 157, and Boehm and Turner pg. 55-57) as working well for small (<10 developers) co-located teams. Agile development is particularly indicated for teams facing unpredictable or rapidly changing requirements. While there are experience reports of teams succeeding with agile development outside of these parameters, there are too few experiences reported as of April 2005 to draw firm conclusions.

Agile development's applicability to the following scenarios is open to question:


 * Large scale development efforts (>20 developers) See: Supersize Me for scaling strategies
 * Distributed development efforts (non-co-located teams) See: Bridging the Distance for strategies
 * Mission- and life-critical efforts
 * Command-and-control company cultures

Boehm and Turner's risk-based approach
Barry Boehm and Richard Turner, suggest that risk analysis be used to choose between adaptive ("agile") and predictive ("plan-driven") methods. The authors suggest that each side of the continuum has its own home ground:

Agile home ground:


 * Low criticality
 * Senior developers
 * High requirements change
 * Small number of developers
 * Culture that thrives on chaos

Plan-driven home ground:


 * High criticality
 * Junior developers
 * Low requirements change
 * Large number of developers
 * Culture that demands order

By analyzing the project against these home grounds, the risk of using an agile or plan-driven method can be determined.

History
The modern definition of agile software development evolved in the mid 1990s as part of a reaction against "heavyweight" methods, as typified by a heavily regulated, regimented, micro-managed use of the waterfall development model. The processes originating from this use of the waterfall model were seen as bureaucratic, slow, demeaning, and contradicted the ways that software engineers actually perform effective work.

A case can be made that agile and iterative development methods are a return to development practice seen early in the history of software development.

Initially, agile methods were called "lightweight methods." In 2001, prominent members of the community met at Snowbird and adopted the name "agile methods." Later, some of these people formed the Agile Alliance, a non-profit organization that promotes agile development.

Early agile methods--created prior to 2000--include Scrum (in management) (1986), Crystal Clear, Extreme Programming (1996), Adaptive Software Development, Feature Driven Development, and DSDM (1995).

Extreme Programming, while it may not have been the first agile method, inarguably established the popularity of agile methods. Extreme Programming was created by Kent Beck in 1996 as a way to rescue the struggling Chrysler Comprehensive Compensation (C3) project. While that project was eventually canceled, the methodology was refined by Ron Jeffries' full-time XP coaching, public discussion on Ward Cunningham's Portland Pattern Repository wiki and further work by Beck, including a book in 1999. Elements of Extreme Programming appear to be based on Scrum and Ward Cunningham's Episodes pattern language.

Agile method
Some of the well-known agile software development methods include
 * Extreme Programming (XP)
 * Scrum
 * Agile Modeling
 * Adaptive Software Development (ASD)
 * Crystal Clear and Other Crystal Methodologies
 * Dynamic Systems Development Method (DSDM)
 * Feature Driven Development
 * Lean software development
 * Agile Unified Process (AUP)

Other approaches include
 * Agile Documentation
 * Agile ICONIX
 * Microsoft Solutions Framework (MSF)
 * Agile Data Method
 * Database refactoring

Examples of similar concepts beyond the realm of software include
 * Lean manufacturing

Criticism
Agile development is sometimes criticized as cowboy coding. Extreme Programming's initial buzz and controversial tenets, such as pair programming and continuous design, have attracted particular criticism, such as McBreen and Boehm and Turner. However, much of the criticism was based on misunderstandings about agile development. 

In particular, Extreme Programming is reviewed and critiqued by Matt Stephens' Extreme Programming Refactored.

Criticisms include level of structure and necessary documentation
 * only works with senior-level developers
 * incorporates insufficient software design
 * requires too much cultural change to adopt
 * can lead to more difficult contractual negotiations

The criticisms regarding insufficient software design and lack of documentation are addressed by the Agile Modeling method which can easily be tailored into agile processes such as XP.

Experience Reports
Agile development has been the subject of several conferences. Some of these conferences have had academic backing and included peer-reviewed papers, including a peer-reviewed experience report track. The experience reports share industry experiences with agile software development.

As of 2006, experience reports have been or will be presented at the following conferences (better links to reports needed; also need clarification of which conferences were peer-reviewed and which have academic credentials):


 * XP (2000, 2001, 2002, 2003, 2004, 2005, 2006)
 * XP Universe (2001)
 * XP/Agile Universe (2002, 2003, 2004)
 * Agile Development Conference (2003, 2004) (peer-reviewed; proceedings published by IEEE?)
 * Agile (2005, 2006) (peer-reviewed; proceedings published by IEEE)