In praise of procrastination, agile methods and antifragility
This article was written by Stefano Borghi, Chief of Software Factory at NSI and expert in agile methodologies.
Nassim Nicholas Taleb, author of “The black swan” (2007) in 2012 wrote “Anti-fragile”, that himself defines his most important opera. In Anti-fragile Taleb takes for granted the claim that black swans dominate society and history, then goes on to consider the practical consequences of this awareness.
What is a Black Swan according to Taleb?
A Black Swan is an unpredictable and anomalous event with a big impact. The most remarkable property of Black Swans, apart from their great impact, is the impossibility of calculating the risk of their occurrence. Moreover, Black Swans tend to deceive us, because retrospectively they can be explained, but the fact is that the probability of a rare event occurring is simply impossible to calculate, the limit is mathematical and unattainable. Add to this the fact that some systems, methods, objects, realities are particularly susceptible to the negative effects of chance, variability, volatility, Black Swans. This is the characteristic of what is fragile.
That which is indifferent (to a certain extent, of course) to chance, variability and volatility is called robust. Robustness, however, is not the opposite of fragility. Within the range of possible reactions to chance, robustness, or resilience, is the neutral point. It is the property of those realities that are not damaged by chance and unpredictability.
That which instead takes advantage of the unexpected is defined by Taleb as anti-fragile.
"Antifragile loves chance and uncertainty, which also means, and this is fundamental, that it loves error, or at least a certain kind of error. Antifragility has the singular characteristic of allowing us to face the unknown..."
Therefore, Taleb argues, we do not waste time trying to predict Black Swans, we do not base our actions and methods on the illusion that we can dominate chance through predictions, but we act in such a way as to be relatively immune to the storms of chance, or we act in such a way as to take advantage of a certain amount of chance, variability, unpredictability, volatility, etc.
What does this have to do with project management and agile methods?
The 2001 study “Extremechaos” by Standish Group International defines the success criterion of a project as being completed on time, on budget and with all originally specified characteristics. According to the same study, only 28% of projects can be considered successful according to this definition.
72% of projects demonstrate their fragility in that: the initial forecasts on which planning was based had almost no reliability. Chance and variability have destroyed them.
In a more recent (2014) Standish's study data from the interviews of a sample of 365 respondents (IT managers) show that the success rate deteriorated to 16.2%, compared to 52.7% of projects out of time, budget or features, and 31.1% cancelled.
Among the unsuccessful projects almost one third experienced cost overruns of 50 to 100%. More than one third of the same projects also experienced time overruns of 100 to 200%. More than a quarter were completed with only 25% to 49% of the originally specified features and functions.
But there is an additional problem to the staggering number of projects that run over time or over budget or end up without all the features set out in the initial specifications.
The additional problem is that even a project that ends up with all the features initially specified is not necessarily a successful project, in fact most probably not. Using Taleb's terminology, we can say that such a project proved to be robust, withstood variability, stayed the course and steered the ship to port. It is a remarkable achievement, no doubt, but insufficient. For it is the very definition of success that is insufficient in this case.
The reason is that initial specifications are rarely the best. The characteristics that a product needs to be equipped with in order to be innovative and successful typically emerge, or become more precise, over time, and setting all the specifications at the outset means doing so at the very moment when the least information is available. The variability of the specifications to which the project has to immunize itself in order to guarantee one of the success criteria - the implementation of all the originally specified features - is precisely the source of that information which could make the final product better.
The users and customers of the product would probably not be satisfied if the ideas of wonderful new features that emerged during the course of the project were rejected in favour of mediocre ones simply because mediocre features were in the initial plan.
We could define traditional project management methods as a search for robustness.
What is robust withstands shocks and remains unaltered. In project management, this means that the aim of a robust project management method is to complete the challenging task by meeting the initial requirements and on time and on budget. In doing so, however, classical project management does not take advantage of uncertainty, which is precisely the mission of Antifragility: instead of resisting the storms of chaos, it improves through them.
It is not a matter of stemming uncertainty and chance, but of turning them to our advantage.
This is, I think, the specificity of agile methods, as stated in one of the four points of the agile manifesto (agilemanifesto.org): “Respond to change rather than follow a plan”.
Iatrogenesis, which literally means «caused by the healer», is the tendency for damage to be caused by the healer, for example when the doctor's interventions do more damages than benefits.
The term is of medical origin, but Taleb generalises it and extends it to the harmful side-effects of excessive interventionism that prevents systems with the capacity to regulate themselves.
The application to project management is mine. I see iatrogenicity in project management that seeks to limit changes in specifications in order to preserve the probability of success. The care taken to preserve projects from the adverse influences of chancedoes, however, lead to embarrassing results from the point of view of the "success" rate of projects and certainly prevents the best use being made of all that knowledge about the product and the method which is developed and refined in the course of project activity.
Agile methods have as one of their programmatic principles the acceptance of change, which is why they work iteratively. In Scrum, the agile framework more famous, the iteration is called Sprint and typically lasts from 1 to 4 weeks.
At the end of each Sprint there are two events specifically devoted to collecting feedback from stakeholders and the team carrying out the work. These events are, respectively, the Sprint Review and the Sprint Retrospective.
The Sprint Reviewis a meeting held at the end of the Sprint to inspect the product increment and adapt the Product Backlog (the equivalent of the requirements list or the Workbreakdown structure (WBS), if necessary. The presentation of the increment is precisely intended to elicit feedback and foster collaboration. The result of the Review is a revised Product Backlog that defines the likely elements of the Product Backlog for the next Sprint. The Product Backlog can also be adjusted overall to meet new opportunities or new knowledge gained during the Sprint.
The Sprint Retrospective is an opportunity for the team to inspect itself and create a plan for improvement to be implemented during the next Sprint. The purpose of the Sprint Retrospective is to:
Inspect how the last Sprint went in terms of people, relationships, process and tools,
Identify and order the main elements that went well and potential improvements,
Create a plan to implement the improvements.
I am a procrastinator and like many procrastinators, I feel guilty for being one. So learning from Taleb about the benefits of procrastination has been a balm for my self-esteem.
According to Taleb, procrastination is positive in some cases because it is a natural form of risk-based decision making. We humans are poorly equipped to filter information, and procrastination can help us to filter it better and resist the consequences of the information onslaught.
I found the same principle in the book "Lean Software Development: An Agile Toolkit" expressed as the Last Responsible Moment (LRM), the strategy of not making a premature decision but delaying commitment and keeping important and irreversible decisions open until the cost of not making a decision becomes greater than the cost of making one. This strategy also aims at maximising learning in the course of a project, maximising the possibility of taking into account information that is only available at a late stage.
Drachten effect, overcompensation and planningchten
Overcompensation is the process of adaptation of an organism to stress factors. According to Taleb, the mechanism of overcompensation is present in many areas. In the field of physical training, the phenomenon is well known and almost all training theory is based on it. In the psychological field, one example is the phenomenon of post-traumatic growth. Much less well known than post-traumatic disorder, post-traumatic growth is a typical example of antifragility in that in it stress does not act negatively (fragility), nor does it remain without effect (robustness), but causes growth. This growth is caused by an overcompensation that triggers an overreaction to a difficulty and this overreaction represents growth.
At the beginning of the 2000s, an experiment was conducted in the Dutch town of Drachten: all road signs have been removed and this deregulation has led to an increase in safety, confirming the anti-fragility of attention and the fact that it is stimulated by a sense of danger and responsibility.
Applied to agile project management, the Drachten effect can be found in the attitude of potentially greater vigilance induced by not following a comforting (and often illusory) Gantt diagram, together with the team members' greater attention to the outcome of their work, given that, working in short iterations, the moment of confrontation with the stakeholders is never too far away in time, thus contributing to maintaining a sense of urgency.
Am I saying that it is wrong to plan? No. What I am arguing is that it is wrong to delude ourselves into thinking we can predict the unpredictable. We agilists plan and plan a lot, but we do not delude ourselves about planning.
A plan is agile if it can be changed easily and is updated often.
The agile planning process, briefly described, is structured as follows:
Construct an initial backlog as a list, ordered by value, of product characteristics described from the user's point of view,
Estimate the total effort required to implement each element of the backlog in a purely relative way (an element worth 2 requires twice as much effort as an element worth 1), with pure numbers from the Fibonacci sequence (The Fibonacci sequence has the remarkable property of having increasing intervals between one number and the next. The elimination of some numbers removes the illusion of being able to really distinguish between a 12 and a 13). Don't go beyond 13 because human beings can estimate better, in a relative sense, within a single order of magnitude. Call these numbers "points",
Run an iteration (a Sprint, Sprint 0) and see how many points the team can work with,
Divide the sum of the points of the whole backlog by the points you managed to work in Sprint 0 and you will have a first estimate (in number of iterations) of how many Sprints you will need to finish, multiply the number of iterations thus obtained by the duration of an iteration (1 to 4 weeks) and you will have a calendar date (very approximate, of course),
Update at each sprint the estimate with the average of the points worked in the last 3 sprints,
At each sprint include as many backlog elements as there are in the order determined by their value.
Such a planning process is relatively immune to some of the flaws that plague WBS and Gantt-based planning:
Thanks to point estimation, task durations are not established at the beginning, as a Gantt diagram requires. Establishing the duration of a task is equivalent to admitting that its duration will be greater or equal to the duration established. By virtue of Parkinson's law: "Work expands to fill the time available for its completion."
Due to the dependency between activities, moreover, any delays propagate. Thus, by virtue of the previous point, an activity never takes less time than expected, then if there is a delay it propagates due to the dependency between activities.
By planning by product features and not by activities, the measure of real progress becomes the number of features implemented, not the number of activities completed. Completing activities without producing product features visible to the end user does not allow for user feedback. One could even reach the paradox of having completed all the activities except one, the last one that will make the product inspectable to the customer or user, only to realise at that moment how far the product is from the real expectations of the stakeholders.
Often in traditional planning there is no explicit ordering of activities intended to ultimately produce features in an order determined by their value (to the end user), activities are instead ordered on the basis of the convenience of the team.
Make mistakes little and often
Working in short iterations (whether short means 1 or 4 weeks depends on the length of the project) allows you to make mistakes little and often.
Without great damage and with the constant opportunity to learn from one's own and others' mistakes.
When a project or project management method is fragile, things need to go according to plan, avoiding changes as much as possible, in this case more harmful than helpful. This is why fragile needs a very detailed forecasting approach. But, on the other hand, predictive systems bring fragility. The project management method is an anti-fragile one, which welcomes deviations because it knows that they bring new knowledge. Moreover, in the trial and error method, the random element is not so random if it is carried out rationally and error is used as a source of information. If each attempt provides information about what works and what doesn't, each attempt is useful and more like an investment than a mistake. And along the way you will discover a lot!
At NSI - Think Outside the Box, we believe that technology simply enables innovation. Together we can transform your software development projects thanks to agile methodologies.