The Environmental Cost of Agile: Why Lean Software Development is the Next Step
Did you know the IT industry is on track to become a leading contributor to global carbon emissions? Despite the vast computing power at our fingertips, a staggering amount of it is reserved, but goes unused, creating a digital landfill of wasted energy and resources. This article takes a deep dive into how the pursuit of speed and agility in software development has paradoxically led to this unsustainable situation. We’ll trace the evolution of methodologies from Waterfall to Agile, revealing how our “growth at all costs” mindset has fostered a culture of over provisioning. But there’s hope: we’ll explore how embracing Lean principles can revolutionise software development, leading to a more sustainable and efficient future for the IT industry.
The Rise of Wasteful Practices
Several factors contribute to the inefficient use of IT resources:
- Misaligned Incentives: Public cloud and data centre vendors often charge based on capacity, incentivising them to encourage customers to consume more resources than necessary. While onboarding and scaling up are straightforward, optimisation and rightsizing are often complex and neglected.
- Lack of Developer Awareness: Many developers lack the knowledge or awareness to optimise IT systems effectively. It’s often easier to add new hardware to compensate for poorly optimised software.
- The Cost of Developer Time: Developer time is expensive, and optimising existing systems can be time-consuming. In some cases, the cost of additional resources may be lower than investing in optimisation efforts.
- Jevons’ Paradox : increased efficiency in resource use can paradoxically lead to increased consumption of that resource. Public Clouds compared to traditional On Prem IT create such gains in efficiency. However, the most significant factor driving this lack of optimisation is likely the prevalent “growth at all costs” mentality, which has led to the widespread adoption of Agile software development practices.
The Historical Context
To understand the current situation, we need to trace the evolution of software development methodologies:
- The Craftsmanship Era (1950s-1960s): In the early days of software development, everything was custom-built and unique. There were few established practices, and the focus was on individual ingenuity.
- The Control Era (1970s-1990s): The rise of waterfall methodologies brought standardisation and predictability to large IT projects. However, the monolithic architectures typical of this era limited growth and adaptability. Systems became slower and more complex, and maintaining quality required ongoing investment.
- The Adaptability Era (2000s-Present): The dot-com boom fuelled an unprecedented demand for growth, leading to the adoption of Agile methodologies like Scrum. Agile prioritised adaptability and quick iterations, enabling faster development cycles. Cloud-native technologies further facilitated this mindset, allowing applications to be easily divided into smaller components and deployed to the limitless resources of the public cloud.
The Agile Paradox
While Agile methodologies have undoubtedly revolutionised software development, they have also contributed to the current state of wastefulness. By prioritising speed and adaptability, Agile inadvertently incentivises over provisioning. The abundance of cloud resources and the ease of scaling have created a perception of limitless capacity, where small amounts of waste are often overlooked. The result is a situation where majority of data centre capacity goes unused, leading to a significant financial and environmental cost.
Total Digitalisation and rise of AI
The question looms large: can we continue to onboard new developers and meet the burgeoning demand for IT services while mitigating the industry’s environmental impact? Historically, the IT industry’s growth cycles were driven by technological advancements and increasing demand. This trajectory is set to continue with the global digitalisation of industries, the rise of AI, and other emerging trends. However, a new imperative is emerging: the urgent need for decarbonisation to combat climate change. This creates a complex dilemma. IT demand is projected to double by 2030 and experience explosive growth by 2050. Yet, to mitigate the worst effects of climate change, carbon emissions must be halved by 2030 and reduced by 90% by 2050. But can it be different? What’s next to allow us to onboard even more new developers and make sure we allow the explosive growth in demand for IT tools and services and at the same time avoid incredible waste of money, resources and enormous damage to the planet?
And what will be the driver of such a change? In the first cycles, the main driver was always explosive growth in the computers and IT industry and such growth will continue with global digitalisation of all the industries, popularisation of AI and lots of other growth trends. But at the same time there is a coming demand for decarbonisation to mitigate the Climate Change effects.
IT demand is expected to double by 2030, and explode by 2050. CO2e emissions halved by 2030, reduced 90% by 2050.
The IT industry stands at a crossroads. The pursuit of growth must now be reconciled with the imperative of sustainability. This requires a fundamental rethinking of our approaches to development, resource utilisation, and overall environmental impact. The challenge is daunting, but the stakes are too high to ignore.
The Path to Lean: A Sustainable Evolution
To address the unsustainable trajectory of the IT industry, a paradigm shift towards Lean principles is essential. It’s a return to the roots of Agile, which ironically emerged from Lean manufacturing practices like the Toyota Production System. However, Agile’s focus on speed of change often overshadowed Lean’s core tenets of waste reduction and continuous improvement. Now, it’s time to re-emphasise those Lean principles in software development. By adopting a culture of continuous optimisation, we can reduce waste throughout the entire development lifecycle, from coding to infrastructure. The benefits of Lean extend far beyond sustainability. By eliminating waste, organisations can also:
- Reduce Costs: Optimise resource utilisation and eliminate unnecessary spending.
- Improve Quality: Identify and resolve inefficiencies that can lead to errors and defects.
- Enhance Customer Satisfaction: Deliver value faster and more reliably. Lean software development is not just an environmental imperative; it’s a strategic advantage that can improve every aspect of your software delivery process.
Implementing Lean in Software Development
Several methodologies and practices can help facilitate a more Lean approach to software development:
- Kanban: This Lean project management methodology emphasises limiting WIP, ensuring that teams focus on completing tasks efficiently before starting new ones.
- SRE (Site Reliability Engineering): Introduced by Google, SRE focuses on maintaining the high availability and reliability of large-scale software systems. By monitoring key metrics like latency, traffic, errors, and saturation, SRE teams define service level objectives (SLOs) that translate into service level agreements (SLAs) and service level indicators (SLIs). This data-driven approach ensures that resources are allocated effectively to meet real business needs.
- FinOps: This practice aligns technology, finance, and business teams to optimise cloud spending. By treating cloud costs as a shared responsibility, FinOps encourages continuous monitoring and optimisation of cloud resources. Additionally, a growing emphasis on sustainability is driving the adoption of Green Software practices. The Green Software Foundation defines three key metrics for reducing carbon emissions in software: energy efficiency, hardware efficiency, and carbon awareness.
Jointly they are defined by a single metric - SCI (Software Carbon Intensity specification)
Furthermore, we should not overlook the importance of developer productivity. By incorporating DORA metrics , which assess deployment frequency, lead time for changes, time to restore service, and change failure rate, we gain valuable insights into our development processes. These metrics help us identify bottlenecks, streamline workflows, and ultimately enhance developer productivity.
Integrating Performance, Productivity, and Sustainability: A Balanced Approach
By strategically combining SRE, FinOps, DORA metrics, and Green Software practices, development teams can create a holistic framework that balances performance, productivity, and sustainability throughout the software development lifecycle.
This integrated approach recognises that the optimal balance of priorities shifts as a product evolves:
- Early Stages (Agile Emphasis): In the initial phases, the focus remains on rapid iteration and experimentation to validate ideas and gather user feedback. Productivity, as measured by DORA metrics like deployment frequency and lead time for changes, is paramount. While resource utilisation and sustainability are considered, they are not the primary drivers at this stage. Agile methodologies like Scrum, which prioritise flexibility and adaptability, are well-suited for this phase.
- Growth and Maturity (Hybrid Approach): As the product gains traction and user base, the emphasis gradually shifts towards performance and sustainability. Resource optimisation becomes increasingly important to meet growing demand while minimising waste and environmental impact. This phase requires a hybrid approach, blending Agile’s adaptability with Lean’s focus on waste reduction and efficiency.
- Maintenance Mode (Lean Emphasis): In the later stages of a product’s lifecycle, stability, efficiency, and sustainability take precedence. Continuous monitoring and optimisation become crucial to ensure resources are aligned with actual usage and to minimise the product’s carbon footprint. This phase is where Lean principles truly shine, enabling organisations to streamline processes, eliminate waste, and maximise the value derived from existing resources.
It’s important to note that Agile and Lean are not mutually exclusive. Rather, they are complementary approaches that can be applied strategically at different stages of the product life cycle. By understanding the strengths and weaknesses of each methodology and tailoring their application accordingly, organisations can create a development process that is both agile and sustainable.
We also must remember that any system that is no longer used should be decommissioned, and its resources reallocated or retired. Many organisations struggle with decommissioning due to friction with remaining users or the effort involved in proper retirement. However, leaving unused systems running perpetuates unnecessary maintenance burdens and environmental impacts. A common concern is how to maintain agility in large, mature projects. While agility is crucial in early stages, mature products often prioritise stability. Trying to maintain high agility in a mature product can be resource-intensive and introduce unnecessary risk. A solution is to decouple innovation from scale and maturity. New features and products can be developed separately, using agile methods, and then integrated into the main product once they are stable and proven. This allows for continued innovation while ensuring the stability of the core product.
Conclusion
The Agile movement has revolutionised software development, but its focus on speed and adaptability has inadvertently led to significant waste. To continue growing the IT industry sustainably, we must embrace Lean principles and prioritise waste reduction. This requires a cultural shift, where every line of code and every provisioned resource is scrutinised for its impact on cost, performance, and the environment. By incorporating Lean practices into our development processes, we can ensure that the IT industry continues to thrive while minimising its environmental footprint. We have the tools and methodologies at our disposal; now, we must commit to using them responsibly. The future of software development depends on it.
Embarking on the Path to Carbon Reduction: The first step in effectively reducing your organisation’s carbon footprint is understanding your current emissions. Gain valuable insights into your IT system’s real-time carbon emissions by joining re:cinq’s early beta program. Visit https://re-cinq.com/ to learn more and take the first step towards a greener future.