Want to Innovate? Let Someone Else Handle the Boring Work.

Illustration: Cloud Native Transformation Patterns

In the dynamic world of technology, achieving rapid innovation is a common goal. It’s widely understood that innovation thrives when developers have the freedom to experiment and iterate within a supportive environment. While innovation can lead to significant value creation, it may also come with trade-offs such as lower efficiency and increased costs.

Conversely, economies of scale and large-scale operations with reasonable costs are often achieved through standardisation and well-defined processes. This approach may appear to conflict with the pursuit of innovation, leading to the perception that organisations must choose between the two.

However, this doesn’t have to be an either/or scenario. Organisations can achieve both innovation and efficiency by strategically determining which parts of their systems will focus on each objective.

The Role of Cloud Native Transformations

Whenever we embark on a new endeavour, there’s an inherent sense of ambiguity and the need for exploration. This is where innovation flourishes. Such comprehensive reinvention often occurs when starting a new project from scratch or undertaking a transformative change that impacts all aspects of an existing project.

Cloud Native represents one such transformative change. It influences the core architecture of the product, along with infrastructure, development tooling, delivery processes, team structure, and virtually every other aspect of the engineering team. Organisations transition from monolithic architectures, waterfall processes, specialised teams, and on-premises servers to microservices, team topologies, cloud infrastructure, CI/CD, dynamic scheduling, and agile methodologies.

This type of change necessitates a reevaluation of everything, including the potential restructuring of existing teams. It’s similar to starting anew.

Innovation in the Early Stages

In this environment, innovation takes precedence. It’s important to remove any barriers, such as standards, best practices, tool choices, and architectural principles, and begin redefining everything from the ground up. The process starts with a small team that has full autonomy to make its own choices and experiment with new ideas. Technical conflicts are resolved through direct human interaction and frequent communication.

As more teams join, they also enjoy considerable freedom to make their own choices. However, as the product expands and the number of people working on it increases, it becomes evident that complete freedom isn’t sustainable. The introduction of standards and the avoidance of reinventing the wheel become necessary.

The Emergence of Platforms

Once recurring patterns emerge, it’s important to consolidate them in a central location, such as a platform. While this may limit choices during development, it reduces duplicated effort and enhances the quality of reusable components.

Typically, two types of platforms evolve:

• Internal development platform: This accelerates development.
• Business functionality platform: This offers pre-built components like banking or e-commerce functionality, preventing teams from having to rebuild them.

It’s crucial to avoid overloading platforms with excessive functionality too early, as this can lead to premature optimisation. Standards are best established after a solution has gained widespread adoption and become a de facto standard. At that point, it’s integrated into the platform and formally recognised as a standard. The new functionality then receives official support from a central platform team. Its quality, performance, security, and other non-functional requirements are continuously improved to benefit all users without requiring additional time investment from them.

Balancing Innovation and Standardisation

It’s reasonable to expect that some teams may encounter unique challenges that can’t be addressed by existing solutions. In such cases, they might opt for a different tool. For example, if all teams are using Postgres but one team requires a graph database not supported by the platform, they should be allowed to make that choice, but they would also assume responsibility for the operational aspects of the new tool.

A common pitfall in platform development is forcing the platform team to become a repository for random tools and technologies chosen by development teams. The platform should only accept a tool when its functionality is stable, operational, and maintenance procedures are well-established. Until then, the new tool should remain under the ownership of the team using it exclusively.

This additional responsibility for the development team serves as an incentive to carefully consider new tools and, by default, favour existing options. Anything supported by the platform team will generally be well-developed, well-integrated, secure, performant, and accompanied by appropriate documentation for successful adoption. These supported tools will also continue to evolve based on the requirements and contributions from all the other teams using them. This is the essence of a platform – all users benefit from the collective contributions of others.

It’s also important to enable new tools to be integrated into the platform as they mature and gain wider adoption across multiple development teams or when they become essential for successful business operations.

The Goals of Platforms

Platforms aim to promote efficiency by minimising waste and enhancing quality. They enable automation, tight integration between components, security, performance, standards, and processes, significantly reducing waste and making systems more sustainable.

They don’t impede development speed because they don’t restrict the freedom to innovate. Instead, they alleviate the burden of repeatedly addressing problems that have already been solved. Platforms serve as the foundation for efficiency, while the rest of the system can continue to innovate.

For instance, if multiple databases or monitoring solutions are used by different teams without clear justification, maintenance efforts are wasted. Since operations teams have limited resources, maintaining duplicate tools strains their capabilities and can lead to service degradation. When the operations team is overloaded or lacks the expertise to maintain high operational efficiency or implement full automation and self-service, too much of their time is spent addressing system failures or performing manual tasks. This results in a detrimental cycle of declining quality and functionality.

The Evolution of Platforms

Initially, there might be no platform, or it might be very basic, perhaps only providing a thin layer of access to the public cloud and some CI/CD tooling. However, as the transformation and product development progress, and more knowledge and experience are gained, they are systematically transferred to the platforms, which become the official repository of organisational wisdom.

Experience suggests that a Cloud Native transformation typically takes about 2-3 years when executed effectively. The first six months are usually devoted to addressing major challenges and establishing the framework for the future system. After six months, a functional MVP developed by one or a few small cross-functional teams is often realised. The following year is dedicated to creating core platforms and developing a full-scale, secure, and performant product. The final year or so is focused on migrating the remaining legacy teams and their components to the new setup while extending product functionality.

By the end of this process, a successfully transformed organisation will possess robust platforms that implement most of the key operational principles. It will be secure, performant, fully automated, and offer a high degree of self-service. At this stage, the platform is managed as a product with new features, a backlog, releases, internal accounting, etc.

Development teams have the freedom to use platform functionality out of the box and build upon it or choose alternative solutions, understanding that they will be responsible for those choices. However, there also needs to be a consistent process for contributing new tools and technologies to the platform according to well-defined guidelines. Platform teams will only accept new functionality if it adheres to existing operational and security standards.

Conclusion

This approach empowers development teams to continue innovating across all areas of the product and infrastructure, while incentivising them to focus their innovation on new and unique aspects of the product. It also discourages them from reinventing components that are readily available through the platform. By adopting this strategy, organisations can maintain a balance between innovation within development teams and efficiency within platform teams. Additionally, SREs can utilise operational metrics to encourage both platform and development teams to continuously enhance the quality of everything by leveraging the platform more effectively, increasing its efficiency, and ultimately freeing up more development time for innovation

Wondering about the maturity level of your Cloud Native platform? Reach out to us for insights.

Are you ready to elevate your cloud-native platforms to a higher level of maturity?

• Assess your current practices and identify areas where maturity and adaptability can be improved. ¹
• Invest in automation and policies that not only scale but also contribute to the platform’s maturity over time.
• Cultivate a culture of continuous learning and openness to change, which are essential for achieving and maintaining platform maturity.

By applying these principles, you’ll be well on your way to achieving excellence in platform engineering and guiding your platforms towards lasting maturity.

Ready to take the next step in your platform engineering journey? Our team of experts is here to help you navigate the complexities of building and evolving cloud-native platforms. We’ll work with you to set up a platform engineering team tailored to your organisation’s needs, ensuring your platforms achieve the maturity required for long-term success. Contact us today to unlock the full potential of your cloud-native platforms.

Notes