It’s All or Nothing When it Comes to Agile—Part 2

Part 1 of this series discussed the current state of Agile architecture in most organizations and the difficulty involved in moving from pseudo-Agile to Agile as an organizational construct.

The Agile methodology has introduced a new paradigm for software development, and for the ambitious ones, for organizational structures. Until very recently it wasn’t very practical to build systems consisting of distributed software components. In fact, the mantra in the industry until very recently was “thou shall not distribute applications.” It’s true that achieving independence and loose-coupling of software components is technically difficult and requires a lot of advancement in the underlying infrastructure as well as a different design approach. But the rapid development of the Internet and the creation of many truly innovative technologies and approaches has provided better foundational blocks to enable a movement towards more modular and loosely-coupled architecture and process structures.

Components & Microservices

Preparing an organization for real agility means, among other things, learning from the missteps and mistakes of the past and allowing developers to build software structures composed of small components that are loosely-coupled. These entities are commonly called microservices. As suggested earlier, transitioning to Agile is as much behavioral and attitudinal as it is technical.  It requires new ways of thinking and software design that can be characterized with the following guiding principles:

1. All functionality exists as smaller components and microservices.
2. Software components are as discrete and self-contained as possible, including owning their own data.
3. Components communicate using (mostly) asynchronous events and (sometimes) services.
4. Components run on a cloud-type infrastructure.

There are a number of tangible and quantifiable benefits to developing, implementing, and sustaining small self-contained software components. First, smaller components are easier and less complex to create. That often means that multiple small teams creating multiple small software components can code new functionality relatively quickly. Second, smaller components are far easier to enhance or even replace in the future. It does not require a careful and time-consuming coordination dance between large teams to retest, implement or even retire functionality. Third, testing and long-term support and maintenance is much simpler. Small development teams can perform their own DevOps (the combination of software development with IT operations as a way to optimize the software development life cycle) procedures nearly independently from any other team. And finally, smaller components are the fundamental building blocks for creating small Agile teams, which in turn helps to create Agile organizational processes. It’s analogous to atoms as the building blocks for…well, everything. For the Agile organization, small software components are the atoms.

That said, it is not practical or possible to change a large enterprise overnight and recreate all of its software as a collection of microservices, but it is important to start the process that drives a gradual transformation. That’s why it’s also important to invest in good solution architecture and design skills, since this design approach is not always easy and requires the experience necessary to develop strong component boundaries.  And while an investment is required, the ultimate benefits far outweigh the costs.

Data

One of the keys to leveraging all of the benefits of smaller software components is to make them as self-sufficient as possible. That means that to truly leverage all of the benefits of an Agile approach, smaller software components should own their operational data. This was one of the more important lessons passed along from the SOA era: microservices should be truly self-sufficient. For example, to provide data access to other components, microservices typically publish events to broadcast changes that occur in the data they own, so others can subscribe to the notifications and react accordingly. And of course, microservices can still provide APIs for data access so other self-sufficient software components can inquire.

Once again, this approach emphasizes the importance of organizations having good design skills to use the right approaches to communication between small software components. Fortunately, the abundance of modern database engine options (SQL and NoSQL to name two) make this much easier to accomplish.  Many of these modern databases are very lightweight and can provide excellent support for microservices.

Communications

Efficient communication protocols are a key ingredient of achieving an Agile organization. The emphasis should be on using asynchronous events for software component communications.  While it’s true that microservices can call each other’s APIs to access data, such real-time communication should be kept to a minimum as it is very rigid and demanding.  The systems that rely on real-time communication protocols quickly become tightly coupled and hard to scale, and they don’t lend themselves well to true agility.

Rather, any preference for microservice communication should be in publishing and subscribing to events. Asynchronous events are a universal data sharing mechanism for large software systems and they help to drive very desirable qualities of the enterprise architecture, including such things as loose coupling, scalability, and the ability to evolve individual components seamlessly. Fortunately, the benefits and overall broader understanding of the importance of asynchronous event-driven architecture has spurred the development of many good supporting infrastructure options such as Apache Kafka—a mature platform ready for companies to use to build sophisticated event infrastructures.

Infrastructure

One of the final key elements necessary for implementing an Agile enterprise architecture—and therefore an Agile process driven organization—is infrastructure. The path forward here is simple: invest in infrastructure that is based on cloud management principles. One of the complexities of microservice-based architectures is that small components, as beneficial as they are, tend to proliferate across an enterprise. This leads to large numbers of software components that must be effectively managed, and that is not easy. The good news is that many cloud providers faced similar challenges in the past and have responded by creating a new generation of infrastructure solutions that they made available to other solution providers and customers.

Cloud infrastructures now typically include container technologies (such as Docker) and schedulers (such as Kubernetes), that help tremendously in managing microservice-based software processes. As a result, each microservice can run in its own isolated container and communicate with others via well-defined mechanisms. And importantly, modern Cloud infrastructures help to provide Agile teams excellent support for their needs without excessive interactions with the infrastructure administrators, eliminating process redundancies and delays.

The following future state illustration is an example of the benefits derived from transitioning to an Agile EA organizational structure:

People & Process

This is perhaps the toughest obstacle to overcome on the road to Agile Enterprise Architecture.  And it’s not that people don’t want to make the shift to Agile with all it entails, it’s more that in doing so a lot of ingrained processes, practices, and habits have to be either modified or done away with altogether, and that’s not easy for anybody at any level. That’s why it’s critical to give people time to adjust and acclimate to this shift. One way to do this is to include the impacted parties early and often in the discussion and planning process. The emphasis should be on the reasons and benefits for making the move to Agile EA, followed by the impacts to them, and the requirements of them.

For example, the use of microservices as part of the shift to discreet and self-contained software components will require changes (probably major) to the incumbent processes used to test, move, and implement code. That process likely involves more people and steps than will be required once the shift to Agile EA begins—and certainly when the shift is finalized—so it’s critically important that the people responsible for such things are included early, and that the communications about the changes are clear and include a detailed task list of the steps required to achieve the new outcome.

At the risk of overstating it, the technology required to successfully implement Agile EA, while complex, is easier to manage than are the behavioral changes required of the people involved and invested in the current set of processes and practices. Simply put, if the behavior doesn’t change, the chances of real success in this effort decreases precipitously. Focus on the people involved early and often—with constructive reinforcement, empathy, and forthrightness—and the chances of overall success increase dramatically.

Expectations  

Last but certainly not least, it’s very important to set and manage stakeholder expectations before, during, and after the shift to an Agile EA organization.  This transformation will not happen overnight and will undoubtedly be filled with challenges—expected and unexpected—that will have to be addressed along the journey.  While it’s important to have short duration milestones and successes that are measured in days and weeks, the overall effort will take months and in some cases years. That’s a long time to ask for any organization’s attention, resources, time, and money.

To manage expectations effectively, it’s critical to keep the focus on the long-term benefits to the organization. It’s easy to get lost in the weeds of choosing this or that technology or platform, and many stakeholders will quickly tire of those kinds of conversations. Rather, the discussion should always come back to the core reasons for adopting Agile EA: creating a flexible, adaptive, and responsive business technology foundation that contributes directly to the success of the organization’s strategic goals and objectives.

Conclusion

In order for IT organizations to truly succeed and meet the demands of their business partners they need to embrace Agile philosophies and approaches. Fortunately, that’s no secret to IT organizations and many have begun the journey. But, as this article argues, Agile and architecture must work together as one can’t fully flourish without the other. Therefore, IT organizations have to invest in modern architecture solutions that support a truly Agile way of life. Thus far that’s the missing link in the efforts of most IT organizations. While embracing Agile processes and approaches, many IT groups are still handcuffed by legacy platforms and behaviors.

The goal of true Agile architectures based on self-sufficient microservices communicating with each other using asynchronous events and running on cloud-style infrastructures is attainable.

It will take planning, prioritization, and most of all perseverance. The alternative to this—standing pat with current systems, processes, and organizational communications structures—is not sustainable. The path forward is clear: embrace and implement architectures that reinforce Agile principals and create a much better foundation for a dynamic enterprise of the future.  Melvin Conway would surely agree.