Agile Enterprise Architecture

When project teams work under the assumption that they can do anything that they want, that they can use any technology that they want, chaos typically results. Functionality and information will be duplicated and reuse will occur sporadically if at all. Systems will not integrate well. Systems will conflict with one another and cause each other to fail. Costs will skyrocket because similar products from different vendors, or even simply different versions of the same product, will be purchased and then operated within production. Although each individual project may be very successful, as a portfolio they may have serious challenges. It doesn’t have to be this way. The cold reality is that very few software-based systems exist in a vacuum, instead they must co-exist with several and sometimes hundreds of other systems. Applications must co-exist effectively with the other systems within your organization. Therefore an application must minimally be developed so that it doesn’t cause adverse affects on your other systems and ideally it should be built to take advantage of and to enhance a shared infrastructure. Every system must be built so it can fit into your existing environment, better yet so that it reflect s the future vision for your organization. This sort of information should be captured in your enterprise architecture, in current state and future state models respectively. One goal for agile enterprise architects is to ensure that this happens in an effective manner, to ensure that the needs of the business stakeholders are understood and anticipated, and to support project teams in their development efforts.

Why are enterprise issues an important aspect of the Agile Data (AD) method? Because data is an enterprise asset. It isn’t your only enterprise asset, but it is an important one. My philosophy is that to be effective as a developer you must consider enterprise issues, which is why Agile Data’s 2nd philosophy “development teams must consider and act appropriately regarding enterprise issues” exists. However, to remain agile your organization must find a way to streamline their enterprise activities to support agile software development teams in this endeavor. Hence Agile Data’s 3rd philosophy “Enterprise groups exist to nurture enterprise assets and to support other groups, such as development teams, within your organization. These enterprise groups should act in an agile manner that reflects the expectations of their customers and the ways in which their customers work.”

In this article, I discuss:

A few assumptions
What is enterprise architecture?
Why enterprise architecture?
Challenges with current approaches
An agile approach
Introducing an agile approach to enterprise architecture
What should your enterprise architecture efforts produce?
Potential problems with the agile Enterprise architecture approach

1. A Few Assumptions

This article has been written with the following assumptions:

You've read The Philosophies of Agile Data, The Vision of the Agile Data Method, and The roles on Agile Data Projects
You're familiar with the values, principles, and practices of Agile Modeling and Agile Documentation
Your organization wants to support agile software development teams, perhaps following eXtreme Programming (XP) or the Agile Unified Process (AUP), with enterprise architecture efforts
You're willing to rethink your existing approach to enterprise architecture, if any.

2. What is Enterprise Architecture?

For our purposes, enterprise architecture consists of the various structures and processes of an organization, including both technical structures and processes as well as business/domain structures and processes. Following this definition, an enterprise architecture model is a representation of those structures and processes. A good enterprise architecture model will depict the organization both as it is today and as it is envisioned in the future, and will map the various views representing the architecture to one another. These views include both business-oriented perspectives as well as technical perspectives. In many ways enterprise architecture models are a communication bridge between senior business stakeholders and senior IT professionals.

The Practical Guide to Enterprise Architecture

Unfortunately, within the IT industry the terminology isn’t used in quite this way. Sometimes we use the term “enterprise architecture” to refer to the group of people responsible for modeling and then documenting the architecture. Other times we use the term to denote the process of doing this work. More commonly we are referring to the models, documents, and reusable items (components, frameworks, objects, and so on) that reflect the actual architecture. Unless otherwise noted, this is the way that I will use the term at this site.

In the Enterprise Unified Process (EUP) I point out how some organizations choose to separate the business/domain side of EA from the technical side of it, something which is reflected in the EUP's enterprise business modeling and enterprise architecture disciplines respectively. If your organization chooses to think of the EA as encompassing both, which I recommend, then your EA strategy encompasses the scope of those two EUP disciplines.

3. Why Enterprise Architecture?

The benefits of enterprise architecture can be summed up using three words: better, faster, cheaper. It is important to realize that the better, faster, cheaper (BFC) benefits come at a price. You must be willing to invest in the underlying organizational and cultural structures to support them. You need to recognize that these costs exist and ensure that the BFC benefits you achieve outweigh them. Better yet, adopt Agile Modeling’s principle Maximize Stakeholder ROI and strive for maximal benefit.

4. Challenges With Current Approaches

As a consultant I have the privilege of working in a wide range of organizations across the world. The result is that I get to see and try many different approaches to software development, including enterprise architecture efforts. Over the years I have observed a common set of problems that organizations seem to experience. I have yet to see an organization that experiences all of these problems, although I have seen some that experience all but one or two. These common problems are:

There isn’t an enterprise architecture effort.
Skewed focus.
Project teams don’t know the enterprise architecture exists.
Project teams don’t follow the enterprise architecture.
Project teams don’t work with the enterprise architects.
Outdated architecture.
Narrowly focused architecture models.
Dysfunctional “charge back” schemes.
A “do all this extra work because it’s good for the company” attitude.

A common thread behind these problems is a focus on processes and tools over individuals and interactions, the exact opposite of the Agile Alliance’s first value. If your organization experiences one or more of these problems then you may want to consider taking an agile approach to enterprise architecture.

5. An Agile Approach

The Agile Data Method's third philosophy is "Enterprise groups exist to nurture enterprise assets and to support other groups, such as development teams, within your organization. These enterprise groups should act in an agile manner that reflects the expectations of their customers and the ways in which their customers work." Let's explore what that actually means.

First and foremost, the values, principles, and practices of Agile Modeling (AM) should help to guide your enterprise architecture modeling and documentation efforts. This is just a good start though. My experience is that to be successful at enterprise architecture you need to rethink your overall approach and address five fundamental issues. These issues are connected in a synergistic manner; you must address all of them otherwise you will put your effort at risk. These issues are:

Focus on people, not technology or techniques
Keep it simple
Work iteratively and incrementally
Roll up your sleeves
Look at the whole picture
Make enterprise architecture attractive to your customers

Think at the enterprise level but act locally.

5.1 Focus on People, Not Technology

Fred Brooks (1995) wrote that “The quality of the people on a project, and their organization and management, are much more important factors in success than are the tools they use or the technical approaches they take.” The reality is that enterprise architectures are developed, evolved, and followed by people. All of the arguments over “which model is right”, “which notation is right”, and “which paradigm is right” are meaningless if you don’t have a viable strategy for working together effectively. You could create a perfect enterprise architecture model but it doesn’t matter if project teams can’t or won’t take advantage of it.

Effective enterprise architects will work with their customers, very often application developers and agile DBAs, in the most effective manner possible. Sometimes this will be face-to-face drawing sketches with them at a whiteboard, sometimes they will work with project teams via video conferencing, sometimes they will answer questions via email, and sometimes they will publish models and documents. I highly suggest following the AM practice Display Models Publicly for your architectural models and documents – publish them online at an internal web site and even consider putting up paper versions of critical diagrams in their workspaces. A common mistake that I’ve seen architecture groups make is to put their diagrams on the walls within their work areas but not the work areas of the application developers. Better yet, as I argue below there shouldn’t be separate work areas to begin with.

Enterprise architects also work as “boundary spanners” between project teams and the people within your organization that have the long-range vision – very often senior IT and senior business executives.

An interesting observation is that enterprises have two organizational structures – the formal one documented by your organization chart and the informal one that people use to get things done. Within IT departments there are always the “go to guys” that developers work with to get things done, very often other developers that have critical skills or knowledge. Agile enterprise architects understand this and actively seek to become “go to guys”.

5.2 Keep it Simple

A critical concept is that your enterprise architecture models and documents just need to be good enough, they don’t need to be perfect. It is naïve to assume that you can produce perfect artifacts, you’re only human: you will never get it perfectly right and nobody expects you to anyway. Furthermore, even if you did manage to create perfect artifacts they’d be out of date the day after you published them because something within your business or technical environment would change (Embrace Change is also an AM principle). Why is this important? My experience is that a hand-drawn sketch today on a plain old whiteboard (POW) can often be far more valuable than a fully documented and validated document several months from now. Timely guidance from an enterprise architect who understands the current environment and the future vision for the enterprise, even when this guidance is imperfect and based on incomplete information, is often far better than the uneducated guesses that a project team will make on their own while they wait for the official architecture to be published.

By keeping your enterprise architecture artifacts simple you increase the chances that your audience will understand them, that project teams will actually read them, and that you will be able to keep them up to date over time. Overly detailed documents might look impressive sitting on a shelf, but a simple model that project teams actually use is what your true goal should be. You might find When is Enough Modeling Enough? to be interesting.

5.3 Work Iteratively and Incrementally

Agile enterprise architects work in an iterative and incremental manner. Agile modelers will follow the practice Apply the Right Artifact(s) and use a wide variety of modeling techniques as required (more on this later). They will also follow the practice Iterate To Another Artifact when they realize that the model that they are working on either isn’t the appropriate one with which to depict a concept or because they are simply stuck and need to break out of their “analysis paralysis”. They will also follow the practice Create Several Models In Parallel so that it is easy for them to iterate back and forth between artifacts. Instead of holding a use case modeling session an agile modeler will focus on requirements modeling, or even just modeling, and work on use cases, business rules, change cases, and whatever other artifact they require to get the job done. The advantage of these practices is that the right model is used for the task at hand.

Agile modelers will also follow the practice Model in Small Increments, modeling just enough to fulfill the purpose at hand and then moving on to the next task. They don’t try to create complete models, instead they create models that are just good enough. When they find that their models are not sufficient they work on them some more. The advantage of this approach is that they evolve their models incrementally over time, effectively taking a just-in-time (JIT) model storming approach that enables them to get the models in the hands of their customers as quickly as possible.

On the preceding advice, some readers may say to themselves “All of this sounds great, particularly for a project team, but enterprise architecture is different. It’s more complex. It’s more important. It requires significant modeling up front to get it right.” Aaarrrrggghhh!!! That’s old-style thinking. Enterprise architects can work in an iterative and incremental manner if they choose to.

Figure 1 overviews how to take an Agile Model Driven Development (AMDD) approach to enterprise architecture. The enterprise architecture team would define the initial architectural vision and models, a process that would likely take several days or even a week or two. Any longer and you’d be at risk of developing architectural models that don’t actually reflect something that would work in practice. Remember, your models need to be just good enough, not perfect. The idea is that your enterprise model(s) start out small and are fleshed out over time based on the feedback you receive from both the business community and from project teams.

Figure 1. Agile Model Driven Development (AMDD) at the enterprise level.

5.4 Roll Up Your Sleeves

Although an important part of the job of an enterprise architect is modeling and documentation, that should not be your primary focus. Supporting the architecture within project teams should be, coaching developers in the architecture and architecture skills. The best way to do this is to get involved with project teams, to work with them to understand the enterprise architecture and to try it out in practice. In other words, the enterprise architects will often take on the roles of "architecture owners" on the application teams. This approach has several benefits:

You quickly discover whether or not your ideas work, and if so then how well.
You improve the chance that project teams understand the architecture because you’re working with them face-to-face.
You cross-fertilize ideas, particularly technical ones, across teams, quickly sharing good ideas and strategies.
You improve the chance that a common infrastructure, both technical and business, will be built and reused over time because the project teams will be working towards the enterprise architecture.
You gain experience in the tools and technologies that the project teams work with, as well as the business domain itself, improving your own understanding of what it is that you’re architecting.
You obtain concrete feedback that you can then act on to improve the architecture, enabling it to evolve over time to meet the actual needs of your organization.
You gain the respect of your primary customers, the application development teams, because they see that you’re participating and not simply pontificating.
You actively help to build software-based systems, the primary goal of IT professionals.
You can mentor the application developers and agile DBAs on the project teams in modeling and architecture, improving their skill sets.
You provide clear benefit to application teams because you’re helping them to fulfill their immediate goals, forsaking the “do all this extra work because it’s good for the company” attitude for a more attractive “let me help you achieve your goals, and by doing so together we’ll do something good for the company” attitude.
You work closely with the development teams and enterprise administrators to ensure that your overall data management (including Master-Data Management (MDM)), security management, network management, ... efforts support and enhance the development teams efforts instead of hinder them.

My experience is that the enterprise architects need to be active members of project teams, and to do that they must be co-located with them. When architects are in a different location, perhaps a different corner, on another floor, or even in another building, their ability to communicate with and work together effectively with the project team(s) they are trying to support is dramatically diminished. The implication is that enterprise architects may need to become nomadic, moving between their “home base” to the work areas of the project team(s) that they support. When you work side by side with someone you pick up more information (often just by overhearing something) and you make yourself easily available to them thus increasing the likelihood that they will take advantage of your expertise.

5.5 Build It Before You Talk About It

Agile enterprise architects will ensure that their technical ideas actually work before they advice project teams to try them by writing a small system to validate the idea. This is called a spike solution in eXtreme Programming and a technical prototype or skeleton in the Rational Unified Process (RUP). The idea is that you write just enough code to verify that what you think will work actually does. This helps to reduce technical risk because you’re making technology decisions based on known facts instead of good guesses.

5.6 Look at the Whole Picture

Agile enterprise architectures, agile modelers in general, believe in the principle Multiple Models and thus strive to look at the whole picture. They don’t just focus on data models, or object/component models, or business models, or whatever type of model might tickle their fancy. Instead they strive to model from several points of view so that their understanding and description of the architecture is more robust.

5.7 Make Your Enterprise Architecture Attractive to Your Customers

Agile enterprise architects realize that they need to make their work, including their services, attractive to their customers (developers and business stakeholders). If your customers perceive that you have value to add, that your enterprise architecture efforts will aid them in their jobs, then they are much more likely to work with you. If, on the other hand, they think that you’re wasting their time they won’t work with you. They’ll find ways to avoid you, to cancel or postpone meetings with you, to find ways around you.

6. Introducing an Agile Approach to Enterprise Architecture

Introducing the techniques and philosophies described in this article will prove difficult in many organizations, particularly those that have an established enterprise architecture group that follows a traditional approach. Adoption agile techniques requires a change in mindset – agile enterprise architects are service oriented, realizing that it is their job to help project teams to succeed and to work with senior stakeholders to define and evolve the corporate vision. Agile enterprise architects realize that they need to make it as easy as possible for other people to work with them and that they must provide perceived value to the teams that they support. They realize these things, and act accordingly, because they know that the people they are supposed to serve will ignore them if they don’t. In the end it’s all about people and the way that you interact with them.

My experience is that the best way to introduce agile architecture techniques into an organization is to start small at first and grow your strategy over time. This approach allows you to gain some initial successes, to learn from your experiences (because everything won’t go perfectly right), and to evolve your strategy based on those learnings. A common mistake is to try to put an enterprise architecture team in place and have all teams start to follow the new vision. The typical result is that existing project teams become confused, the enterprise architecture team becomes swamped trying to play catch up, and fighting ensues over “which is the best way”. The fundamental mistake that is made with this approach is that it doesn’t allow the enterprise architects to build a solid foundation from which to work, to build up the proof that their approach actually works, and basically to get ahead of the project teams.

7. What Should Your Enterprise Efforts Produce?

If you’re hoping for an exact list of deliverables here then you need to go back and re-read this article because you haven’t gotten it yet. Sorry for being harsh, but sometimes you just need to say it as it is. However, it is important to define a set of goals that should be achieved. In priority order, these goals are:

Customer support for the enterprise architecture.
A vision and plan to achieve that vision.
A collection of models and documentation describing the architecture.

8. Potential Problems With The Agile Enterprise Architecture Approach

No approach is perfect, including this one. I would be remiss if I didn’t identify these known issues:

It does not include an explicit way to ensure compliancy (although having enterprise architects embedded on the teams goes a long way towards this).
It depends on people being responsible.
It requires you to actively strive to keep things simple.
It requires you to accept an agile approach to modeling and documentation.

The approach described in this article works incredibly if you let it. Your most important take away point is that it’s all about people. Fancy tools based on theoretically sound frameworks, metamodels, or modeling languages are great to have but they won’t do anything for you if developers don’t use them. It’s all about people. Sophisticated models and documents are interesting to create, but they offer little value if nobody reads them. It’s all about people.