Architecture Envisioning: An Agile Best Practice

A common agile practice is to perform some high-level architectural modeling early in the project to help foster agreement regarding your technical strategy within the team and with critical stakeholders. The goal at this point is to identify an architectural strategy, not write mounds of documentation, enabling you to do this swiftly. You will work through the design details later during development cycles in model storming sessions and via TDD. This article addresses several critical questions:

  1. When should you do initial agile architectural modeling?
  2. Why should you do some initial agile architecture modeling?
  3. What should you model initially?
  4. What modeling tools should you use?
  5. How much modeling do you actually need to do?
  6. Why do you need to do less initial architectural modeling than you think?
  7. Are people actually doing this?
  8. Parting thoughts


1. When Should You Do Initial Agile Architecture Modeling?

Agile Model Driven Development (AMDD), see Figure 1, explicitly includes an initial architectural modeling effort during Iteration 0 of an agile project (what some processes might call the Warm-UP, Inception, or Initiation phase). Initial architecture modeling is particularly important for scaling agile software development techniques to large, complex, or globally distributed development (GDD) efforts.


Figure 1. The AMDD lifecycle: Project Viewpoint.


2. Why Should You Do Some Initial Agile Architecture Modeling?

Some people will tell you that you don't need to do any initial architecture modeling at all. However, my experience is that doing some initial architectural modeling in an agile manner offers several benefits:

  1. Improved productivity. You can think through some of the critical technical issues facing your project and potentially avoid going down fruitless technical paths.
  2. Reduced technical risk. Your team gains the advantage of having a guiding vision without the disadvantage of having to overbuild your system – just because you've modeled it doesn't mean you have to build it.
  3. Reduced development time. Initial agile architecture modeling enables you to make better cost and time estimates for your project, two pieces of information which management will want.
  4. Improved communication. Having a high-level architecture model helps you to communicate what you think you're going to build and how you think that you'll build it, two more critical pieces of information desired by management.
  5. Scaling agile software development. Your initial architecture model will be a key work product in any "agile at scale" efforts because it provides the technical direction required by sub-teams to define and guide their efforts within the overall project.
  6. Improved team organization. Effective teams are organized around the architecture or line of business, not around job function. As you scale to larger and/or distributed teams the sub-teams should each be responsible for one or more sub-systems -- you don't want to organize your sub-teams around job function (e.g. an architecture team, a development team, a testing team, ...) because that requires you to increase the documentation and bureaucracy overhead which in turn increases risk, cost, and time to value.
 Practical Guide to Enterprise Architecture

3. What Should You Model Initially?

Early in the project you need to have at least a general idea of how you're going to build the system. Is it a mainframe COBOL application? A .Net application? J2EE? Something else? To do this the developers on the project will get together in a room, often around a whiteboard, discuss and then sketch out a potential architecture for the system. This modeling work is based on, and performed in parallel to, your initial high-level requirements modeling efforts. Your architecture will evolve over time so it does not need to be very detailed yet (it just needs to be good enough for now), and very little documentation (if any) needs to be written.

When I'm doing initial architectural modeling I'll typically focus on high-level free-form diagrams which overview how we think we'll build the system. I will typically focus on: Agile Modeling

3.1 Technology Diagrams

Often some form of technology stack diagram (Figure 2) or deployment diagram will do (Figure 3). These diagrams are useful because they depict the major software and hardware components and how they interact at a high level. This includes legacy assets, including legacy databases and legacy systems, which may need to be analyzed in greater detail later in the project. Right now your goal should be to identify these assets and perhaps discuss their viability as sources of functionality and/or data for your system.

Figure 2. A technology stack diagram.


Figure 3. A UML deployment diagram.

You will also be identifying technical constraints at this point in time. For example, although you would love to work with the latest version of DB2, unfortunately your corporate database standard is Oracle (as you see in Figure 2) and you're therefore constrained in that architectural choice.


3.2 User Interface Flow Models

Another common diagram to create is a user interface (UI) navigation or UI-flow diagram, see Figure 4, to explore how you will architect the UI of your system by exploring the flow between major UI elements, including both screens/pages and reports. This is critical to your system's success because the user interface is the system to your stakeholders. Not the technology. Not the data. Not really cool frameworks that you're working with. If you do not architect the user interface effectively you run the risk that you will build a system that your stakeholders aren't interested in working with.

Figure 4. A UI flow diagram.


3.3 Domain Models

Part of your initial architectural modeling efforts, particularly for a business application, will likely include the development of high-level domain model as you see in Figure 5. This model should be very slim, capturing the main business entities and the relationships between them. Some people consider this type of model to be an initial requirements model instead of an initial architecture model -- it doesn't really matter because as you see in Figure 1 both of these initial modeling efforts are performed in parallel anyway. Figure 5 depicts an example using UML data modeling notation (you can use any notation that you like when agile data modeling, I prefer UML). The initial domain model will be used to help guide both the physical data model as well as the class design, potentially captured via a UML class diagram. I will often create this type of diagram using a whiteboard initially and then later I might transfer it into a drawing tool if the effort of doing so provides sufficient value.

Figure 5. Initial domain model.


3.4 Change Cases

Finally, another common architectural model that I sometimes consider capturing are change cases which are potential architecture-level requirements which your system may need to support. Figure 6 provides examples of two change cases, the describing a potential a technology change and the second a potential business change. Change cases allow you to test the long-term viability of your architecture without requiring you to overbuild your system because you can think through the impact of likely changes to ensure yourself that your system will still work.

Figure 6. Two change cases.

Change case: Registration will occur completely via the Internet.

Likelihood: Medium likelihood within two to three years, very likely within ten years.

Impact: Unknown. Although registration will be available online starting in September, we currently expect less than one quarter of registrations to be made via the Internet this year. Response time will be an issue during the peak use periods, which are the two weeks prior to the beginning of classes each term, as well as the first week of classes.

------------------------------------------------------------------------------------------------------------

Change case: The university will open a new campus.

Likelihood: Certain. It has been announced that a new campus will be opened in two years across town.

Impact: Large. Students will be able to register in classes at either campus. Some instructors will teach at both campuses. Some departments, such as Computer Science and Philosophy, are slated to move their entire programs to the new campus. The likelihood is great that most students will want to schedule courses at only one of the two campuses, so we will need to make this easy to support.

4. What Modeling Tools Should You Use?

Agile modelers will use the simplest tool which will get the job done. As you can see in Figure 2 and Figure 4 I've found whiteboard diagrams typically suffice for initial architecture models. Your goal is to think through the critical technical issues at this point in the lifecycle, not create pretty diagrams. There's nothing wrong with using more sophisticated drawing tools, as you can see in Figure 3 and Figure 5, just ensure that the investment you're putting into the effort is worth it. It all depends on the situation that you're working in. Furthermore, paper-based tools are also quite common. For example I would be tempted to initially capture the change cases of Figure 6 on index cards and only capture them using a more sophisticated tool if appropriate.

5. How Much Initial Architectural Modeling Should You Do?

Let's consider several common situations that you project team might find itself in, and then reflect upon how much architectural modeling would be appropriate in that situation. These scenarios are:

  1. Your team is working with known technologies that it has worked with before. This is the simplest situation you can find yourself in and at most I suspect that you need to create a single whiteboard sketch to ensure that everyone is working to the same vision, and frankly you likely don't even need that. For example, if this is the tenth web-based application that your team has built using Websphere and DB2, how much architectural modeling do you really need to do?
  2. You have little or no experience with the technologies. Traditionalists will often tell you that to reduce your risk that you should model everything in detail, but if you step back and think about it for a minute, this actually increases your risk. Does it really make sense to do a lot of detailed modeling when you don't really know what you're doing? No, all that could possibly achieve is that you waste a lot of time, write a lot of documentation which will likely be ignored by the developers because they'll know pretty quickly that you don't know what you're talking about, it give you false confidence that you know what you're doing, and it will likely give management a false sense of confidence that the team has a solid architecture plan in place. What you really want to do is do a little bit of modeling to help you identify what you don't know and, if possible, you want to invite people on to your team who do have the experience that you're missing. Then you need to gain some hands-on experience with the actual technologies themselves via architectural prototyping/spiking, and then based on that experience evolve your architectural models so that they reflect what actually works. In other words, let your architecture emerge throughout the project.
  3. Somewhere in the middle. This is where you typically find yourself: perhaps you have experience with some but not all of the technologies rr perhaps your team is experienced with the technologies but simply hasn't used them all together at once. In middle ground situations such as this it makes sense to do a lot more modeling initially (perhaps even several days) so as to identify potential risks and likely strategies to mitigate them. Modeling makes sense in this situation because it enables you to define a shared technical vision within the team and to potentially think through major issues. In other words, some initial architectural modeling will offer value to your team.

Figure 7 depicts the value of modeling, showing that models reach their point of maximal value when they are barely good enough, and for initial architecture envisioning this is typically after several days or a week or so. Many people are initially shocked by this claim, but when you think back to all the projects that you've ever been on, how many of them, if you were allowed to, could you have modeled a pretty good initial architectural strategy within a week by putting a few smart people together in a room with a lot of whiteboards? I've asked this question of several thousand people now and only been told about a handful of projects where this wasn't true. A lot of people thought that they had examples where it didn't work out, but their problems almost always revolved around not being able to get people who knew what they were doing or logistical problems of getting the people together. The people that did need more than a week were in situations where the team was distributed so they needed to invest more time to identify the subsystems/components and the interfaces to them so that they could then organize subteams around implementing those subsystems or subcomponents (a practice called API First or Conway's Law). Or they were working on life critical systems (or death critical systems in the case of a couple of military projects), or were in situations where a significant investment in software or hardware was required.


Figure 7. The value of modeling.



6. Why Do You Need to Do Less Initial Modeling Than Traditionalists Think?

Traditional modelers assume that it is desirable to model the problem and solution domain in detail at the beginning of a project. I refer to this as "big modeling up front (BMUF)", something which encompasses "big requirements up front (BRUF)" where you create a detailed requirements specification early in the lifecycle and "big design up front (BDUF)" where you create a detailed requirements specification early in the lifecycle. These practices make sense when you're a modeling specialist, and sure enough traditional modelers always have seem to have good reasons for why BMUF is desirable. Table 1 summarizes the common arguments for doing detailed architecture models early in a project and argues why they're wrong.

Table 1. The Arguments, and Counter Arguments Against, Architectural BMUF.

Argument Counter-Argument
We can think the technical details through up front This may be true of very simple systems, but for even moderately complex systems we cannot hope to think through the details because of the various nuances of each technology, changing usage patterns of the components, and changing technologies themselves. It may appear that everything in your architectural model is going to work, but until you prove it with code and then act on what you learn you'll never know for sure. The best architectures evolve over time, they're not defined up front.
We need to identify the best technical strategy Yes, you need to identify a good strategy but that doesn't mean you need to write a lot of detailed documentation. With a BDUF approach you're being asked to make serious technical decisions when you know the least about the problem domain, increasing the chance that you'll choose the wrong path. Lean/agile approaches instead advise you to make these sorts of decisions at the last possible moment when you can safely make them, instead of at the beginning.
We need to build the technical foundation first You need a technical vision to work towards and good communication/collaboration between the people on the team. A detailed architecture/design motivates you to overbuild your system -- it's in the design, so you'd better build it, whether the stakeholders really need it or not. How many times have you seen project teams spend the first six months of a project building great frameworks which will support all future needs, only to discover that a lot of the framework isn't needed at all? Unless you get the design perfectly right, which is very difficult to do in practice, you'll end up building far more than you need to.
We need to specify the design for the coders Coders are really smart people. They may need some guidance and leadership, but they very likely don't need to be told in detail what to do. The fact is that you put a project at risk when you present a detailed architecture to coders. Some coders may choose not to follow the design in the belief that they know better (and perhaps they do). Worse yet, the more modeling and documentation that you do, the greater the chance that'll you'll commit to the documented approach because of the investment that you've made in the work. This occurs even when the design is known to be questionable because it's often too much effort to rework the design approach once you've started down a specific path.
Architecting software is just like architecting a bridge The bridge analogy generally goes like this: "Building a bridge is a very complex thing, just like building software. Bridge builders create complex blueprints up front and then work to those plans, therefore we should do the same thing too." If someone doesn't have direct experience building bridges and building software then shouldn't we question their assumptions? I know some physical architects, and they work in a far more creative and evolutionary manner than you might think. To see a world-class architect in action, I highly suggest watching the documentary "The Sketches of Frank Gehry". I can very safely say that Gehry is an agile modeler, not a traditional one. So, perhaps the bridge analogy does hold, but not in the way that traditionalists think.
The architecture should be defined in detail just as the requirements are Have you ever tried to fully specify your design when you are decorating a room in your home? No matter how much effort you put into it, you'll always discover that the couch is at the wrong angle, that the wall art doesn't work with the area rug, and so on. You always have to adjust your design and sometimes even make major changes. If you can't do something as simple as specify up front the decoration of a room, what makes you think you can do the same for a system which is many orders of magnitude more complex? People don't know what they want; at best they may understand their goals/intent. People are very good at saying what they don't like, and providing suggestions for possible improvements. This implies the need to take an evolutionary approach where you obtain feedback on a regular basis, not a serial, documentation-based approach. This assumes of course that you actually want to build a system which reflects the needs of its stakeholders. The bottom line is that requirements change throughout a project and you need to accept this and act accordingly.
Detailed documentation and models add value This argument might be partly true, although writing comprehensive documentation early in the lifecycle is the wrong time to do it. Agile Modeling recommends that you write documentation only when the information being captured has stabilized AND if someone understands and is willing to cover the total cost of ownership (TCO) of that documentation. You don't want to document speculative things, and documenting requirements and proposed designs early in the lifecycle are clearly speculative, because speculative things have a tendency to change and thereby increase your TCO for that documentation.
There's no proof that evolutionary techniques work Actually, yes there is. When people ask for "irrefutable evidence" what they're often saying is that they don't want to move away from their preferred way of working. Traditional approaches were adopted with far less evidence than what we have today supporting evolutionary approaches, and to be quite blunt I think that the traditional track record speaks for itself -- the evidence, if you're willing to accept it, clearly points away from BMUF.

Think big but act small.

7. Are People Actually Doing This?

Yes! Here's some evidence from several surveys that I've done over the years:

  1. Initial architecture modeling effective. In the DDJ Agile Adoption Rate survey in March 2007 one of the questions that I asked was how effective initial agile architectural modeling was on agile teams. The results are summarized in the histogram of Figure 8. As you can see, Agilists do in fact model in practice. 77 percent of respondents who said that their organizations were doing Agile indicated that those teams were also doing initial high-level agile architecture modeling, and that 88% of those teams found the effort worthwhile (they gave it a rating of 3 or more out of 5).
  2. The majority of agile teams are doing initial architecture modeling. Figure 8 shows that 77% of agile teams were performing initial architecture modeling. Table 2, which summarizes results from the 2009 Agile Project Initiation Survey, shows that 83% of respondents indicated that they did some form of architecture modeling on their most recent agile project. It also showed that a few teams are leveraging enterprise models, hopefully they're interacting with agile enterprise architecture teams, and sometimes even industry models.
  3. Various strategies are being applied. The DDJ 2008 Modeling and Documentation survey found that sketching was the most common primary approach to modeling (see Figure 9). It also found that agilists are more likely than traditionalists to model on software development projects. Finally, it found that some agile teams are using software based modeling tools (SBMTs), also known as computer aided software engineering (CASE) tools, as well.
  4. It's common for Internet-based development. Many open source systems, and other shared source environments, take an approach where they begin their work with a rough consensus as to what and how to build something along with some working code.

Figure 8. Adoption Rates of Modeling on Agile Teams.


Table 2. Adoption rates for initial architecture strategies on agile project teams (2009 Agile Project Initiation Survey).
Rate Strategy Discussion
70% High-level initial architecture modeling I'm surprised that this figure isn't a bit higher, although it would be reasonable to imply that the teams which are doing detailed initial architecture modeling are very likely also doing high-level initial architecture modeling as well. As you see below, 76% of teams do some sort of initial architecture modeling.
25% Detailed initial architecture modeling I'm surprised that this number is a high as it is, perhaps it's a sign that organizations haven't yet made the full transition to agile strategies from traditional strategies.
12% Initial architecture models provided to team I unfortunately didn't ask why this was happening. Plausible reasons are that this is due to waterfall-driven contracts, e.g. in outsourcing situations the customer develops the IT architecture for the system and provides it to the service provider or that the organization still has a waterfall-driven governance strategy where the architecture work is performed by a team different than the one that's developing the system.
14% Use enterprise models as a reference Apparently some agile teams are in fact taking a disciplined approach to development where they leverage enterprise models and better yet work closely with agile enterprise architects.
12% Use industry models as a reference In some industries, such as the insurance industry or the telecom industry, there are industry consortiums or large vendors (such as IBM), which produce "industry standard" architecture models. The goal of these models is that they should help drive some of the business architecture on development projects. An example of an industry consortium model is the ACORD Insurance Data Standards and an example of a vendor-provided model is IBM Insurance Application Architecture.
76% Do some form of initial architecture modeling (high-level or detailed) Over three quarters of agile teams do some sort of initial architecture modeling. This rate is very close to the 77% rate for initial agile architecture modeling found by the DDJ Agile Adoption Rate survey.
83% Do some form of initial architecture modeling (high-level or detailed) or have initial architecture models provided Almost 7 in 8 agile teams based their work on some sort of initial architecture model.
86% Do some form of initial architecture modeling, or have initial models provided, or leverage available models (enterprise or industry) The vast majority of teams take a model-driven approach to architecture. However, this doesn't imply that they create huge architectural tomes (as you see, only the minority of agile teams appear to do this, instead the majority take a light-weight, high-level approach).

Figure 9. Primary approaches to modeling.


8. Parting Thoughts

You should strive to do just enough modeling for the situation at hand and no more. The implication is that there is no one correct answer for how much modeling each project team will do, that instead it will vary from team to team. My philosophy is that repeatable results, in this case having a shared architectural vision within the team, are far more important than following a repeatable process that ensures that each team does the same amount of modeling each time.

In later iterations both your initial requirements and your initial architect models will need to evolve as you learn more, but for now the goal is to get something that is just barely good enough so that your team can get going. In subsequent releases you may decide to shorten iteration 0 to several days, several hours, or even remove it completely as your situation dictates. The secret is to keep things simple. You don't need to model a lot of detail, you simply need to model enough. If you're writing use cases this may mean that point-form notes are good enough. If you're domain modeling a whiteboard sketch or collection of CRC cards is likely good enough. For your architecture a whiteboard sketch overviewing how the system will be built end-to-end is good enough. I cannot say this enough: your goal is to build a shared understanding, it isn't to write detailed documentation. A critical success factor is to use inclusive modeling techniques which enable active stakeholder participation.

Many traditional developers will struggle with an agile approach to initial modeling because for years they've been told they need to define comprehensive models early in a project. Agile software development isn't serial, it's iterative and incremental (evolutionary). With an evolutionary approach detailed modeling is done just in time (JIT) during development iterations in model storming sessions.