Architecture Change: Breaking Conway's Law

In Architecture Change: Recognizing Conway's Law we looked at the profound influence Conway's Law has on architecture.

Recently I've looked at two gutsy declarations that an architecture was broken. One recognized that a three-tiered architecture was too complex for their needs. The other recognized that the Ontology tools weren't performing well, and perhaps weren't helping.

My point is that these architectural mistakes are the result of Conway's Law. They aren't inherently flawed.

The Root Cause

What's flawed is not the architecture. What's flawed is the organization that built the architecture.

A three-tiered architecture is workable. In some cases, it's necessary. In other cases it could be overkill. But it isn't the cause of the problems.

An Ontology is often a good thing. However, using the ontology to represent what is -- essentially -- a Star Schema fact table is poor use of the technology.

Declaring the architecture broken is not a technical statement. It' an organizational statement. It says that the organization, the teams, the areas of responsibility are broken.

Rule 1: A Broken Architecture Is A Broken Organization

Complexity

One can try to make a distinction between Essential Complexity and Accidental Complexity. One can claim that essential complexity is part of the solution and accidental complexity is just other staff that accretes. This doesn't make any sense, since software development is not "accidental". Software doesn't "happen". It's hard to call something "accidental complexity" without saying that software involves random accidents. Blaming "accidental" complexity is a dodge, an attempt to obscure the root cause.

One might call it incidental or tangential complexity. But that still hides the fundamental problem.

To be more honest, one must separate Problem Complexity from Solution Complexity. The Problem Domain may be inherently complex. In which case, simplification is hard and 2 tiers, 3 tiers or N tiers don't matter. The problem itself is hard, no matter what architecture is chosen.

An ontology, for example, is very helpful when the problem itself is inherently hard. The formalization of relationships in an ontology can help beat a path through a tangled problem domain.

In most cases of a broken architecture, the solution is has grown out of scale with the problem's inherent complexity. If we're doing actuarial risk analysis, we don't really need an ontological model of "Risk": we need facts that help us measure the risk factors.

Rule 2: A Broken Architecture Means the Solution Doesn't Fit the Problem

Corollary: The Organization Doesn't Fit the Problem

Kinds of Broken

Why would we declare an architecture broken? Generally, we've got a grotesque failure due to the very structure of the solution. These can be decomposed into five areas.

• Failure to satisfy the need; i.e., the software doesn't have the required functions or features.
• Failure to use resources effectively; i.e., the software is slow, uses too much disk or too much network traffic.
• Failure to be maintainable; i.e., bugs cannot be fixed.
• Failure to be adaptable; i.e., new features cannot be added.
• Failure to fit other organizational needs (cost, licensing, etc.); i.e., it's too expensive.

The two broken architectures I've heard about recently have different problems. One is unacceptably slow (as well as hard to adapt). The other is described by some as impossible to maintain and adapt.

Rule 3: All Architectural Problems Are Symptoms of Organizational Problems

In short, a broken architecture is not a simple technical problem and it doesn't have a simple technical solution. It's an organizational problem, and it has a multi-part solution.

Making Progress

It's important to acknowledge that Conway's Law, like Mutual Attraction and Thermodynamics is a feature of the universe. It cannot be "broken" or even "subverted". You cannot win, you cannot break even, you cannot quit the game.

Axiom: Conway's Law Cannot Be Broken.

Given that Conway's Law is like Thermodynamics, you have to work with it.

Conclusion: Architecture Must Drive Organization; Problem Must Drive Architecture

The only way to make progress is to restart the project at a fundamental level. You have to -- effectively -- fire everyone and rehire then to create brand-new team. The broken architecture came from a broken organization. To fix the architecture, you need to fix the organization.

Example #1, Unmaintainable Stored Procedures

Consider an application with stored procedures (SP) so badly broken as to be unmaintainable. Let's say it's many hundreds of lines of code. A Cyclomatic Complexity so high as to be laughable. Clearly, the folks responsible for building this need to be reassigned and new folks need to be brought in. If the new folks are simply assigned to the same old separate SP/DBA group, then a new unmaintainable mess will eventually replace the existing unmaintainable mess.

Conway's Law applies: If the SP developers are separate, they will evolve in their own direction. If you want to have a "technical" reason for SP's, then you have to prove that they're more effective than a non-SP implementation. That means spike solutions to compare SP's and your other application programming languages point-by-point.

To prevent stored procedures from getting out of control there are two choices.

1. Don't use stored procedures. Put that logic in with the rest of the application, where it belongs. Same code base, not a separate language buried in the database. One team, one language.
2. Don't make stored procedure writing a separate "team". The stored procedure writing must be part of application writing. One team, multiple languages.

Note that choice #2 leaves it to the team to use stored procedures if they have a provable improvement on performance. Things are not handed over to the DBA's because SP's must do all database interface or SP's must maintain "low-level" rules or other blurry lines. Things are not handed to the DBA's -- the team solves the problem.

Example #2, Too Many Tiers

Consider an architecture with too many tiers. The inter-tier communication is blamed as creating "accidental complexity". This is a dodge. The coordination between teams is what creates complexity.

To prevent inter-tier communication from being a problem, one doesn't need to remove tiers. One needs to remove organizational structure. There's really only one choice.

Fail: Team Follows Technology

Win: Team Follows Features

For a given feature set, everyone involved has to become part of one, unified team working one one sprint attending one daily stand-up meeting.

"But that's unwieldy," you say. "DBA's have to be kept separate."

That's Conways' Law in action.

To work with Conway's Law, you must create a team that owns the feature set -- all tiers -- all technologies -- and can make all the implementation choices required to bring that feature set to the users.

Example #3, Overuse of Ontology

Consider an inappropriate use of an Ontology where a Database would have been a better choice.

1. Remove the old team. Assign them to hard problems where the ontology pays dividends, get them away from easy problems where the ontology is a solution looking for a problem.
2. Create a new team around the new solution. Each feature has a team that has a complete skill set -- front-end, bulk processing, persistence, web server, database, network -- everything.
3. The new team stands alone and builds the solution.

Excuses Excuses

The number one cultural impediment is the "Skill Focus" excuse. These are just Conway's Law in action.

• "We can't have application programmers doing database design. They might 'mess things up'."
• "We don't want our DBA's assigned to application development teams. They have operational responsibilities that trump new development."

The number two cultural impediment is the authorization excuse. These are also Conway's Law, wrapped in the mantel of "security".

• "We can't allow application developers sudo privileges to configure Apache (or MySQL, or Oracle, or -- frankly -- anything.)"
• "We can't assign a DBA or SysAdmin or anyone to support new development..."

Conclusion

Stop organizing teams by skills.

Start organizing teams by deliverable.

Stop carving out random technology features without proof that the technology solves a problem. Stored Procedures, Middle Tiers, Ontologies are just potential solutions. Don't commit to them until they're proven.

Start creating spike solutions to measure the value of a technology. If a spike solution doesn't work, stop development, change the plans, change the schedule and start again based on the lessons learned.

Stop forcing a deadline-driven death march.

Start learning technology lessons and making project changes based on what was learned.

Architecture Change: Recognizing Conway's Law

I've got lots of examples of places where Conway's Law has turned a good idea into a poor implementation. A classic is a data warehouse where there were three project managers, so they broke things up three ways, leading to a crazy mess of dumb duplication.

Countering that, I've recently look at two gutsy declarations. It takes real courage to declare an architecture wrong. Our basic human nature prevents us from acknowledging that an existing architecture is a liability, not an asset.

Pitching a fix is easy. Locating the root cause of the original problem is hard. Trying to fix a broken architecture means that you will run afoul of Conway's Law. In addition to having the guts to acknowledge that something is broken, figuring a way to work with Conway's Law is essential to success.

Broken 3-Tier Architecture

The biggest reason for broken architectures is dumb over-engineering. And most of the dumbosity has Conway's Law as its root cause. Yes, organizational structures will impose a solution structure that doesn't match the problem. There are lots of examples.

If you read too much and build too little, you find a ton of articles on .Net 3-Tier Architectures. Google and you'll get a mountain of hits, each with a distinctive spin on 3-Tier. For reference, start with this: Building an N-Tier Application in .NET. It's the party line on splitting things into tiny buckets consistent with the MS product offerings.

A "3-Tier" presentation is very seductive because it plays by Conway's Law.

• Manager A. Lobbies for Web-based solution; takes over "presentation" development and builds a team to create flashy front-end stuff with cool tools and technologies: HTML, CSS, JavaScript, Silverlight, etc. The front-end developers are as much graphic designer as programmer; they have distinct skills. Conway's Law says that since they're separate from "other" programmers, presentation must be a separate tier.
• Manager B. Manages the DBA's. DBA's must be kept separate because a database is "infrastructure", like a network and a web server. Somehow database development is usually lumped in with database administration and development competes with operation for resources. Conway's Law says that DBA's are separate so there must be a separate data tier.
• Manager C. All of the interfaces and batch loads have to be done by someone. There's no sizzle to this; it isn't fun for DBA's. It's frankly boring stuff. Another manager is assigned to create "back-end" interfaces, and other stuff. Conway's Law says that we'll introduce a "middle tier" to give these people something to contribute to the web application.

At this point, some people call "shenanigans". They say that this Conway's Law analysis is crazy talk: I'm just fitting the evidence to my theory. Here's my question? What's the alternative to the 3-tier architecture? Are they claiming that the three tiers are logically necessary?

Necessary Decomposition

If three tiers were logically necessary, we wouldn't discuss N-tier architectures.

Clearly, folks have decomposed things into more than three tiers. So, three tiers isn't necessary. It's just convenient. QED: There's no necessity to three tiers; it's just a handy team size.

Some scalability works out well with a three-tier separation. In particular, serving a lot of static content (CSS files, PNG's, static HTML) can be delegated to a front-end tier. Serving the dynamic content is better handled by a separate process (perhaps even a separate processor). Database processing -- because it's I/O bound, is often well-handled by a separate process.

However, if the "middle-tier" has a lot of work or relies on slow external web services, it might decompose into sub-tiers. No more three-tier solution. Similarly, one can make a case for splitting static content services into two sub-tiers: reverse proxy and proper content server. Again, no more three-tier solution.

Three tiers, five tiers or N tiers: the architecture could have been driven by necessity or it could be driven by Conway's Law. Clearly, Conway's Law has a profound influence. Indeed, most of the time, Conway's Law trumps all other considerations.

Otherwise we wouldn't have broken architectures. If the decisions were technical, we'd have technical spikes and we'd discard broken ideas. Instead we pursue broken ideas in that weird deadline driven project death-march.

Apostasy

One consequence of Conway's Law is Stored Procedures. That's the tier assigned to the DBA's. The idea that stored procedures might be a bad idea strikes at the very heart of all DBA's (and their managers) and is therefore unthinkable. Try suggesting that stored procedures be replaced by middle-tier application logic. Everyone says that replacing SP's with application code is heresy.

Less than two years ago I sat in a meeting where I was told, very plainly, that the only provably scalable solution was a CICS transaction server and a mainframe DB2 database. The entire room was told that web architectures were a bad idea. Only CICS could be made to work. This is just as dumb as claiming that stored procedures are essential.

This kind of thing leads to a Conway's Law Hybrid solution (CLH™) where the web front-end used SOAP web services to talk to a CICS back-end that merely invoked DB2 stored procedures. No other architecture was discussable. The architecture documentation had to be rewritten to put the simple web site into an appendix as an "alternative". The primary pitch was a hell-on-earth hybrid.

Since there was no DBA bandwidth to write all these stored procedures, the project could only be cancelled. Business rules in Java were unthinkable, heretical. As a former DBA, my suggestion to give up on stored procedures makes me apostate. Stored procedures can be driven by necessity or Conway's Law.

Conway's Law

From http://www.code-muse.com/blog/?p=7

This concept is known as Conway’s Law, named after Mel Conway, who published a paper called “How Do Committees Invent?”. Fred Brooks cited Conway’s paper in his classic “The Mythical Man Month”, and invented the name “Conway’s Law”. Here’s the definition from Conway’s own website (which also has the original paper in full):

Any organization that designs a system (defined more broadly here than just information systems) will inevitably produce a design whose structure is a copy of the organization’s communication structure.

http://en.wikipedia.org/wiki/Conway%27s_Law

More Broken Architectures

Another example.

• Manager A. FLEX front-end development.
• Manager B. Ontology development.

Wait, what? Ontology? No database?

Not at first. Clearly, a good ontology engine will handle the information processing needs. It will be great. The FLEX front-end can make SPARQL queries, right?

Actually, it doesn't work out well. SPARQL is slow. Hardly appropriate for a rich user interface.

So here's another pass at this.

• Manager A. FLEX front-end development.
• Manager B. Ontology development.
• No One In Particular. Backend Web Services Development between FLEX and the Ontology.

"Aha!" you say. "An example that proves Conway's Law is wrong."

Actually, this is evidence that Conway's Law can't be patched. The initial ontology-based application is entirely Conway's Law in action. Trying to create the necessary architectural features without creating a proper organization around the solution ran aground.

Calling It Quits

A really hard thing to do is call it quits when something isn't working. A fundamental law of human behavior says that we hold onto losers. Partly, this is the Endowment Effect -- once the architecture is in place, it can be salvaged. Partly, this is Loss Aversion -- declaring the old architecture broken realizes that the investment created a liability, not an asset.

How do you restart the project with a new architecture?

How do you avoid Conway's Law in the next generation of a web application?

Stay Tuned for part 2 -- Architecture Change: Breaking Conway's Law.