Measuring Architecture
Architecture Evaluation
What's a good architecture? And how do you know?
... architecture requires:

a coherent definition of quality
a manifestation of quality in a set of mechanisms
An operational definition of architecture:

"A system's architecture codifies a set of decisions that are both hardest to change and have the most significant impact on the way the system manifests its quality attributes."
Understanding a system's quality attribute goals, and their priorities, is paramount
We should have a goal of optimality, but realize we can't achieve it
Transparency demands documentation of rationale
Tradeoffs in decisions must be made explicitly
How do architecture decisions usually get made?
Definition stuff
Background
Weighting
Rank-Order Centroids
Build a Quality Tree
Solutions
.NET vs. Java at a major insurance company
Choosing among 8 key management vendors
Critical application rearchitecture effort
Business decisions
Selecting a best-fit partner for a new venture
System design
Functional decomposition for a cross-organization service-oriented application
Standards
Company-wide portal platform at a major financial services company
.NET naming standards at Vistaprint
What's it good for?
Inspired by the Architecture Tradeoff Analysis Method
Flexible and adaptable
"Lightweight"
Don't need to lock everyone in a room
MSDN Magazine Test Run article: "Competitive Analysis Using MAGIQ" by McCaffrey and Koski – inspiration for use of ranking and rank order centroids (Barron and Barrett – 1996).
Motivation: better reflects the inherent prioritization distribution that people have.  Lots of research in this area.
You can't have it all
There are always tradeoffs
Ranking a list of scenarios is too hard


Instead, rank hierarchically in the quality tree
Other approaches:
Linear
Manual (last resort!)
For each scenario/alternative pair, evaluate
Keep it simple
Poor, fair, adequate, good, excellent  works well
And easy to turn into numeric values
Comparing alternatives is ok, if you don’t cheat
The Tools
Defining Quality
Quality Trees
Scenarios
Reasoning
Sensitivities
The Lightweight Architecture Alternative Assessment Method
It's pronounced "lamb", not "lame"
Tradeoffs
Economic Approaches to Measuring Architecture
Value-based Software Engineering

Cost Benefit Analysis Method 

Quality-Attribute-Based Economic Valuation of Architectural Patterns
References
Portfolios
Options
Technical Debt
The Goal
Architecture projects: improving quality without (usually) adding revenue-generating capabilities
The Context
Vistaprint is an online supplier of graphic design services and customized printed products to small businesses and consumers. In the past few years the company has grown very fast. 

Short release cycles for projects bring the necessity for constant tradeoffs between architectural quality and customer facing features.

Flexibility is important, but there is an upfront cost associated with it. How can you help me in convincing my higher management that the cost spent in architecting today will reap us high benefits tomorrow? And how do we decide at where the cost is best spent?
Me
@ Vistaprint
Ipek
Rick
Software Engineering Institute
Quality:
Operability
Developer productivity
The Approach
Evaluate a completed project using:
Tickets that document the work
The codebase, before and after
A dependency structure matrix


Build a model that predicts benefit based on change classification
Dependency Structure Matrix
The Project
Applied
Ticket classification
Time and LOC
Results
Bad News
Why?

Ticket scope wasn't normalized
Ticket classification was wrong
Good News
Easier integration of new users of pricing (decreased coupling to concrete implementation)

Easier integration of new pricing logic (consolidation)

Correctness

New capabilities

Reduced error rate

Improved testability
Anecdotally,
Reset
Effort by LOC by category isn’t informative or architecturally significant

Anecdotal evidence and coarse effort analysis is positive

Consider propagation cost to look for correlation
Start with the DSM, call in M

Compute M , based only on pricing





Compute propagation cost as proportion of non-zero elements of V
Propagation cost = 21/25 = 84%
Results, Round 2
Before
Propagation cost: 15%
During
Propagation cost: 20%
After
Propagation cost: 6%
What did we learn?
It is possible to correlate an architectural characteristic change to a concrete benefit to the organization

Tool support is a must to bring economic techniques to mainstream

Companies have access to architecture level data about their systems, most often created with the motivation of managing refactoring efforts. They just do not know how to make sense out of them for purposes of economic reasoning

Dependency metrics and work reports, such as tickets, are two classes of data that can easily be collected
Cost-Benefit Decision-making
P
Quality-Attribute-Based Economic Valuation of Architectural Patterns

http://www.sei.cmu.edu/library/abstracts/reports/07tr003.cfm
There is no universal goodness
A hierarchy of goodness
Seed with commonly-important quality attributes
Common templates
Totally unrelated to the alternatives being considered
The leaves of the tree are scenarios
-ilities are only management warm-and-fuzzy
Be precise! What does it really mean for a system to have “scalability”? Or “flexibility”?
Context
Stimulus
Response
Outline
Build a quality tree
Rank at each node
Figure out your alternatives
Assess each alternative/scenario
Let a tool do some math
Think hard about the results
Ranking
Alternatives
Assessment
Wrap-up
Architectural Decision-Making
We want to make good decisions early
Reduces risk
Ensures we don't build the wrong thing
Sensitivity analysis

Hybrid solutions

Persistent value in the artifacts

Caveats
What about cost?
We're reasoning about potential
This is usually the easy part
Implies
So
Then
http://www.amazon.com/Value-Based-Software-Engineering-Stefan-Biffl/dp/3642065317/ref=sr_1_1?ie=UTF8&s=books&qid=1280343941&sr=8-1
http://www.sei.cmu.edu/architecture/tools/cbam/index.cfm
Motivate architecture projects in the context of revenue-oriented cost-benefit decision-making
http://technogility.sjcarriere.com/2009/05/11/its-pronounced-like-lamb-not-like-lame/
http://www.sei.cmu.edu/library/abstracts/reports/07tr003.cfm
Important Note
We want to make good decisions, fast
Reliable
5-9's
Maintainable
Fast
Portable
Vistaprint decided to factor out from the existing system all the functionality and features related to “pricing” features as a service
The new pricing element within the Vistaprint system acts as a service to other consumers of pricing information
The effort took longer than the initial estimates
Simultaneous enhancements to old and new pricing
Unexpectedly needed to enhance several other components
While the team believed it to be worth the effort, needed quantifiable evidence to prove it