Continuous Deployment

• Web Based Financial Product
• Heavily Regulated
• Millions of dollars under managment
• Deploying to production fifty times a day

or

Based on a real life story

• Code in development matches production
• No need to switch branches

• Quick iterations
• Obsoletes process, e.g. "cutting a release"
• Reduce risk
• No throwing code over the wall

20 minutes

Faster iteration lead to

customer - product market fit

kaChing's

Architecture

Running and accepting

production traffic

UnAannounce

via ZooKeeper

Clients stop calling service

Shut down

Starts with

new version

Self Test

Announce

via ZooKeeper

Production

Traffic

• Automatic rollback with the slightest suspection of something wrong
• Automatically annotate graphsPrefer Business metricsMonitor Statistical deviation, not absolute values

• Standard integration tests
• Running all services and shutting them down in a staging environment
• After each and every commit !
• No Broken Windows !

• Isolate new deployment in production
• Gradually shift load to fresh services

No QA Team

Attracts the right kind of engineers

• Quality driven culture
• Visual and audio signals when anything is broken
• Accountability across the board

Quality by design

Not by process

Deploy Fast Deploy Often

Dozens of Times a Day

Release is a Marketing Concern

Deployment is Engineering's Job

Deployment is reducing code inventory

80/20

• Realize that 80% of what you build will go to waste
• Strive for 80% of the value for 20% of the cost
• Validate before investing more time

Bucket

Hose

Traditional Release Organization

Continuous Deployment

What is a Startup ?

Timeline

5-10 Minutes

1-4 Weeks

Deploy

Release !

Fix P1 Bugs

Stage

Integrate

Patches

Development

Release Cut

Stage

Fix P1 Bugs

Integrate

Patches

Development

QA

Cut a

Release

Automated QA

Testing and Staging

Release train on regular cyclic period

• aka code freez
• Features get integrated into trunk
• Trunk goes on production
• Trunk is the truth

QA Ensures correctness of each train

• Software organized as a tree
• Engineers work on feature branches

P1 Bugs ger patched back into the train

Using Experiments to hide

half baked features

. . .. ....

|

... Monitor...

|

• Startup is a human organization on a quest
• The quest is to find a market fit between a product and customers
• Finding market fit is done through itration and learning from customer behaviur in the real world
• Market fit must be found before the startup runs out of cash

Story Time

What is Continuous Deployment?

• Continuous, successful and repeatable methodology to deploying code
• Automates all steps of taking checked in code and making it run on production servers, used by customers

• Service Oriented Arcitecture
• Vertical sharding
• Coordination using ZooKeeper
• Data interchange using JSON and Protobuf
• Data store in MySql, Redis and Voldemort
• Guice for IoC
• Everything uses the same platform

"Connect Investors with

Outstanding Investment Managers"

Kawala

Functional Query Engine

Inspiered by functional languages (Scala)

Open

Source

Scale and Availability

Less then ten minutes ?

One hour to one day ?

One day to two weeks?

Typical Stack

Two weeks to a month ?

One to six months ?

Six months to a year ?

• Clusered services storage
• Replicated databases (MySql, Redis)
• Caching (e.g. memcached)
• Denormalized Data

More than a year ?

How much time does it take

your new line of code to meet customers?

Testing

• Hudson
• Selenium
• Static analysis (PMD, FindBugs)
• DbUnit
• In memory DB
• Enhenced jUnit Runners

Continuous Deployment

Eishay Smith

Presently: Director of Engineering at kaChing

Past: Principal Engineer at LinkedIn

IBM Research Labs at Almaden (Silicon Valley) and Haifa

@eishay #leanstartup

Life of a Deployment

`

Announce:

An animal lets the ZooKeeper know it woke up and avaliable

UnAnnounce:

An animal lets the ZooKeeper know it is not avalialbe

Stable trunk

• No branch switch
• No merges
• No conflicts

• Managed by a deployment manager
• State is propogated by Zookeeper
• Exponentially increase deployment group size
• Increased monitoring during deployment
• Self Test
• Automated rollbacks

Small, frequent commits

Deploying either using the UI or more commonly using hash tags like "#release:UM" in the commit comment

Forward and Backward compatibility

Automated

Rollback

Steady state

Monitoring Fail

Self Test Fail

Trivial reverts

Read/write to db,

communicate with other services

Please don't send me production traffic

Trivial rollbacks

Immune System

Multi level monitoring

Philosophy

Production Self Test

• Could be perceived as part of integration tests
• Checks a service does before it announce itself

Engineering

• Lines of defense will ever be broken
• Design for swift medigation

We know everything !

Business

aka online KPI

Much much more

(can't disclose)

Continuous Integration

Test Driven Development

Only automated testing matters

!

Empowers engineers to change anything

and scale the team

Test each commit!

If it isn't tested,

it isn't finished or corect

Write testable code

More cost effective then debugging

Embrace abstractions

Facilitates continuous refactoring,

allows the code to get better with age

Culture

• Continuous Deployment at kaChing eng blog: http://eng.kaching.com/search/label/continuous%20deployment
• Deployment Infrastructure for Continuous Deployment (David Fortunato ) http://eng.kaching.com/2010/05/deployment-infrastructure-for.html
• Continuous Deployment at IMVU: Doing the impossible fifty times a day http://timothyfitz.wordpress.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/
• In praise of continuous deployment: The WordPress.com story http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/