Continuous Deployment

Eishay Smith 80/20 Realize that 80% of what you build will go to waste Strive for 80% of the value for 20% of the cost Validate before investing more time Deploy Fast Deploy Often What is Continuous Deployment? Continuous, successful and repeatable methodology to deploying code Automates all steps of taking checked in code and making it run on production servers, used by customers Dozens of Times a Day Release is a Marketing Concern Deployment is Engineering's Job Deployment is reducing code inventory Bucket or Hose Traditional Release Organization Continuous Deployment . . .. .... | | | | | | Development Stage Release Cut QA Fix P1 Bugs Integrate Patches 1-4 Weeks Software organized as a tree Engineers work on feature branches aka code freez Features get integrated into trunk Trunk goes on production Trunk is the truth Release train on regular cyclic period QA Ensures correctness of each train P1 Bugs ger patched back into the train Release ! Release ! Cut a Release Stage . . .. .... Integrate Patches | Development | Fix P1 Bugs | | Timeline Timeline Automated QA Testing and Staging | 5-10 Minutes | | Deploy | Using Experiments to hide half baked features Immune System Story Time Based on a real life story 20 minutes Code in development matches production No need to switch branches Faster iteration lead to customer - product market fit Quick iterations Obsoletes process, e.g. "cutting a release" Reduce risk No throwing code over the wall Continuous Deployment Continuous Integration Test Driven Development Culture kaChing's Architecture Service Oriented Arcitecture Vertical sharding Coordination using ZooKeeper Data interchange using JSON and Protobuf Data store in MySql, Redis and Voldemort Guice for IoC Everything uses the same platform Scale and Availability Typical Stack Clusered services storage Replicated databases (MySql, Redis) Caching (e.g. memcached) Denormalized Data Stable trunk Small, frequent commits Deploying either using the UI or more commonly using hash tags like "#release:UM" in the commit comment Forward and Backward compatibility Trivial reverts Trivial rollbacks Only automated testing matters ! No QA Team If it isn't tested, it isn't finished or corect Write testable code Embrace abstractions Empowers engineers to change anything and scale the team More cost effective then debugging Facilitates continuous refactoring, allows the code to get better with age Attracts the right kind of engineers Testing Hudson Selenium Static analysis (PMD, FindBugs) DbUnit In memory DB Enhenced jUnit Runners Life of a Deployment Managed by a deployment manager State is propogated by Zookeeper Exponentially increase deployment group size Increased monitoring during deployment Self Test Automated rollbacks Running and accepting production traffic UnAannounce via ZooKeeper Clients stop calling service Shut down Starts with new version Self Test Announce via ZooKeeper Production Traffic Automated Rollback Self Test Fail Monitoring Fail Automatic rollback with the slightest suspection of something wrong Automatically annotate graphsPrefer Business metricsMonitor Statistical deviation, not absolute values Multi level monitoring Production Self Test Business Philosophy Lines of defense will ever be broken Design for swift medigation Standard integration tests Running all services and shutting them down in a staging environment After each and every commit ! No Broken Windows ! Test each commit! Isolate new deployment in production Gradually shift load to fresh services ... Monitor... @eishay #leanstartup Presently: Director of Engineering at kaChing Past: Principal Engineer at LinkedIn IBM Research Labs at Almaden (Silicon Valley) and Haifa How much time does it take your new line of code to meet customers? More than a year ? Six months to a year ? One to six months ? Two weeks to a month ? One day to two weeks? One hour to one day ? Less then ten minutes ? "Connect Investors with Outstanding Investment Managers" Web Based Financial Product Heavily Regulated Millions of dollars under managment Deploying to production fifty times a day Startup is a human organization on a quest The quest is to find a market fit between a product and customers Finding market fit is done through itration and learning from customer behaviur in the real world Market fit must be found before the startup runs out of cash What is a Startup ? Announce: An animal lets the ZooKeeper know it woke up and avaliable UnAnnounce: An animal lets the ZooKeeper know it is not avalialbe Engineering Could be perceived as part of integration tests Checks a service does before it announce itself We know everything ! Much much more (can't disclose) Kawala Open Source No branch switch No merges No conflicts aka online KPI Quality driven culture Visual and audio signals when anything is broken Accountability across the board Quality by design Not by process ` Functional Query Engine Inspiered by functional languages (Scala) Steady state Please don't send me production traffic Read/write to db, communicate with other services Continuous Deployment at kaChing eng blog: http://eng.kaching.com/search/label/continuous%20deployment Deployment Infrastructure for Continuous Deployment (David Fortunato ) http://eng.kaching.com/2010/05/deployment-infrastructure-for.html Continuous Deployment at IMVU: Doing the impossible fifty times a day http://timothyfitz.wordpress.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/ In praise of continuous deployment: The WordPress.com story http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/