Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

Save the Snake

No description
by

Jennifer Crowe

on 19 January 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Save the Snake

One for each environment Pre Reqs, Problems, Departmental Requirements AMT into Tii Colin - Needs Centos6 RPM's - Tii RPM System builts AMT RPM What needs to be moved into Configs? (TABLE for Snake Aid) Process Canonical names for Hosts Memcache and for SEU (DPR Docs, SEU-HTML, Disk servers, Report view servers, all stuff that points to balancers and failovers, etc. DB's already meet this requirement Need Staging & Production repositories for RPM's Release notes RPM Features --> ensures production configs are stable No more Sudo on Production Hosts outside Ops! Not a small project Colin says everything needs to be rewritten to support configs for real JIRA release tickets should link to successful builds on Jenkins Environments Sprint/Autotest - QA controls deploys via Jenkins Last call -> Ops deploys for testing release instructions/rollback Headless, bare-metal build (very nice to have, not a blocker) Sproutcore CDN - Static turnitin.com for CDN Possibly separate Sproutcore from Palladium PERL This will change Developer workflow Need a system to release to static.turnitin.com Database Currently we have no db change mgmt. We need one. When devs are working on db changes they also need to work on rollback scripts Only the DBA should be working on Production database Need to research options for database Next Steps DB Discussion Next Monday - Rich, Bill, Christopher, Hulse, Christian SL SL AMT --> CentOS6 RH RH SL HW RH/SL SL HW SL RH HW Tiger Team Hire Build Release Mgr. CM We want to fix this Our Tigers Are See your initials on action items above. Our meeting is now WEEKLY Thursdays, 10-11am Status 11/8 RPM's: build #'s vs. release #'s
how should we generate these?
who consumes this information?
we liked timestamps

Need ticket for - Fix post-install script, to retain owner UID

SEU discussion for canonical hosts/configs

Diff sprint/production - need a ticket, add to the Save the Snake project Database 11/8 Goal: we want to avoid missing db changes when we release.
Summary from last time - version'ing schemas, pros/cons. WC API (x3) iOS Pd Tii Funnel DB Averaging about 5-6 schema changes per month, not all associated with sprints tho. It's semi automated currently.

We'd like a system where we have a check in the release process that says "are all the changes needed done?" In all environments, needed in places like Sandbox too,

DB needs to know what changes have been applied.
We need some way to track this.

We talked about this, but decided not to do it:
Dev's declare dependencies and a ticket is written for it that points to the script. Dev also provides a test that it needs to pass (as a primary check).
Track JIRA tickets in the db, to ensure that changes were made. Automate it somehow. For every sql change, the last step is to log the JIRA ticket in the change log as a unique row. Ticket name/date/time/user, etc.
Then write a pre-release check.
DB diffs Sprint/Production
Tii-11533 HW SL/BM/CG, follow up HW SL/HW/BM, follow up, BLOCKER We're Unblocked! Build/Version #'s 11/8 Builds and Versions are not one in the same! 1) Builds# = SVN branch + Revision or Timestamp (SL to define)
Tii ..... Revision #
Pd ..... Revision #
Sproutcore.... Timestamp

2) Versions = Product + Timestamp:
On release tickets --> turnitin_20121105 = tiibranch + ####
More to come on this with the Release Notes draft (HW)
NOTE: We should include AMT once we have it set up. AMT RPM re-Discussion 11/9 Two questions to answer:

1. Steven: AMT should be in its own repo instead of merged with tii (this is probably an easy "yes" decision).
Steven wins! Yay, The build scripts will take care of branching the various repos.

2. Holly: should releasing AMT via RPM be a blocker for the currently staged IH&Jeff AMT issues?
Colin - It's not a blocker. Fred can do what he needs to do.
Will - This is our opportunity to do this. If not now, when?
We originally thought we could do this after Jeffs/IH goes out.
Lots of change going on, additional risk?
We're already working on it, but getting it out in Saving the Snake is another matter entirely.
As far as we are aware, there are not urgent AMT changes needed. Conclusion

AMT will wait until Laverne & Shirley. However, we do not want to lose momentum and it will be a top priority.

We also concluded that we need more time specifically for Sproutcore. Another update: SL is currently setting up P0, Last call environments. Proposed solution - DB Verification Process, Phase 1 It is not a release blocker for Snake Saving, but is already indicated as "Must Have" for Laverne & Shirley (2/12).
Phase 1 (Pre-reqs for Snake Rescue/Antelope Digestion) uncovered additional challenges related to Sproutcore, including the need for additional hardware.
12/3 was too ambitious, but we wanted to at least try to hit that small window between release blackouts.
Working between other release blackouts, Thanksgiving, and Christmas, our next opportunity will be 1/7/13
This change to the schedule should not impact the Laverne & Shirley and MacGyver release dates (2/12 & 3/19) JIRA release tickets should link to successful passing builds on Jenkins!
BUILDS AND RELEASE VERSIONS ARE NOT ONE AND THE SAME
Builds are the specific RPMS. Build versions: Steven to decide: code package name + branch + SVN revision or timestamp
Release versions: productname+year+mo+day, turnitin20121108 - this will be the affects/target/fix version in JIRA
Release notes will include Steven's packages and release instructions- so a set of RPMs and related instructions constitutes the release version
Multiple product release versions can be in a single sprint.
So Laverne & Shirley might be turnitin20130202, writecheck20130202 and ithenticate20130202
Release notes should include the tickets shipping with the release. Database Will need a separate discussion.
For Phase 1 - table to log DB related JIRA tickets.
When the devs submit a script, they should have a step to insert into the table. Rich will have a script to timestamp submissionts. RPM'ing of AMT Current Thinking:Release Notes Standards/Versions Release Date Status 11/13



DONE:

Rich: https://jira.iparadigms.com/browse/TII-11531

Rich reports: Here's the file that needs to be applied to sprint in order to make it the same as prod. Apparently there are several foreign keys and columns (and a table or two) that exist in prod, but are not in sprint...that's bad :/
Other than that, there seem to be several functions that need to be changed in sprint in order to match prod

Richard H/Todd SITEOPS-2491: Staging2 and Staging3 need to migrate to Centos6. Note- these will now be "patch, p0, and lastcall". Steven's working on the build support for these environments

Steven: TII-11532 Fix post-install script


DB: Need larger discussion about db release process/untangling it from dependencies (Will/Christoper to lead that?)


-----------------------------

Per Rich: TII-11533: DB verification process: phase one: table to log db-related JIRA tickets.
The devs, when they submit a script, they should include that as the last step in the script.
So they'd insert into that table with the ticket ID that the script fulfills and on my end I'll have a timestamp to see when they submitted it

We'll need to communicate this as a new submission step to the developers when they submit db substasks.

-----------------------------


Steven: still to do: TII-11498 Tiger Team project: Create repositories for RPMs supporting jenkins changes
-----------------------------


In progress:

HOLLY: TII-11495 release notes standards: current notes

Requirement 1: JIRA release tickets should link to successful passing builds on Jenkins!

BUILDS AND RELEASE VERSIONS ARE NOT ONE AND THE SAME

Builds are the specific RPMS. Build versions: Steven to decide: code package name + branch + SVN revision or timestamp

Release versions: productname+year+mo+day, turnitin20121108 - this will be the affects/target/fix version in JIRA

Release notes will include Steven's packages and release instructions- so a set of RPMs and related instructions constitutes the release version

multiple product release versions can be in a single sprint. So Laverne & Shirley might be turnitin20130202, writecheck20130202 and ithenticate20130202

Release notes should include the tickets shipping with the release.


STEVEN: TII-11426: Project to improve serving of static resources. Steven has begun to separate out the code. Two SiteOps dependencies have been filed for CDN and hardware
SITEOPS-2518: Rename modes for staging# to patch, p0, lastcall

OPS: SITEOPS-2497. needs to get some additional machinery for Lastcall- we're awaiting the new db server to arrive Holly's Notes - Just for the record NOTE: Make sure Fred is aware of what he'll need to do at release time. Database Discussions - Take 3 https://jira.iparadigms.com/browse/TII-11600 We need instructions as to restart, graceful restart, etc.
For config - run a diff and attach it for the record. QA would attach this in the ticket. Colin will make a command line tool to run the diff.
For the record - we want a list of all tickets that went out with a release. It's not the same as all the tickets collected on the dashboard oftentimes. Let's find a way to do this - could be a link to a spreadsheet, list of tickets in a comment, a Confluence page. JC/HW to investigate. Linking out individual tickets is unrealistic. Release Notes Things to address Status 11/15 Static Content Discussions HW SL Pd ...
was OK
Tii ...
/.../ there was a minor change: renaming physical files
Sc ...
Single Build
Static servers for Sc version'ed assets
CDN can point to these static servers
We need a static server and system to redirect to them. SL is still working on this.
Follow ups required, but we're making great progress BM https://jira.iparadigms.com/browse/SITEOPS-2520 Multiple product/codelines dependent on the same db, which all have different schemas etc.
The diff of the db made this evident. We have different structures in each environment. This is bad. We need to document this so we don't go nuts. How do we manage this cleanly moving foward? Outage post mortem, Discuss Centos, Consider P1s, Establish tactical release improvements.
.Make build process consumable by SiteOps for releasing to Production and Staging 90% there
RPM and build/push steps for staging. Create checklist of configs for ops (not including AMT)
Identify places in the hardware where this could break down. Will finish in rolling forward
Define db release process. Still need to finish.

Need a plan for resolving all the diffs between sprint/production db. Database Discussion Take 4 11/20 There's no way to sync up releases reliably.
How to do this going forward b/c everyone depends on the DB?
Make it manageable, everyone needs to know what's where and when.
Possible solutions:
decouple the db?
Registry of db changes. (part of the Save the Snake outcomes. Tii-)
Shared enviroments? Maybe everyone should get their own and Rich is the ultimate gatekeeper?
In JIRA, individual DB Subtasks (any product, any code line)
For any commit to a db subtask, automate roll forward roll back to see if it breaks anything.
Note on JIRA
Code test
Validation step before we continue with a code push. DB tests that need to pass.
When we do this is up for discussion. Ci? RPM? Server restart check? We need someone to figure this out. Working Group - SL, JC, BM Code test ideas, etc
What gets tested?
Where?
How? Need to document this as part of wrapping up Save the Snake/Tiger Process https://jira.iparadigms.com/browse/TII-11533 https://jira.iparadigms.com/browse/TII-11618 Status 11/29 Agenda
Open Tickets Runthrough
SL will roll back to last known production code.
When to roll forward? Monday 12/3
P1 Bugs (node config, etc)
Release Playbook
Last Call
DB Change Process Finalization Repository Discussions Production Jenkins 4 Seasons Production OLD CentOS6 QA RPM's Dependencies? CP CP Dev SVN CentOS Darkpan BUILD RPM's Patch Definition: A patch is... Ops All
Dev
QA Confirms we have Release Candidate
Reboot
SL communicates RPM list
QA writes ticket to Site Ops to communicate push to Last Call, which includes the RPM list & the Build Checklist (including things NOT in the RPM's, ie everything RY needs to do. In this case AMT, DB)
RPM's are moved to Last Call Repo
QA comes up with draft Release Notes (JIRA tickets going with the push)
SiteOps installs RPM's onto Last Call
QA manually spot checks and runs automations against Last Call
Nuke from Space! (Brand new system, blank machine, reformat, reinstall, bare metal)
Release current production code to Last Call
Roll forward!
Reboot
Roll back!
Reboot
Reformat, reinstall from Last Call
Release Go-No Go Discussion happens
If it's GO - QA Creates Push to Production Ticket
Everything looks good to QA
Everything looks good to Ops
Specifically, application runs as expected under all test scenarios, load seems normal, no crazy log messages during and after automations run, etc.
Packages get moved to Production Directory
If it's a NO - packages are NOT pushed to the Production Repo. (We have bigger problems) Last Call Build/Push Process 1/11/13 Post Snake Saving Projects 1/17 AMT - RPMize branched in sync with Pd/Tii (pre Salesforce migration)
WriteCheck Snake
Various Environment Round Up/Clean Up - what do we really need? What can we discontinue?
DB update process next steps
More projects!
Full transcript