Introducing
Your new presentation assistant.
Refine, enhance, and tailor your content, source relevant images, and edit visuals quicker than ever before.
Trending searches
Ansible --turbo
38
- Thundering herd of thousands of hosts
- API starts throwing errors
- Tasks fail
- Plays leave hosts behind
- You have a bad day
- Too much competition for the same resource
- Everything slows down
- Tasks fail
- Plays leave hosts behind
- You have a bad day
- Each task has a cost
- Costs add up faster in large environments
- Lots of quick tasks overwhelm the calling system
- Diminishing returns on more forks
Getting Content
Image Credits
What's going on?
Patric -- memgenerator.com
2x4 -- The Internet (jbrunningink.blogspot.com)
sanity -- duplexchick.com
back off -- funnycats4u.blogspot.com
firewalls -- Rackspace, Inc.
too many poeple -- AFP (via golfnews.com)
too many bicycles -- The Internet (jokeroo.com)
english muffin -- The Internet (miataturbo.net)
swimming cat -- The Internet (theberry.com)
banana -- thechive.com
random house -- Random House, Inc.
Reducing Task Count
Spread the load with mirrors or torrents
Who am I?
What's going on?
Jesse Keating
Linux Systems Engineer IV -- Rackspace, Inc.
Next gen Cloud Servers product
Deployments/Automation team
twitter: @amjkeating
freenode: jlk
github: https://github.com/j2sol/
email: jesse.keating@rackspace.com / jkeating@j2solutions.net
Be Careful!
Push the work to the nodes with custom modules
APIs with Limits
Custom Modules
- Group together sets of tasks that are all related and all form a general ok/fail/change block
- Allows more hosts to get the set of actions to accomplish before sending results back
- Care less about intermediate task results and more about body of work as a whole
- Also keeps complicated logic out of playbooks
What's going on?
Use serial mode and pause
- Serial mode to batch how many hosts you want to run in parallel for that task
- pause module to clear any limit there is on X number of API calls within Y seconds/minutes
Ansible at high speed is Easy!
This basic stragegy works great for small environments.
Inventory Management
Random Observations
What's going on?
- Running inventory script with --host for each host
- Looking for vars files for each host
- all.yaml
- group
- group.yam
- group.yaml
- group-hosts
- group-hosts.yml
- group-hosts.yaml
- hostname
- hostname.yml
- hostname.yaml
Use the _meta key and pre-populate all the host vars.
- Generate all the vars in one execution
- Ansible skips calling the inventory script for every host
- Saves 10s of minutes
- Beware of ulimits when using lots of forks
- Beware of MaxStartups and delegation
- Beware of known_hosts growing too large
But what happens in a large environment?
Ansible --turbo: Keeping Ansible fast in large scale environments
This is our firewall
We can haz water?
after torrenting
Back off!
Too many cycles
Rackspace Public Cloud
Real world experience at running Ansible on large scale environments.
A look at challenges and ways to meet those challenges.
Doing the same thing over and over again...