Send the link below via email or IMCopy
Present to your audienceStart remote presentation
- Invited audience members will follow you as you navigate and present
- People invited to a presentation do not need a Prezi account
- This link expires 10 minutes after you close the presentation
- A maximum of 30 users can follow your presentation
- Learn more about this feature in our knowledge base article
Do you really want to delete this prezi?
Neither you, nor the coeditors you shared it with will be able to recover it again.
Make your likes visible on Facebook?
You can change this under Settings & Account at any time.
Distributed Coordination with Python
Transcript of Distributed Coordination with Python
by Ben Bangert
Things to Watch For
Failure will Happen
Tools of the Trade
Distributed Databases (Cassandra, Riak, etc.)
Distributed Computing (Hadoop, etc.)
Distributed Event Analysis (Storm, etc.)
txzookeeper (use with twisted)
kazoo (use the rest of the time)
unified client works with gevent
implements wire protocol
Store settings in node data
Organize node structure
Set watches on nodes of interest
Join a party, find out who else is around
Elect a leader if desired
See recipe in kazoo
Last for duration of client session
Session dies when closed or expires
Can't have children znodes
Supply node name to create, get node name back with trailing sequential number (0001,0002, etc.)
Can be combined with ephemeral!
zk.create('/node', ephemeral=True, sequence=True)
Create lock parent node if needed
Create ephemeral+sequence node in it, store node name returned
Get children of lock node
Sort children list by last portion (the numbers)
First child in the list has the lock!
Avoid thundering herd, use watches only when needed
When our node isn't the lowest, we set a watch only on the one in front of us
Only the next one in line is 'woken' when is removed (lock released)
Fail fast, fail completely. Session expiration is a good time to sys.exit
Always include jitter (kazoo includes jitter on its connection and command retry operations)
But many of those have one need
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
CA (NOT partition tolerant! (Mostly))
Hierarchical data structure in znodes
Distributed (cluster determines leader automagically)
Znode watches (push notifications of changes)
Ephemeral and Sequential ZNodes