Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

Distributed Coordination with Python

No description
by

Ben Bangert

on 24 July 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Distributed Coordination with Python

Distributed Coordination
with Python

by Ben Bangert
Coordinating
Recipe: Lock
Basic Steps
Things to Watch For
Failure will Happen
Sequential Znodes
Party Membership
Ephemeral Znodes
Configuration
What isn't
coordination?
Python Clients
Apache Zookeeper
Tools of the Trade
Distributed Databases (Cassandra, Riak, etc.)
Distributed Computing (Hadoop, etc.)
Distributed Event Analysis (Storm, etc.)
txzookeeper (use with twisted)
kazoo (use the rest of the time)
unified client works with gevent
implements wire protocol
Store settings in node data
Organize node structure
Set watches on nodes of interest
Join a party, find out who else is around
Elect a leader if desired
See recipe in kazoo
Last for duration of client session
Session dies when closed or expires
Can't have children znodes
Supply node name to create, get node name back with trailing sequential number (0001,0002, etc.)
Can be combined with ephemeral!
zk.create('/node', ephemeral=True, sequence=True)
Create lock parent node if needed
Create ephemeral+sequence node in it, store node name returned
Get children of lock node
Sort children list by last portion (the numbers)
First child in the list has the lock!
Avoid thundering herd, use watches only when needed
When our node isn't the lowest, we set a watch only on the one in front of us
Only the next one in line is 'woken' when is removed (lock released)
Fail fast, fail completely. Session expiration is a good time to sys.exit
Always include jitter (kazoo includes jitter on its connection and command retry operations)
But many of those have one need
in common!
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
Cap theorem
CA (NOT partition tolerant! (Mostly))
Hierarchical data structure in znodes
Distributed (cluster determines leader automagically)
Session based
Znode watches (push notifications of changes)
Ephemeral and Sequential ZNodes
Full transcript