NoSQL - Maslow's Hierarchy of Reads (and Writes)

NoSQL - Maslow's Hierarchy of Reads (and Writes) by @adrianco Netflix Cloud Architect (with added discussion of Cassandra monitoring) Self - In Memory Hold for the duration of a request Cache repeated use of the same data Selfish - no sharing, no copies Reads and writes in microseconds No network or disk access Esteem - Shared Memory Share between requests Maintain session state Share across instances Reads and writes in milliseconds One network access, no disk access Belonging - Partitioned Shard across shared memories Improved scalability and availability Reads and writes in milliseconds One or two network accesss, no disk access Safe - Persisted Write to disk, read from disk Write copies to several disks Lots of options - CAP Theorem What happens when it breaks? Most NoSQL support a range of options Would you feel safer with MySQL? Durable Data copied to multiple availability zones Example S3, SimpleDB Distributed Wide Area Multi-Region Support Cassandra's Rack and Datacenter Aware Topology Backed-up Business Continuity Planning Data on tape or copied to a different cloud NoSQL Needs Human Needs Self - In Memory Esteem - Shared Memory Belonging - Partitioned Safe - Persisted Durable - Backed-up Java Hashmap Concurrent Hashmap single memcached sharded memcached hash routed at source http://www.java-tips.org/java-se-tips/java.util/how-to-use-of-hashmap.html http://memcached.org/ Consistent Writes (CP) Guarantee that data is always in sync Refuses writes when it's failed May refuse reads if it can't be sure Membase default behavior Available Writes (AP) Guarantee that writes always succeed Guarantee that reads always succeed Data may not be latest or in sync Inconsistencies are repaired eventually SimpleDB, Cassandra, Riak normal case Apache Cassandra Mark Atwood - My-NoSQL Redux SimpleDB sharded membase read requests vbucket routed at source memcached sharded membase write requests vbucket routed at source, sideways copied and persisted SimpleDB Persists in the Cloud behind the scenes it replicates sideways and persists Cassandra read/write requests (quorum) connection to ring routes to location of data and replicates/reads desired number of copies Cassandra RF=2 Cross Region Cassandra read/write requests (local quorum) connection to ring routes to location of data async read-repair checks that all copies are good async writes to remote non-quorum copies Local Cassandra Remote Cassandra wide area link Cassandra Cassandra Backup and Restore via S3 snapshot, copy compressed SSTables to S3 Restore starts with S3, restores to new Cassandra instances. Full dump and incremental Disclaimers This is a snapshot viewed through a lens April 2010 - expect rapid change... Stuff I have worked with (not all NoSQL) My imperfect understanding of things Stuff I care about - running NoSQL on AWS Comments/corrections welcome @adrianco Monitoring Cassandra Underlying Data Sources Java JMX and Garbage Collection Disk space and performance CPU, Network and Memory Datastax.com OpsCenter Cassandra Specific GUI AppDynamics.com Application, Java and System Metrics http://prezi.com/veagqhsz38u8/nosql-maslows-hierarchy-of-reads-and-writes/ New in 3.2, Cassandra Support (Screenshots by AppDynamics) AppDynamics Dashboard (old version) Single Region Deployment Patterns at Netflix Tomcat based side process Integrates with naming/discovery service Manages token assignment Manages instance replacement Implements full and incremental backups Configuration pattern Replication Factor 3, Quorum writes RackAware Topology mapped to AWS AZ Multi Region Deployment Patterns at Netflix Tomcat based side process Manages mapping to EIP Cross region public/private IP issues Configuration pattern Replication Factor 3, Local Quorum writes Secure inter-node transport Datacenter Topology mapped to AWS Region