MongoNYC

Big Fat JSON Pipeline {_type: "stream", _id: ObjectId("xxxx"), game: {$ref: "game", $id: ObjectId("zzzz")}, scorer: {$ref: "user", $id: ObjectId("uuuu")}, events: [{$ref: "event", $id: ObjectId("yyyy")}]} GET /stream/xxxx POST /stream/xxxx/save DELETE /stream/xxxx REST(ish) API JSON Python Dictionaries Django Models Stupid SQL Relational Model The Original Stack 16,000 Queries. ...optimization... "Only" 500 Queries. WTF? play play_participant play_types play_properties play_participant_role play_property_type sport AHA! {_type: "event" _id: ObjectId("aaaaaaaa"), participants: [{player: {$ref: "player", $id: ObjectId("bbbbbb"), roles: ["pitcher"]}], code: "SB", properties: {...}} Performance != Scaling - marshalling - calculations - "business logic" Cache Everything! Denormalization Rules obvious choice: memcached non-obvious option: MongoDB? Now This is Caching {_id: "/stream/xxxxxxxx", json: "{...}", expires: "Fri May 21st...", dependencies: ["/team/yyyyyyy", "/player/zzzzz"]} db.cache.ensure_index(("dependencies", 1)) db.cache.remove({"dependecies": {"$in": ["team/yyyyyyy"]}) yay! R W oh... Original GameStream <meta http-equiv="refresh" content="60" /> Entire page processed through Django templates. Optimization Strategies Baseline - 9 seconds for 9 inning game. Optimizing data processing - 3 seconds. Cache on read - 3 seconds on cache miss, 50 ms on cache hit. High enough volume, lots of people will experience a cache miss. Long Polling Using AJAX to retrieve data. If there is data, return it immediately. No data, leave the connection open and pend. When data appears, send it and close the connection. Client processes and then immediately opens a new connection to wait again. $.ajax({ url: '/push/stream/1234?index=10, dataType: 'json', type: 'GET', error: ajaxError, timeout: 0, success: processEvents }); Wait New data comes in Return JQuery + AJAX Tornado "push" Cheap reads $.ajax({ url: '/push/stream/1234?index=0, dataType: 'json', type: 'GET', error: ajaxError, timeout: 0, success: processEvents }); Returns immediately Tornado and Nginx 1000s of Open Connections NGINX epoll > Apache New GameStream Cache on write Where does MongoDB fit in here? - Denormalization - Caching - Write Speed Caveats Mobile Browsers Internet Explorer Learning MongoDB {_id: ObjectId("xxxx"), event_block: "...lots and lots of json...", save_date: "2010-05-21 12:08:00", stream_id: ObjectId("yyyy")} ensure_index([("stream_id", 1)]) find( {'stream_id': self.stream_id, 'save_date': {'$gt':self.last_event_block_save} } ).sort('save_date', 1) ensure_index([("stream_id", 1), ("save_date", 1)]) $.ajax({ url: '/push/stream/1234?index=11, dataType: 'json', type: 'GET', error: ajaxError, timeout: 0, success: processEvents }); Synchronous Polling Still Fast Kiril Savino, @kirilnyc, kiril@gamechanger.io Jerry Hsu, jerry@gamechanger.io