MongoEngine + Relational + Privileges = Goodness!

mongoengine-relational mongoengine-privileges Progressive Planning started out as Django + PostgreSQL app (2009) Progressive Planning use case doesn't fit RESTful API through Django-Tastypie Wonderful. but. very. inefficient. Why abandon Django? not at all scalable without hard-coded SQL magic and lowest-level adding to Django and modding of Tastypie no object level privileges no offline working/syncing no `private clusters` for corporates through partitioning forced to optimize the wrong stuff in the wrong domain started to `feel wrong` (aka `got smelly`) everything is JSON (BSON)! deployment is a breeze web-scalable and flexible out of the box (Replica Sets) very active development + growing community `feels` much better and more human(e) Why adopt MongoDB? computers can do more than digitize paper-era-thinking `real` projects are about creative, excited, communicative, opinionated, multi-skilled human beings trying to achieve stuff together. `real` projects are volatile, highly interdependent, multi-constrained, interactive, networked, constantly changing, uncertain, stressful Progressive Planning supports REAL PROJECTS, i.e. YOU. Why Progressive Planning? (not about `resources` with statistical deviations that need to be measured and controlled) NoSQL = NoSchema = No Deal small, transparent, active, mature, well-documented, readable, responsive authors Django is not all bad: it seemed easier to port our existing code to something with similar syntax Why MongoEngine? turned out to be true after we created TastyMongo, mongoengine-relational and mongoengine-privileges for our inherently highly related data we have 2-way relations (1:n, n:m) we need denormalization we need caching we need triggers when stuff changes we don't want headaches Why `mongoengine-relational`? TastyMongo MongoEngine Progressive Planning + Relational + Privileges + TastyMongo = Goodness! Marcel van den Elst no joins! hard-coded brittleness, inconsistency, boiler-plate As it turned out we hardly used any of Django's features and most legacy stuff (non-AJAX tailored) actually got in the way Django-Tastypie is lovely until you need performance manages changes to relations (1:1, 1:n, m:n) automatically updates the other side (atomic saves/deletes). memoizes document fields to monitor differences creates hooks for custom code when relational fields change correctly handles and compares mixed types and states(!) of mongoengine relations only 1 tiny extension to mongoengine's declarative syntax: just using the `related_name` argument sets everything up. mongoengine-relational features transparently caches documents in a thread-safe, self-attaching DocumentCache on the request object setup couldn't be easier: DocumentCache( request ) suffices interact with the cache through `add`, `get`, `remove`, or let the overridden internals take care of it: accessing a related field with <document>.<fieldname> uses the cache bonus: DocumentCache ObjectId strings, un-dereferenced ObjectIds, dereferenced objects, cached objects and DBRefs we need document-level and field-level permissions we couldn't find an existing mixin that correctly and transparently handled object level privileges across relations Why `mongoengine-privileges`? on_change(...) on_change_foo() mongoengine-privileges leverages mongoengine-relational mixin with straightforward API mongoengine-privileges features may_foo(...) we had a RESTful API (TastyPie) that worked like a charm with our backbone-relational framework we LOVED that functionality and separation of concerns! but Tastypie is Django-Tastypie and we ditched Django so we took all the good parts and created `TastyMongo`! Why `TastyMongo`? (except for its gross inefficiency) RESTful API for mongoengine (HTTP conform) very similar to the great Django-Tastypie, but more consistent and simpler API declarative syntax (like Django's or mongoengine's) Django-style cross-relational filtering (`books__author__name`) that constructs the required separate NoSQL queries error handling (without forcing HTML down AJAX throat...) small functions that can easily be overridden where required managed relations between resources (mapping to Documents) Document field introspection and mapping to Resource fields custom URI schemas pagination throttling authentication (de)serialization (JSON, XML, CSV, whatever you write) TastyMongo features because we had no install base and legacy to support again: no joins in NoSQL! Pyramid 'cause `they done things right`! better Request/Response objects (Webob) compared to Django nice traversal API we actually use 90% of Pyramid, and used only about 20% of Django Why `Pyramid`?