Elasticsearch meetup

Agenda inSided Old platform issues New platform wishes Finding the right solution Architecture Purposes Multi tenancy Usage example(s) IQL (inSided Query Language) MySQL & ES synchronisation inSided facilitates interaction with and among customers About me @woutneyn wouter@insided.nl Co-founder inSided, Techlead 31 yrs old Studied Information technology in Amsterdam Programming since 12 yrs old (VB, C, C++, ASP, etc.) Pocketinfo.nl / Mobilyz.com '99 During college period; told first employer (.com hype): 'sure I am an expert PHP programmer'. Bought a book while biking back home. Started 2 days later as one of 6 devs. And here we are. A practical aproach.. 2 years old | SaaS & Services | Social CRM | Growing | Scalable | API Platform after 10 years 10 year legacy Not DRY (Don’t Repeat Yourself) Scattered code Not well documented Literal search and no ranking Our own hardware NEW PLATFORM WISHES Customer/partner self service Extensive logging capabilities No single point of failure Decouple components Flexible and scalable (no limits) Real time data Easy (but extensive) search capabilities Multi language & customer Secure (ISO27001) Architecture AWS to the rescue New platform needs: better data storage Finding the datastore/search solution ES setup on AWS Purposes e 3. Debugging 1. Events site_pageview topic_create message_send topic_merge Each event can hold ‘custom’ data 2. Search Topic (Ngram) Post (Ngram, parent connection to Topic) User (Edge NGram) Tag (Edge NGram) 4. Logs Scalable Flexible Queues ISO27001 Decoupling Secure Virtual hardware Apache, PHP, MySQL What Where When How .. To make it truly scalable, flexible? 'Hell.. they wrote the books!.. they should know !' 'Where are your servers? - Dunnow.. somewhere in Ireland.' noSql Need to (be able to) store everything: Multi-customer Multi-environment Multi-customer-environment Easily add (new) data (types) Realtime availability: Analytics Relations: not needed MySQL stays master: stable and in-house experience Easy search A-synchronous storage (next to MySQL) Must run on AWS 'So.. we need to store LOADS of data and have it always available. Fast.' PM jumped on board.. User (push) notifications Community analytics Activitystream API Calculations They talked to our customers.. Search CRM Segments (retail, telecom, etc.) Badges (rewards) Scalable filters Realtime Easily extendable - Config, Backups, Multi-tenancy, AWS + Costs, Performance, Query, PHP Client - Config, Backups, Query (doable) + Costs, AMI install, Multi-tenancy - Costs, Multi-tenancy, Query, PHP Client (AWS PHP Lib) + Monitoring, Scaling, Backup, Performance (Werner wants our €) - Backup, Query + Scaling, Multi-tenancy, Config, - Multi-tenancy (costly), Setup indexes, Query + PHP Client, Scaling, Costs + Costs + Scaling + Backups (S3 via plugin) + Performance + Setup & config + Query - PHP Client '...but it's specifically for search..' U storing data? - Yes U search data? - Yes All green? - Yes Use it! Multi tenancy One index for events per customer /{customer_id}/{year}m{month} localhost:9200/insided/2013m01 Search: One index for each app per customer /{customer_id}_{appname}/{contenttype} localhost:9200/insided_forum/topic Mappings Hardcoded in main config Overridable per customer per type (if a customer needs additional analyzers) Is set automatically if index does not exist Default index Type index with parent (topic) Examples of the end result: ElasticSearch & inSided platform site_pageview topic_create topic_create message_send post_create App SQS App ElasticSearch Some more problems Devs @ customers might not get the complex ES queries IQL (Insided Query Language) Information might be incorrect (because it's out of sync) `lastchanged` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP Syncing every X seconds. Per customer per application PHP Daemon AMI with the current version of ES Autoscaling by workload. Up and down, creating extra servers, or removing them. ES Security Group, which is only accessible from our instances and our office IP ES S3 bucket - Flexibility, Config, Scalable, Performance (while indexing) + Costs, Query, PHP Client Questions..