Loading presentation...

Present Remotely

Send the link below via email or IM


Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.


Elasticsearch meetup

No description

Wouter Wout

on 10 January 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Elasticsearch meetup

Agenda inSided
Old platform issues
New platform wishes
Finding the right solution
Multi tenancy
Usage example(s)
IQL (inSided Query Language)
MySQL & ES synchronisation inSided facilitates interaction with and among customers About me @woutneyn wouter@insided.nl Co-founder inSided, Techlead
31 yrs old
Studied Information technology in Amsterdam
Programming since 12 yrs old (VB, C, C++, ASP, etc.)
Pocketinfo.nl / Mobilyz.com '99
During college period; told first employer (.com hype): 'sure I am an expert PHP programmer'.
Bought a book while biking back home.
Started 2 days later as one of 6 devs.
And here we are. A practical aproach.. 2 years old | SaaS & Services | Social CRM | Growing | Scalable | API Platform after 10 years 10 year legacy
Not DRY (Don’t Repeat Yourself)
Scattered code
Not well documented
Literal search and no ranking
Our own hardware NEW PLATFORM WISHES Customer/partner self service Extensive logging capabilities No single point of failure Decouple components Flexible and scalable (no limits) Real time data Easy (but extensive) search capabilities Multi language & customer Secure (ISO27001) Architecture AWS to the rescue New platform needs: better data storage Finding the datastore/search solution ES setup on AWS Purposes e

3. Debugging 1. Events site_pageview
topic_merge Each event can hold ‘custom’ data 2. Search Topic (Ngram)
Post (Ngram, parent connection to Topic)
User (Edge NGram)
Tag (Edge NGram) 4. Logs Scalable Flexible Queues ISO27001 Decoupling Secure Virtual hardware Apache, PHP, MySQL What
To make it truly
scalable, flexible? 'Hell.. they wrote the books!.. they should know !' 'Where are your servers? - Dunnow.. somewhere in Ireland.' noSql Need to (be able to) store everything:
Easily add (new) data (types)
Realtime availability: Analytics
Relations: not needed MySQL stays master: stable and in-house experience
Easy search
A-synchronous storage (next to MySQL)
Must run on AWS 'So.. we need to store LOADS of data and have it always available. Fast.' PM jumped on board.. User (push) notifications Community analytics Activitystream API Calculations They talked to our customers.. Search CRM Segments (retail, telecom, etc.) Badges (rewards) Scalable filters Realtime Easily extendable - Config, Backups, Multi-tenancy, AWS
+ Costs, Performance, Query, PHP Client - Config, Backups, Query (doable)
+ Costs, AMI install, Multi-tenancy - Costs, Multi-tenancy, Query, PHP Client (AWS PHP Lib)
+ Monitoring, Scaling, Backup, Performance (Werner wants our €) - Backup, Query
+ Scaling, Multi-tenancy, Config, - Multi-tenancy (costly), Setup indexes, Query
+ PHP Client, Scaling, Costs + Costs
+ Scaling
+ Backups (S3 via plugin)
+ Performance
+ Setup & config
+ Query
- PHP Client '...but it's specifically for search..' U storing data? - Yes
U search data? - Yes
All green? - Yes
Use it! Multi tenancy One index for events per customer /{customer_id}/{year}m{month} localhost:9200/insided/2013m01 Search: One index for each app per customer /{customer_id}_{appname}/{contenttype} localhost:9200/insided_forum/topic Mappings Hardcoded in main config

Overridable per customer per type (if a customer needs additional analyzers)

Is set automatically if index does not exist Default index Type index with parent (topic) Examples of the end result:
ElasticSearch & inSided platform site_pageview topic_create topic_create message_send post_create App SQS App ElasticSearch Some more problems Devs @ customers might not get the complex ES queries IQL (Insided Query Language) Information might be incorrect (because it's out of sync) `lastchanged` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP Syncing every X seconds. Per customer per application PHP Daemon AMI with the current version of ES

Autoscaling by workload. Up and down, creating extra servers, or removing them.

ES Security Group, which is only accessible from our instances and our office IP

ES S3 bucket
- Flexibility, Config, Scalable, Performance (while indexing)
+ Costs, Query, PHP Client Questions..
Full transcript