Just do less
Access data explicitly and in bulk
Focus on memory
Twitter's Growth
Performance Engineering at Twitter
Evan Weaver
@evan
FE throughput
Usage
Hardware
Let the code make informed decisions
Ruby GC
Tweet Storage
Concurrency
Events are the right model for server workloads
Kiji 0.11
Future Ruby
Ruby 1.8
Single Ruby heap space
Malloc heap
Longlife Ruby heap
Eden Ruby heap
Mark barrier set
Malloc heap
Longlife mark array
Eden mark array
RValues colocated with data on the malloc heap
Longlife context:
AST compilation, constant assignment, singleton assignment

Collects when fixed-size Ruby heaps are full
Move objects? Just don't bother

Collects based on RSS pressure
Pays the mark and sweep cost for unfreeable objects again and again

Collects frequently, just in case
Malloc
TCMalloc for unthreaded throughput, JEMalloc otherwise
Row storage
Hash partitioning,
by Tweet id
Replica 1
Replica 2
Index storage
Hash partitioning,
by user
Replica 1
Replica 2
Double ring;
hash partitioning by
user + timeline
Row cache
Timeline pool
K writes p/s
K reads p/s
K writes p/s
M reads p/s
M writes p/s
100K reads p/s
K writes p/s
K reads p/s
Highly normalized
Storage mirrors the cache
Cache mirrors the query
Repository pattern
★
★
★
★
★
★
★
★
★
LESS
★
★
★
INFO
★
★
★
MEM
★
★
★
EXPL
BULK
★
MEM
★
★
★
MEM
★
★
★
MEM
★
★
★
MEM
★
★
★
MEM
★
★
★
MEM
★
★
★
MEM
★
★
★
MEM
★
★
★
★
INFO
★
★
★
★
INFO
★
★
★
★
INFO
★
★
★
★
INFO
★
★
★
EXPL
BULK
★
★
★
EXPL
BULK
★
★
★
EXPL
BULK
★
LESS
★
LESS
★
LESS
★
LESS
★
★
★
★
INFO
★
LESS