Apache Hama - Streaming and Big Data Processing @ Daum Communications

Apache Hama Edward J. Yoon <edwardyoon@apache.org> Streaming and Big Data Processing Edward J. Yoon @eddieyoon Founder of Apache Hama Committer of Apache Hadoop, BigTop Author of Hadoop Book for Korean Translator of MongoDB, O'Reilly Mentor of Google Summer of Code Who Am I? Open Source Written In Java Under Apache 2.0 License Currently 3 official releases, 6 PMC members and active contributors. What's Hama? Really? Provides a Pure BSP computing engine M/R-like Input/Output Formatter SequenceFile, Text, Accumulo, HBase, .. Hadoop-like Job manager Checkpoint Recovery Pregel-like Graph computing framework Network Weather Monitoring System Characteristics Supports message-passing paradigm style of application development Provides a flexible, simple, and easy-to-use small APIs Enables to perform better than MPI for communication-intensive applications Guarantees impossibility of deadlocks or collisions in the communication mechanisms Compare to M/R and MPI Originally introduced by Valiant A sequence of supersteps Bulk Synchronous Parallel? Processing Big Data with complicated relationships e.g., graph or network. Iterative or Recursive scientific applications Continuous Event Processing So, fit for what? 2008 2012 2010 Accepted to Apache Incubator Introduced BSP World Ready to Graduation to Apache TLP!!! Pluggable RPC Architecture for message transfer e.g., Hadoop RPC, Avro RPC, ..., etc. Message Collector, Bundler, and Compressor to reduce network overheads and contentions Internals public abstract void bsp(BSPPeer<K1, V1, K2, V2, M> peer) throws IOException, SyncException; BSP programming API Pi calculation Sparse Matrix-Vector multiplication K-means Clustering[1] Parallel Support Vector Machine[2] BSP Examples public void compute(Iterator<MSGTYPE> messages) throws IOException; Graph API In-link Count Single Source Shortest Path PageRank Bipartite Matching Graph Examples SSSP Performance A SSSP for a random graph (100 million vertices,1 billion edges) is computed in 30 minutes on 3 Racks cluster Which is the Big Data? See http://codingwiththomas.blogspot.com/2011/10/apache-hama-realtime-processing.html Hama and Stream Processing 1. http://github.com/willmore/hama-kmeans 2. http://github.com/truncs/psvm Real-Time Network Traffic Analysis Analyze user actions and patterns Social Target Marketing PageRank calculation Observe evolution of the Social Networks Detect SPAMs Business Intelligence Extract Hot Trends Could be applied to Daum Services Search, News, Shopping, Yozm, Daum MyPeople, Poll, TV Pot, Maps, ..., etc. 1995 2010 2005 2002 March Information Retrieval Information Suggestion Sexy + Suggestive Who would you like to talk to? See http://wiki.apache.org/hama More Documentation Develop your own BSP applications http://wiki.apache.org/hama/DevelopBSP Q&A? Pregel? @Override compute(Iterator<IntWritable> messages) throws IOException { int currMax = getValue(); .sendMessageToNeighbors(new IntWritable(currMax)); (messages.hasNext()) { IntWritable next = messages.next(); (next.get() > currMax) { currMax = next.get(); } } (currMax > getValue()) { setValue(new IntWritable(currMax)); } { voteToHalt(); } public void if else while this if