Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

Big Data Analytics: Telco domain

No description
by

Thilina Anjitha

on 15 August 2014

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Big Data Analytics: Telco domain

Big Data Analytic: Telco Domain
In real life, most data are Big
Data Avalanche (Moore’s law of data)
We are now collecting and converting large amount of data to digital forms
90% of the data in the world today was created within the past two years.
Amount of data we have doubles very fast
Think about a day in your life?
Use case 2 : Targeted marketing
Collect data and build a model on
What user like
What he has brought
What is his buying power
Making recommendations
Giving personalized deals
Where does it originate from?
Machine logs

Social media

Archives

Traffic information

Weather data

Sensor data (IoT)
What is Big Data? 3V's
Velocity
(how data flow, at high rates)

Volume
(large amounts of data gathered)

Variety
(various degrees of structure)
What is the best road to take?
Would there be any bad weather?
How to invest my money?
Are my servers running healthily?
Should I take that loan?
Is there a way to do this faster?
Which products should I buy?
There are many decisions that you can do better if only you can access the data and process them.
Web does millions of activities per second,
and so much server logs are created.
Social networks e.g. Facebook, 800 Million active users, 40 billion photos from its user base.
There are >4 billion phones and >25% are smart phones.
Observational and Sensor data– Weather Radars, Balloons
hSenied Mobile systems
15m API calls
per month
60m Events
processed per day
6000 SMSes
send per second
Internet of Things (IoT)
Currently physical world and software worlds are detached
Internet of things promises to bridge this
It is about sensors and actuators everywhere
In your fridge, in your blanket, in your chair, in your carpet.. Yes even in your socks
What can we do with Big Data?
Optimize
: 1% saving in Airplanes and turbines can save more than 1B$ each year (GE talk, Strata 2014). Sri Lanka’s total export 9B year
Save lives
: Weather, Disease identification, Personalized treatment
Technology advancement
: Most high tech work are done via simulations

Why Big Data is hard?
How to store?
Assuming 1TB of Disk. It takes 1000 computers to store a 1PB
How to move?
Assuming 10GB network, it takes 2 hours to copy 1TB, or 83 days to copy a 1PB
How to search?
Assuming each record is 1KB and one machine can process 1000 records per sec, it needs 277CPU days to process a 1TB and 785 CPU years to process a 1 PB
How to process?

Convert algorithms to work in large size
Create new algorithms
Why Big Data is hard?
That handles lots of data
Running complex logic
This pushes us to frontier
of Distributed Systems
and Databases
More data does not mean there is a simple model
Some models can be complex as the system
Making Sense of Data
To know (what happened?)
Basic analytic + visualizations (min, max, average, Standard Deviation, distribution)
Interactive drill down
Real-time analytic to track things fast enough and Alerts, Searches(web Search, graph search)
To explain why (Why it happened?)
Data mining, classifications, clustering Understanding underline patterns from data
To forecast (What will happen?)
Neural networks, decision models , Electricity demand forecast, Targeted marketing
Use case 1: Travel
Collect traffic data + transportation data from sensors
Build a model (e.g. Car Following Models)
Predict the traffic. Possibilities on congestion
E.g. divert traffic, adjust traffic lights
Use case 3: Subscriber Churn Prediction
Use case 4: Ad Insertion Engines
Use case 1: Travel
We are at hMS preferred
platform provider for telco
No of Transactions generated by a Sri Lankan Mobile Operator is nearly
50M
per day.
SMS
Voice Call
Data
PRBT
Reload
MMS
Activation
Switch
WAP
GPRS
Regular Charge
e-commerce
Registration
de-registration
Transfer Owner
Sim Delete
Life Cycle
VSMS
Bill Payment
Device Purchase
No of Transactions Per Day = 50M

For a year => 50*365 = 18.25 Billion Transactions

If a single transaction is 3KB, then

Total Data size per
Year
=
50,000,000 * 365 * 3/(1024*1024*1024) =
51 TB
Statistics
hSenid Mobile Big Data Platform
Data Collection
Can receive events via multiple data sources as Files(Binary, ASCII, CSV etc..), XML, External DWH, HTTP etc..
Retrieve via remote location (FTP,SFTP)
Event collection is highly optimized and scalable (500K events collection per Second)
Default Adapters available and you can write custom adapters.
How to store the data?
Relational databases (Scaling is the challenge)
Block data stores -> HDFS
Column oriented -> HBase, Cassandra
Document based -> MongoDB, CouchDB
In-Memory -> VoltDB
How to analyze data?
Two options:
Batch processing
: Schedule data processing jobs and receive the processed data later

Real-time processing
: The queries are executed and the results are retrieved instantly

Batch Processing with Hadoop/Hive : Scheduled campaigns
Batch processing

Apache Hadoop: Map/Reduce processing system and a distributed file system
Batch processing - Data Warehouse
Apache Hive - Hadoop based framework for working on large scale data stores with SQL-like queries
MapReduce/ Hadoop
First introduced by Google, and used as the processing model for their architecture
Implemented by open source projects like Apache Hadoop and Spark
Users writes two functions: map and reduce
The framework handles the details like distributed processing, fault tolerance, load balancing etc.
Widely used, and the one of the catalyst of Big data
Immediate campaigns
(Real-Time Analytic)
Process data as they are
received in streaming fashion

Very fast output

Lots of events (few 100k to millions)
Use case : hMS LAP
Scheduled Based Analysis
Giving a reward for Gold Profile, Teenagers who is calling more than 5 times per month to SAARC countries during week days
Real Time Analysis
When ever Reload event received with the amount > Rs.100 and the user is a Pizza Lover and During last 3 months traveled to Kandy for more than 5 times and stayed with the network for more than 1 year will receive Rs. 2000 worth of Pizza Hut Voucher from Kandy City Center.
hMS Loyalty Analytic Platform
Dashboard and Last Mile : Visualization
Presenting information
– To end user
– To decision takers
– To scientists
Interactive exploration
Web Visualization Libraries
eg: D3, Flot etc
Demo
LAP Dashboard
http://d3js.org/
GapMinder
Parallel Computing
GPU Programming:
A clustering problem with 1 billion 2-dimensional data points, and 1000 clusters, can be processed in less than 30 seconds per iteration with a GPU, compared to about 6 minutes per iteration with our highly optimized CPU version on 8 cores.
“Clustering Billions of Data Points Using GPUs by HP Labs”
FPGA Programming:
Significantly faster for some applications due to their parallel nature.

Dark Side
Privacy
Invasion of privacy
Data might be used for unexpected things

Big Brother
Data likely to used for control (e.g. governments)
Challenges
Speed (e.g. targeted advertising, reacting to data)
Extracting semantics and handling multiple representations and formats
Security Data ownership, delegation, permissions,and Privacy
Making data accessible to all intended parties,from anywhere, anytime, from any device,through any format
Handling Uncertainty
Integration of big data technologies into enterprise landscape
Leveraging cloud computing with
big data storage and processing.
Sample CDR file >>
Full transcript