Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Big Data

What is it? Where does it come from? How can we use it? So what!?
by

Vizwerx Group LLC

on 8 November 2012

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Big Data

Receivables Accts Payable Inventory Locations Employees Management Action Direction Delivery Executive Planning Where does Big Data come from? Hundreds of systems and databases Billions of smart tags, sensors, meters & other transmitters Billions of Digital devices, applications & "conversations" vs Modelling
Forecasting
Predictions
Trends
Profiling
Fraud Detection
Software Quality Market Positioning Decisions Understanding
Customers Adding color to the B&W digital landscape Competitive Analysis Strategy Development Improving Production Quality Identifying Waste Understanding Consumer Preferences Identifying Consumption Patterns Food Safety Distinguising Signals from Noise Product Decision Support Executive Planning Data Driven Organization Management Direction Direction Execution Delivery Action Delivery 340 M "tweets" per day RFID sensors surveillance devices observations (internet blogs, ratings, reviews) Software and hardware & software log files smart meters set top boxes, digital video recorders, xbox, etc. click data cable modems/home network devices internet connected tvs text text text over 1 Billion Social Media users worldwide 4.1 B text messages per day in the US 600M to be installed by 2016 visualization servers transactional servers transactional servers Social Media Mgt Cloud Management Product (and other business) Decisions Disruptor Leap Frogger Trend Setter Steady Eddie Fast Follower One Step Ahead Market Leader Dominator M ar ket Integration & Transformation Cloud Data Cloud Analysis Cloud Strategic Thinking
&
Planning Planning (worldwide inspection results & analysis available) Surfaces
Indicators
Trends
Patterns
...
...
Relationships vs Disruptor Leap Frogger Trend Setter Steady Eddie Fast Follower One Step Ahead Market Leader Dominator M ar ket Data Driven Decisions Focus groups, surveys, contests, promotions, campaigns, etc. Customer Calls, on-line ratings, automated interactions, etc. Smart homes, smart appliances, etc. big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools White, Tom (10 May 2012). Hadoop: The Definitive Guide. O'Reilly Media. p. 3. ISBN 978-1-4493-3877-0. [1] [1] Big Data Infrastructure (highly parallel) The dreaded data knot transactional systems unmanageable transaction and log files transactional systems transactional systems Where does BIG DATA come from? Internet connected appliances
(electicity usage, maintenance, filters, etc) Online shopping list (check prices, availability, consumption data, coupons, etc) A connected planet Financials Human Resources Procurement Customers Products Products Customers Billing Employees Locations Skills DB Equipment Trouble Tickets Maintenance
Contracts Network Servers Email Proposals Pricing Telecom Intranet Prospects Reporting 2 KB of data on each tag (approx one typewritten page of information) 2.5 quintillion bytes per day From the beginning of time until 2003, about
5 billion gigabytes of data have been created. The tools can handle the data but . . . . e-bill pay Click to chat 300 M call records per day 350 billion meter readings per year 12.5 Terabytes of tweets per day time volume "1 of 3 business leaders do not trust the information they use to make decisions" - International Business Machines Global Reach Functions Connected Data Discipline Aligned Targets Trends Trends Trends ( A few examples ) GPS The output can be overwhelming (examples) Able to use complex analysis to Visualize Possibilities Possible pitfalls of not using Big Data Possible benefits of using Big Data only transactional systems Many Input/Output systems Big Data Architecture Master Data Management Reporting Mgr EXAMPLE Mgr Anal Store Source and Store Social Cont Ingestion
Process Pattern Recognition
Trend Identification
Examining Scenarios
Big Picture Data Views Micro/Macro Workflow Message Transformation Federated Content Search Federated Query Manager Social Media Connect Metadata Services Discovery & Profiling CEP ETL Message Classification Advanced Analytics Analytic Collaboration Enterprise Search Text Analytic Natural Language
Processing Visualization Mashup Engine Content Classification Map Reduce Security &
Governance Basic Analytics Business Intelligence Analyze & Classify The tools aren't up to the task but they create ouput that makes it feels like ..... Reporting Engine Overload Data smudging, blurring & leaning Database meltdown Relational Databases Traditional Data analysis tools Reports Charts Graphs Data analysis tools Relational Databases Big Data means different things at different levels of the organization What can we use Big Data for? Big Data Lots of Data Unleashing the Power of Big Data Unleash the power in your organization Receivables Accts Payable Inventory Data Data Data Mgr Mgr Overlapping Targets Disconnected Targets Disconnected Targets Raining Data No data discipline Trillions already deployed! (and growing) Loosely connected
systems Operational gymnastics required to manage Super human effort required Other possible uses for Big Data Inventory Management Sensor clusters Trillions already deployed Operations Engineering Disconnected Functions and Departments (but little actionable intelligence) Mostly tactical thinking & planning why? why? why? why? why? I get it transparency visbility BIG DATA What is it? Where does it come from? What is it used for? BIG DATA So what? (security cams, traffic cams, . . .) Perform location, inventory, ance, Security
software Database
log files Configuration
Settings User
activity
logs Financial transactions Customer
Equipment
Readings Social
Media
Traffic Customer
Comments (and growing) Sources Processing Outcomes (BIG DATA) (Lots of data) (and growing) (and growing) Boundary & Exception management Hot Spot Management Collaboration Oct 2012 Sep 2012 more more Strategy Data Driven Data Driven Executive Using ALL the data to support decision making other silos of data The output isn't usable Optimal Path Provide Funding
Decision Support
Insights
Intelligence
Strategy
Targets
Compliance Responsibilities & Functions (Executive) Responsibilities & Functions Responsibilities & Functions
Justify Funding
Oversight
Supervision
Translation
Interpretation
Recommendations (Management) Capturing
Processing
Designing
Developing
Managing
Maintaining (Execution level) - current and historical - Traditional Tools and Databases Traditional Tools and Databases Device Data Sensors from Data from Tags Data from Cameras Media Blogs Feeds Customer reviews Blogs Blogs Blogs Data Transactions Transactions Transactions Database logs logs Secu streams ent Mgr Data Hadoop File Content XML Transa Data ctional rity ytic In Mem Object ory Ingestion
Process "90% of all the data in the world was created in the last two years!" - IBM website Fields - Paul A. Laudicina, Wall Street Journal
Oct 2012 In 2013, 5 billion GB of data will be created every 10 minutes! Data Mining and Federated Searches EXAMPLE EXAMPLE EXAMPLE Content Streams Social Media Feeds Blogs Sensor
Data BIG DATA Traditional Tools transparency visbility more vs (Lots of data) What happens if we try to use traditional tools? (All available work order data too voluminous to be useful) Large scale collaboration not realistic work orders work orders work orders Estimates are that Social Media has created a situation where there are only 8 degrees of separation between every person on earth! EXAMPLE Customer Care
Advisor System (including historical info from all Customer interaction sources) tweets text msgs email FaceBook Posts Blogs trouble tickets (virtual collaboration) All Customer interaction data calls Transacational servers Customer Driven - Strategy A - Strategy B What is Big Data? big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools White, Tom (10 May 2012). Hadoop: The Definitive Guide. O'Reilly Media. p. 3. ISBN 978-1-4493-3877-0. [1] [1] data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data forecasting modeling weather atmospheric telecom & entertainment air traffic control & other surveillance (artifical groups) (bars too small) (artifical ceiling) Backup > 24 hrs Searches won't run Content not relational or hierarchical dataset size > DB spec Trends and Cluster Identification Map of Cholera Deaths in London in the 1840's Control Systems data data data data data data data data data data data data data data data data data data data Eat at Joe's Exit now! Volumes Greater Greater Greater Variety Velocity Than ever before!! BIG DATA BIG DATA BIG DATA BIG DATA BIG DATA BIG DATA LOTS of DATA LOTS of DATA LOTS of DATA Hadoop Distributed File System data data data data data data data data data data data data data data data data data data data data data data data Shin-Tomei Expressway, Japan April 2012 Volume data data data data data data data data Online shopping list (check prices, availability, consumption data, coupons, etc) data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data data Variety What is Big Data? What makes BIG DATA challenging? Hadoop: The Definitive Guide, Tom White 2009 Figure 2.1 - MapReduce logical data flow Integrate & Transform (high level) * * Possible BIG DATA output (Can't take advantage of all the available data) (imagine being able to plan your next move with the help of insights from every move of every chess game ever played!) (business transactions and functions) Operations (too large to handle with traditional tools) (formats, structures and protocols that
don't work with traditional tools) vs. LOTS of DATA Signal Mapping Velocity (arriving too fast to handle with traditional tools)
Full transcript