Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

Copy of Student Information System Using Hadoop

Soplets is a new way of embedding the model of an application inside the code by just using (Java/.Net) enumerations and annotations. The result: Really transparent, maintainable and manageable code!
by

ravi g

on 22 October 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Copy of Student Information System Using Hadoop

.......code.........................................
......................................................
......................................................
............................code....................
......................................................
......................................................
.................code...............................
......................................................
..................................code..............
{
and this bit of introductory information is just one dot in our overall system...
A
Anatomy of Hadoop
HAD OP
the difference
How goes the working of Master- Slave?
hmm...
this seems more like a one-way route...
B
seems like a cool feature...

hey, with the Soplet Studio at www.soplets.com you can do just that! ;)
This is what soplets.org is about. Let's explore what we can do with it...
C
Hadoop Cluster & MapReduce
There are basically 2 components of Hadoop:
1.) HDFS
2.) MapReduce
Modules Implemented
Visit www.soplets.org!
Try it out!
Discuss with others!
Join the team!
Placement Analysis
Server Log Analysis
Attendance Analysis
Result Analysis
Information
Retrieval
using
Why Hadoop
The Design...
summing
it up...

(coming soon...)
1
3
so what are you waiting for?
Guide
“Prof. C.O. Banchhor”
Student Information System Using Hadoop
Project Review
On
“ Student Information System Using Hadoop”
by
Sinhgad College of Engineering
Department of Information Technology
Student Information System Using Hadoop
Project Area : Distributed Systems.

Sponsorship : In-House Project
Student Information System Using Hadoop
Student Information System Using Hadoop
The Aim of this Project is to facilitate the IT Department with an Information Exchanging System Using Hadoop.
Our aim is to provide information sharing system among the people involved in the I.T. department.
Information shall be stored in the form of small data blocks across various data nodes in the cluster.
Whenever a user requests for information, parallel processing will take place across all the data nodes, and original information shall be obtained.
Problem Definition
Student Information System Using Hadoop
To learn Distributed Computing System.

To enable the IT Department for using Student Information Exchange System.
Objective
The need for an Information Exchange System has gained much importance.
A system that would provide easy storage for large amount of data is on great demand.
Student Information System Using Hadoop
Need of the Project
`
Student Information System Using Hadoop
General Scenario: Traditional System
Student Information System Using Hadoop
Student Information System Using Hadoop
Disadvantages of Traditional System
Difficulties of developing distributed System
Networking Problems
Security Problems
Computing Power
Requires Structured data
In Normal Scenario
Student Information System Using Hadoop
Hadoop Distributed File System
2
Imagine How Big data we can store
& Perform Parallel processing on that Big data
Student Information System Using Hadoop
Hadoop Cluster
Student Information System Using Hadoop
Map Reduce Model Working over a Cluster environment
Example of Map- Reduce
Student Information System Using Hadoop
Placement Analysis
Server Log Analysis
IP Address Hits
Error Analysis
Attendance Analysis
Calculating Average Attendance
Summarizing the Attendance
Result Analysis
Information Retrieval
Prize Analysis
Student Information System Using Hadoop
The input to the system will be a file having records of all students of BE containing the roll no. and company in which he/she is placed.

The system will give the report containing the name of the company and the no. of students placed in that company.
Module 1 Placement Analysis
Student Information System Using Hadoop
A server log is a log file (or several files) automatically created and maintained by a server of activity performed by it.

Log files contain too much information, and often lead to massive headaches.

The IP Address hits give the number of times a particular user accesses the server.

Sometimes error occurs during the access to certain system resources. Error Log Analysis finds such errors from the log file and their number of occurrences.
Module 2 Server Log Analysis
Student Information System Using Hadoop
The input to the system will be a attendance of each and every subject.
Based on the attendance of each subject average attendance is calculated.
From the average attendance no. of students in the defaulter list is calculated.
Module 3 Attendance Analysis
Student Information System Using Hadoop
The information belonging to the same student is collected from multiple directories and it is summarized into 1 file as result.
Module 5 Information Retrieval
Student Information System Using Hadoop
Classes Used in Each Module:

Mapper

Reducer

3. Driver
Coding…
Student Information System Using Hadoop
Implementation Details
Student Information System Using Hadoop
Setting The Stage
Collect configuration information.
Run the Job.
Student Information System Using Hadoop

Driver{ //Collects Configuration Information

Define Container(i.e. Object of JobConf class) to collect CI
Data types identification
M & R function Identification
Specifying i/o paths
Submit the job & wait.

}
Driver Class
Student Information System Using Hadoop
Reducer: Reducing is often used to produce “summary” data.

`

reducer (word, values):
sum = 0
for each value in values:
sum = sum (Operator) value
emit (word, sum)
Reducer Class in Module:
Student Information System Using Hadoop
Mapper: Transforms each elements individually to an o/p data elements.

mapper (filename, file-contents):
for each word in file-contents:
emit (key, value)
Mapper Class in Module:
Student Information System Using Hadoop

No values stands on its own, every value has a key associated with it.

Both functions receive (key, value) pair.
Basic Concept used in implemented MR Modules:
Student Information System Using Hadoop
Steps Implemented:
Install Sun java (JDK).

Install SSH (OpenSSH).

Disabling IPv6

Hadoop Installation & Configuration

Formatting the HDFS file-system via the Namenode

Start execution
Configuration of Hadoop
Student Information System Using Hadoop
A couple of computer systems with same configurations as far as possible.

Hadoop Software framework by Apache.

Linux operating system Ubuntu 11.10.

Java Development Environment.
Hardware and Software Requirements
Full transcript