Send the link below via email or IMCopy
Present to your audienceStart remote presentation
- Invited audience members will follow you as you navigate and present
- People invited to a presentation do not need a Prezi account
- This link expires 10 minutes after you close the presentation
- A maximum of 30 users can follow your presentation
- Learn more about this feature in our knowledge base article
The Integrated Data Hub: The Next Generation Data. Warehouse
Transcript of The Integrated Data Hub: The Next Generation Data. Warehouse
SYSTEMS THE INTEGRATED DATA HUB: THE NEXT GENERATION DATA WAREHOUSE Staging
Layer Orchestrator Business Rules
Engine Archiving Data
Vault IDH Concepts Hub Logical
Work Services Metadata
Canonicalization The Data Warehouse Schism
Inmon and Kimball The New Data Warehousing‘s Challenges The Integrated Data Hub Defined
The Integrated Data Hub is a Hub of Integrated Data.
A Hub because, it is the place where all the data of the enterprise, that have to be validated by corporate business rules will have to go through.
A Hub because, the IDH communicate with the surrounding environment in every direction. West with the source systems, North with the Service layer, East with the target system, mainly Business Intelligence but also other operational systems and South with the Metadata layer.
Integrated, because, the main purpose of the IDH is to give the user a single version of the truth. You will thus find all data related to a business concept in a single place.
We could add to the concept of Integrated Data, those of Non Volatile, Time Variant and Subject Oriented, to stick with the Inmon’s definition of the Enterprise Data Warehouse.
The Integrated Data Hub is reference architecture for Data Integration but it is also a set of concepts, development framework, best practices and other technical considerations.
This Data Integration Reference Architecture is based on different layers, each one having its own and only purpose.
In addition to architectural considerations, you will find behind the IDH concept a set of guidelines and best practices to ensure that this architecture is built to answer the future challenges of Data Integration.
This architecture was thought to be most relevant to when:
Your EDWH has to be fault resistant
Your EDWH is supposed not to be impacted by changes in the surrounding environment, and particularly when new sources are added frequently or existing ones change often.
Business rules are subject to change over time.
Auditability is required.
The structure of arrival data is not always known.
Historization and time slices are required. The Message Oriented Layer
Another important concept of the IDH. The ability to the Data Hub to communicate in all direction using canonical data embedded into messages. Meta Data Driven Automation
This concept would allow avoiding the use of traditional ETL engine. The idea here is to describe the transformations you want to see applied to the data, and the orchestrator service will do the rest. Business Ownership and Natural Language
Another important concept is the fact that all business rules are described in a natural language by the owner of the business rule, normally the business. These business rules must be historized in order to reload the EDW with a correct picture of the past. The Metadata Driven Automation service will manage the application of these business rules. The Staging Layer
You will find here mainly a copy the source systems tables and a delta load of the data needed to refresh the EDW. Small transformations can occur in a second phase of the staging layer. The Core Data Warehouse Layer
This is where the data integration and the decoupling with the source systems are achieved. The Data Model is mainly business oriented. Historization and data lineage are performed here also. The Presentation Layer
This is where you will find the data marts designed to answer business questions. The Access Layer
Mainly composed of a corporate semantic layer. The Metadata Layer
This is where we record every action done on and with the data going through the IDH. The main purpose of this layer is to allow full data lineage and impact analysis. The Service Layer
This is one of the main concepts behind the IDH. You will find the MDM engine, The Business Rules Engine, the Orchestration services, etc. The Operational Data Store Layer
Here you will find a complete historization of the staging layer, in order to provide a valid source to reload the data warehouse and to perform operational reporting. The Integrated Data Hub:
The Next Generation Data Warehouse IDH The Logical Unit Of Work
This is the smallest part of code that is allowed to be written by an ETL developer. It must be designed keeping three main concepts in mind: Restartability, Parallelism, and Orchestration. A Logical Unit Of Work is supposed to do only data movement from one border of a layer, to the other border of the following layer. All other parasites work has to be performed by a service.
This is the IDH ;-) Simple isn't it ? :-) Soon on
AMAZON.COM ! May I Have Your Questions ? THANK