EDAN

EDAN

Enterprise Data Asset Network

• What is OCIO supporting
• Background/History
• First Steps 1.0
• Evolution
• Stats

EDAN 1.0 (one butterfly)

metadata

Enterprise Data Asset Network

Silos of data were piling up.

Over 150+ databases existed.

What OCIO Supports

Enterprise Data divided into two main parts

~15-17 RHEL servers

metadata assets.

~44 million metadata records.

• data about 'things'...
• paintings, sculptures, artifacts, specimens, people, archives

image

image (2d and 3d) assets.

• ~23 million image files.

We were ramping up a centralized asset management system (DAMS).

initial concept.....

One search yielding results from across all Smithsonian data sets. Union catalog of all Smithsonian holdings..

goal: streamline access, centralize data

Andrew Gunther

Evolution

Outcome

How..

• A centralized data pipeline from content management systems to an indexed repository.

• Common Metadata Format
• flexible to support all SI data
• expressive
• accessible, having good quality index

• No longer just an index
• 40+ million json content store
• myriad of different types of data
• stories, exhibitions, transcriptions...
• extended APIs to contribute and consume data
• open access api

Access:

• Since we were elimintating silos we had to offer consumers a way to get their data.

Current Technologies

Process..

Partners would map their data to our

metadata model and we'd ingest the content and index it.

Major technology would be an open source product from ASF called Apache/Solr Lucene (v 0.9)

APIs.

application programming interface

• All our APIs are over the http protocol.

• Apache Kafka (messaging 1 million/sec)
• Apache Zookeeper and Solr 6,7
• Maria DB, ArrangoDB, MongoDB, reTHINKdb.
• Myriad of open source libraries.

• All our APIs* are authenticated.
• Each consumer is issued an application id and a private key in which they sign their requests.
• Consumers are generally applications written in php, java, objective-c, c#, python.

Dynamic Zooming

How:

Pyramidal Tiling

Goals:

Megapixel:

A megapixel means one million pixels. The resolution of digital cameras and camera phones is often measured in megapixels. For example, a 12-megapixel camera can produce images with 12 million total pixels.

slice in small pieces

resize image/2

repeat.

1000px x 1000px = 1MP

3000px x 3000px = 9MP

IDS.. Image Delivery Services

Same strategy used by maps.

8000px x 8000px = 64MP

6.6 Giga Pixel

92417 x 70093 =

**JPEG spec stops at 65,535×65,535 pixels

For every asset marked public a high resultion jpeg is delivered to IDS

• Provide access
• Serve the needs of the consumer.
• All images can be dynamically resized.

OCIO Support Tech

• Splunk
• Nagios
• Zabbix

All images are made available via http(s)

http://ids.si.edu/ids/deliveryService?id=SG-1988-029_001

https://ids.si.edu/ids/deliveryService?id=SG-1988-029_001&max=200

Statistics August 25th.

roughly ~4 million api requests.

99.7% request success (2xx, 3xx)

200: 3,718,114 99.7%

404: 5,021 0.1%

302: 3,873 0.1%

403: 417 0.0%

206: 360 0.0%

301: 355 0.0%

304: 84 0.0%

401: 38 0.0%

400: 26 0.0%

500: 26 0.0%

502: 7 0.0%

405: 5 0.0%

406: 1

Where is EDAN.

Open Access and IIIF

Data is now mirrored out to AWS/github.

Nearly every major website at SI uses the metadata APIs as well as the image delivery services.

Universal Syntax

for Image Delivery (IIIF)