Data Storage and What to Do About It ?

FAMSF
112,000 works and shrinking
400 staff
264 regular employees
400 + accounts
325 computers (joined in AD)
40% macintosh
60% Windows XP
Exchange 2003, Windows Server 2003 and 2008
4D for image Collection
Over 84,000 images in high, medium and low resolution plus zoomified versions
Dwight Bailey, Harn Museum of Art, Gainesville, fL
Michael Webb, San Francisco Museum of Fine Arts, San Francisco, CA
Yvel Guelce, Indianapolis Museum of Art, Indianapolis, IN 
data storage overview
we're all experiencing uprecedented growth in data, we'll share our experience about how we are coping with growth and hopefully, you'll take some things with you to help you improve your data storage scenario
please stop us for questions or suggestions at any point...
here's what we'll look at:

our environment
data growth rates
what you should know about your data
how long is long enough?
stategies for managing growth
storage technologies for your data
our environment

HARN
nearly 7,000 works
collections of Asian, African, Modern, Contemporary and Photography
86,000 sq.ft. facility / 32,800 sq.ft. exhibition space 
about 55 employees
generally 80 users (including interns/volunteers)
65 computers (AD) - 0 MACs
Windows 2003/2008 Std & Ent Servers
Exchange 2007 (Federated UF)
Dual NAS arrays 3TB each (formatted)


What you should know about your data
where data is stored
Data types
sizes (files/folders)
data growth rates
portable drives?
DATA GROWTH RATES
how fast is your data growing
departmental drives
email
home directories
local computer files
databases
image shares
backups
archives
Software requirements
How long is long enough
Record retention and retrieval policies
how long do you need to keep data
how often and by who is data accessed
What data can be backed up or archived
When does delete really mean delete?
Strategies for managing data growth
Know your data
data storage needs change over time
know data sources/classify and align with business processes
Know and manage growth rates
use reports to understand data production and growth
For example: space usage, space by path, last accessed, duplicate files, storage trending, file size and types
Determine data source needs based:
reports, projects, institutional change
associated technologies (ROI, TCO, ROA, SLAs, hardware, software)
Know and manage data retention schedules
data lifecycles
storage management, provisioning
Automate (policy based administration)
IMA
50,000 works
299+ staff
304+ accounts
265 computers
27 Windows/Linux servers
Exchange 2003
Web/Patron Edge Products/DB
32TB EMC SAN offsite replication
NAS 1.8 TB
Storage technologies for your data
Hardware: 
Storage arrays
Servers
Infrastructure

Software:
Data Analysis
Backups
Archiving
ILM/HSM
Cloudware
Harn
90% data stored on NAS:
Registration - 18GB
Collections Images -300GB
IT - 87GB
Home directories 75GB
User Profiles - 7GB
Marketing - 78GB
Main share - 67GB
Curatorial - 32GB
Education - 12GB
Interns/Volunteers 61 GB
Development 5GB
Finance 1.5 GB

FAMSF
User share - 300GB
Private - 900GB
Image Server - 900GB
Applications
 -Heavy data storage
 email 250GB
 museum store 50GB
-Medium:
ticketing system
development system
-low:
Financial system
Admin system
Human resources
IT system data

IMA
new media/digital video content 2.5TB
photography digitial images 5.5TB
conservation
File server:
departments 1.52TB
Photography backlog 1.1TB
user share 652GB
Shared (inter-department) 109GB
DATA TYPES
IMA - .doc, pdf, dng, tiff, psd 
mov, wav, wma, mp4..etc
Harn - YES
FAMSF- DITTO THAT!
PORTABLE DRIVES
HARN
usually 2-4GB flash drives, except for Registration (500GB)
used by Finance, Marketing, Education, Registration, IT, Director
for: PowerPoints, take home work, digitization image transfer, intern/staff file sharing
IMA
temporary project files
local storage for faster access
transport work between home and office
one TB drives or 2-4 GB flash drive
FAMSF
used by marketing to exchange data with vendors
work from home
drives are 300-750GB


HARN
1year - permanent
i.e. - fianancial data (permanent)
TMS image files (permanent)
working image files(1 year)
emails?
backups?
copyright (permanent)
FAMSF
2 years - permanent
IMA
permanent
HARN
data accessed by all staff, interns and volunteers
IMA
authorized personel, based on folder permissions for individuals or departments
FAMSF
all staff who use a museum computer or system from financial system to ticketing/membership


IMA
Backed up:
VSS
Replication
daily incremental
full backup (weekends)
Windows and Linux servers
Exchange, VMware, Windows/Linux
Digital Asset Management to tape
Archived:
keep differencials for 7 days
backup tapes shipped to Recall weekly
10 week rotation on tapes
tapes kept indefinitely.




10 point plan*:
define a storage mission and objective
begin by understanding your business and communicating with models
use models for data classification and business alignment
use virtualisation techniques to group, move, map data efficiently
consider archiving solutions that can be integrated with existing backup solutions
consider a cross-platform data management solution for SAN,NAS,DAS, SRM and business level reporting
consider a backup solutions that offer advanced features and can easily intergrate with compliance solutions.
maximize current employee resource training in lifecycle management solutions
model business requirements
start with subset of infrastructure and build upon success.
*from Data Lifecycles - Managing Data for Strategic Advantage (2007)
Storage drives
SATA/SCSI/SAS
BER/MTBF
RAID1/RAID5/RAID6
Tape and VTL
Tiered storage
Disaster Preparedness/Tolerance
RTO/RPO
CDP

Questions ?
FAMSF
Backed up:
all data
databases, systems, images, documents
Disc to Disc
nightly incremental
weekly snapshots to table (LTO3/4)
Archived:
all data backed up is archived
quarterly full archives
yearly permanent archive
Deleted:
we do not delete files
HARN
Backed up:
incrementals/VSS/replication
department shares
home directories
web server files
image files
databases
Archived:
files before 2006 that were not accessed in 3 years or more
archives only accessible by IT
Delete permanently: user initiated, IT deleted