Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Make your likes visible on Facebook?

Connect your Facebook account to Prezi and let your likes appear on your timeline.
You can change this under Settings & Account at any time.

No, thanks

HPC@BCH

Overview of neuroimaging computation on the Boston Children's Hospital PICES cluster.
by

Rudolph Pienaar

on 29 November 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of HPC@BCH

"HP"
384 GB RAM
144 Cores
Purpose:
Natural Language Processing
Uses cTAKEs application (UIMA framework for Natural Language Processing)
64 GB RAM
GPU
RHEL
64 GB RAM
GPU
64 GB RAM
GPU
RHEL
64 GB RAM
GPU
RHEL
64 GB RAM
GPU
RHEL
64 GB RAM
GPU
RHEL
"GDL"
320 GB RAM
160 Cores
Purpose:
Genetic analysis -- genomics
64 GB RAM
GPU
RHEL
36 GB RAM
Quadro FX 2800M
Ubuntu 12.04
36 GB RAM
Tesla C2075
Ubuntu 12.04
36 GB RAM
Tesla C2075
Ubuntu 12.04
``rc-thunderball''
``rc-twice''
``rc-drno''
36 GB RAM
Quadro FX 2800M
Ubuntu 12.04
``rc-russia''
36 GB RAM
Quadro FX 2800M
Ubuntu 12.04
``rc-goldfinger''
"PICES"
180 GB RAM
80 Cores
Purpose:
NeuroImaging
Primarily MRI, including
Volumetric Processing
Diffusion Processing
64 GB RAM
GPU
RHEL
64 GB RAM
GPU
RHEL
64 GB RAM
GPU
RHEL
64 GB RAM
GPU
RHEL
36 GB RAM
Tesla C2075
Ubuntu 12.04
36 GB RAM
Tesla C2075
Ubuntu 12.04
``rc-thunderball''
``rc-twice''
36 GB RAM
Quadro FX 2800M
Ubuntu 12.04
``rc-russia''
36 GB RAM
Quadro FX 2800M
Ubuntu 12.04
``rc-goldfinger''
"PICES"
180 GB RAM
80 Cores
16 GB RAM
Ubuntu 12.10
24 GB RAM
Ubuntu 12.10
``pretoria''
``shaka''
12GB RAM
Ubuntu 12.10
``paris''
"AUTUMN"
52 GB RAM
32 Cores
36 GB RAM
Quadro FX 2800M
Ubuntu 12.04
``rc-drno''
36 MB RAM
Tesla C2075
Ubuntu 12.04
36 MB RAM
Tesla C2075
Ubuntu 12.04
``rc-thunderball''
``rc-twice''
36 MB RAM
Quadro FX 2800M
Ubuntu 12.04
``rc-russia''
36 MB RAM
Quadro FX 2800M
Ubuntu 12.04
``rc-goldfinger''
36 MB RAM
Quadro FX 2800M
Ubuntu 12.04
``rc-drno''
24 GB RAM
Ubuntu 12.10
``pretoria''
16 GB RAM
Ubuntu 12.10
24 GB RAM
Ubuntu 12.10
``pretoria''
``shaka''
12GB RAM
Ubuntu 12.10
``paris''
24 GB RAM
Ubuntu 12.10
``pretoria''
MOSIX
Cluster scheduler
Custom Linux kernel
Linux rc-drno 3.2.21-mosix
What is MOSIX?
MOSIX
What does it provide?

Users can login on any node and do not need to know where their programs run.
No need to modify or link applications with special libraries.
No need to copy files to remote nodes.
Automatic resource discovery: whenever clusters or nodes join (disconnect), all the active nodes are updated.
Automatic workload distribution by process migration, including load balancing, process migration from slower to faster nodes and from nodes that run out of free memory.
MOSIX
Migratable sockets for direct communication between migrated processes.
Secure run time environment (sandbox) for guest processes.
Live queuing – queued jobs preserve their full generic Linux environment.
Batch jobs.
Multi clusters...
Other benefits...
Storage Architecture
Small local storage on each node
33GB
Central Shared Isilon NAS connected directly to backbone
Provides fast local space that is also NFS exported back to "front end offices"
GPU available on each node for GP-CPU processing
Two nodes have nVidia Tesla C2075s for higher end features
Front End Resources
Linux Workstations
form own sub-cluster
linked to backend cluster
Directory view on "Front End" machine
Directory view on "cluster" machine
Unified Directory Space between local workstations (Linux and Mac) and HPC nodes
Neuroimaging on an HPC
Prerequisites
Unified file space
Batch-orientated processing
Command line familiarity
Reality...
Lack of user CLI knowledge
Lack of user interest in CLI
especially in biological/medical sciences
Batch-processing
Input data
Transformed by intermediate processes
Output result
For the most part, other than some user "boundary condition" specification, batch pipelines run with little to no user interaction.
NeuroImaging Pipelines on PICES
freesurfer
diffusion tractography
connectivity analysis
PACS access
freesurfer single-shot executables
3D Slicer Modules (execution model)
ChRIS
Ch
ildren's
R
esearch
I
ntegrative
S
ystem
Opensource (MIT licensed)
Web-based
PHP
Javascript
Expandable Plugin System
Collaborative
Hooks to backend HPC and Hospital data sources
Overview
NeuroImaging
and High Performance Computing

Implementation and Use at Boston Children's Hospital
Rudolph Pienaar, D.Eng
Technical Director:

Fetal-Neonatal Neuroimaging and Developmental Science Center
Staff Scientist:
Boston Children's Hospital
Instructor in Radiology:
Harvard Medical School

HPC in the Boston Area
Harvard and Harvard Medical School Institutions
Boston Children's Hospital

HPC Hardware Overview
HPC Scheduler
Software Provided
Directory / filespace

Custom ``Middleware" System
Typical Use Cases
Expandable plugin architecture

Background / HPC Environment
PICES Cluster for Neuroimaging
User Perspective

MIT
Boston University
IBM Blue Gene
Harvard
rcclu
erisone
launchpad
Partners Healthcare
MGH
McLean
BWH
Conclusion
Background / HPC Environment
PICES Cluster for Neuroimaging
User Perspective and ChRIS middleware

Acknowledgements
Ch
ildren's
R
esearch
I
ntegrative
S
ystem
Effective
232 GB RAM
112 Cores

rc
23,000 cores
8PB storage
hms
6,000 cores
3PB storage
seas
200 cores
0.5PB storage
iqss
500 processing cores
0.5PB storage
Software
FreeSurfer
3D Slicer
Diffusion Toolkit
MatLAB
Python
numpy
pylab
nipype
# Authenticate

# Local Workstation
login: <user>
password: <passwd>

# Find your data
> cd /some/data/directory

# Copy to cluster filespace
> scp -r studyData <user>@cluster.node
# log into cluster node
$> ssh <user>@cluster.node

# go to remote directory
$> cd /dir/with/studyData

# source FreeSurfer env
$> source /dir/to/FreeSurfer/FreeSurfer_env.sh
version: dev
revision: -system default-
Host type is x86_64-Linux
-------- freesurfer-x86_64-unknown-linux-gnu-dev5-20120717 --------
Setting up environment for FreeSurfer/FS-FAST (and FSL)
FREESURFER_HOME /chb/freesurfer/x86_64-Linux/packages/freesurfer/dev-July2012
FSFAST_HOME /chb/freesurfer/x86_64-Linux/packages/freesurfer/dev-July2012/fsfast
FSF_OUTPUT_FORMAT nii.gz
SUBJECTS_DIR /chb/users/rudolphpienaar
MNI_DIR /chb/freesurfer/x86_64-Linux/packages/freesurfer/dev-July2012/mni
FSL_DIR /chb/freesurfer/x86_64-Linux/packages/fsl419
# Start analysis
# Step 1 -- setup dir structure
> recon-all <inputSeries> -s <SubjectName>

# Step 2 -- start analysis
> recon-all -autorecon-all -s <SubjectName>

# wait...

# also, don't KILL this terminal!
# Done!

# Now what?

> ls <SubjectName>
bem/ label/ mri/ scripts/ src/ stats/ surf/ tmp/ touch/ trash/

> ls <SubjectName>/stats
aseg.stats lh.aparc.a2009s.stats lh.aparc.stats lh.BA.stats lh.curv.stats lh.entorhinal_exvivo.stats rh.aparc.a2009s.stats rh.aparc.stats rh.BA.stats rh.curv.stats rh.entorhinal_exvivo.stats wmparc.stats

# etc... etc
# Visualize...

> cd $SubjectName
> tkmedit <SubjectName> brainmask.mgz -aux T1.mgz -surfs -aseg









> tksurfer $SubjectName lh inflated
Daniel Haehn, MSc
Nicolas Rannou, Msc
FNNDSC
Ellen Grant, MD
Emi Takahashi, PhD
Mathieu Dehaes, PhD
Kiho Im, PhD
BCH
Jonathan Bickel, MD
Thomas Boetcher
Harvard University
Andrew Kiruluta, PhD
Source Code
https://github.com/FNNDSC
Full transcript