Research Data Ecosystem ('RDE') Storage Information and Policies

Research Storage

The Research Data Ecosystem ('RDE') is comprised of 8.5PB connected high-performance storage. This is split between two vendors (VAST and Pixstor) with NFS/GPFS mounts to client servers or the Hellbender cluster.

  • Storage lab allocations are protected by associated security groups applied to the share, with group member access administered by the assigned PI or appointed representative.

 

What is the Difference between High Performance and General Performance Storage?

On Pixstor, which is used for standard HPC allocations, general storage is pinned to the SAS disk pool while high performance allocations are pinned to all flash NVME pool. Meaning writes and recent reads will have lower latency with High performance allocations.

On VAST, which is used for non HPC and mixed HPC / SMB workloads, the disks are all flash but general storage allocations have a QOS policy attached that limits IOPS to prevent the share from the possibility of saturating the disk pool to the point where high-performance allocations are impacted. High Performance allocations may also have a QOS policy that allows for much higher IO and IOPS. RSS reserves the right to move general store allocations to lower tier storage in the future if facing capacity constraints.

Use Cases for General Storage

  • Workloads that may require intensive computing but do not require sustained read and write IO with speeds in the multiple GB/s

  • Workloads that utilize SMB shares ON VAST

  • Workloads that require single server NFS mounts (VAST)

Use Cases for High Performance Storage

  • Workloads that require sustained use of low latency read and write IO of multiple GB/s, generally generated from jobs utilizing multiple HPC nodes

  • Workloads that require sustained use of low latency read and write IO with multiple GB/s, generally generated from jobs utilizing multiple NFS mounts

Snapshots

  • VAST default policy retains 7 daily and 4 weekly snapshots for each share

  • Pixstor default policy is 10 daily snapshots

 

Policies

None of the cluster attached storage available to users is backed up in any way by us, this means that if you delete something and don't have a copy somewhere else, it is gone. Please note the data stored on cluster attached storage is limited to Data Class 1 and 2 as defined by UM System DCL. If you have need to store things in DCL3 or DCL4 please contact us so we may find a solution for you.

 

Print Article