Cascades, CPU Cluster

The Cascades cluster, after due notice, was decommissioned during a maintenance outage January 17-19, 2023. Please consider using the [Tinkercliffs](tinkercliffs) and [Infer](infer) clusters instead.

Overview

Cascades is a 236-node system capable of tackling the full spectrum of computational workloads, from problems requiring hundreds of compute cores to data-intensive problems requiring large amount of memory and storage resources. Cascade contains four compute engines designed for distinct workloads.

  • General - Distributed, scalable workloads. With Intel’s Broadwell processors, 2 16-core processors and 128 GB of memory on each node, this 190-node compute engine is suitable for traditional HPC jobs and large codes using MPI.

  • Very Large Memory - Graph analytics and very large datasets. With 3TB (3072 gigabytes) of memory, four 18-core processors and 6 1.8TB direct attached SAS hard drives, 400 GB SAS SSD drive, and one 2 TB NVMe PCIe flash card , each of these two servers will enable analysis of large highly-connected datasets, in-memory database applications, and speedier solution of other large problems.

  • K80 GPU - Data visualization and code acceleration. There are four nodes in this compute engine which have - two Nvidia K80 (Kepler) GPUs, 512 GB of memory, and one 2 TB NVMe PCIe flash card.

  • V100 GPU - These nodes were migrated to Infer in Spring 2022.

Technical Specifications

COMPUTE ENGINE

#

HOSTS

CPU

CORES

MEMORY

LOCAL STORAGE

OTHER FEATURES

General

190

ca007-ca196

2 x E5-2683v4 2.1GHz (Broadwell)

32

128 GB, 2400 MHz

1.8TB 10K RPM SAS200 GB SSD

Very Large Memory

2

ca001-ca002

4 x E7-8867v4 2.4 GHz (Broadwell)

72

3 TB, 2400 MHz

3.6 TB (2 x 1.8 TB) 10K RPM SAS (RAID 0)6-400 GB SSD (RAID 1) 2 TB NVMe PCIe

K80 GPU

4

ca003-ca006

2 x E5-2683v4 2.1GHz (Broadwell)

32

512GB, 2400MHz

3.6 TB (2 x 1.8 TB) 10K RPM SAS (RAID 0)2-400 GB SSD (RAID 1) 2 TB NVMe PCIe

2-NVIDIA K80 GPU

Notes:

  • K80 GPU Notes: There are 4 CUDA Devices. Although the K80s are a single physical device in 1 PCIe slot, there are 2 separate GPU chips inside. They will be shown as 4 separate devices to CUDA code. nvidia-smi will show this.

  • All nodes have locally mounted SAS and SSDs. /scratch-local (and $TMPDIR) point to the SAS drive and /scratch-ssd points to the SSD on each node. On large memory and GPU nodes, which have multiple of each drive, the storage across the SSDs are combined in /scratch-ssd (RAID 0) and the SAS drives are mirrored (RAID 1) for redundancy.

Network:

  • 100 Gbps Infiniband interconnect provides low latency communication between compute nodes for MPI traffic.

  • 10 Gbps Ethernet interconnect provides high speed connectivity and access to storage.

Policies

Cascades is governed by an allocation manager, meaning that in order to run most jobs, you must be an authorized user of an allocation that has been submitted and approved. For more on allocations, click here. The Cascades partitions (queues) are:

  • normal_q for production (research) runs.

  • largemem_q for production (research) runs on the large memory nodes.

  • dev_q for short testing, debugging, and interactive sessions. dev_q provides slightly elevated job priority to facilitate code development and job testing prior to production runs.

  • k80_q for runs that require access to K80 GPU nodes

//: # (* v100_normal_q for production (research) runs with the V100 nodes)

The Cascades partition (queue) settings are:

PARTITION

NORMAL_Q

LARGEMEM_Q

DEV_Q

K80_Q

V100_NORMAL

V100_DEV

Access to

ca007-ca196

ca001-ca002

ca007-ca196

ca003-ca006

ca197-ca236

ca197-ca236

Max Jobs

24 per user, 48 per allocation

1 per user

1 per user

4 per user, 6 per allocation

8 per user, 12 per allocation

1 per user

Max Nodes

32 per user, 48 per allocation

1 per user

32 per user, 48 per allocation

4 per user

12 per user, 24 per allocation

12 per user, 24 per allocation

Max Cores

1,024 per user, 1,536 per allocation

72 per user

1,024 per user, 1536 per allocation

128 per user

288 per user, 576 per allocation

336 per user

Max Memory (calculated, not enforced)

4 TB per user, 6 TB per allocation

3 TB per user

4 TB per user, 6 TB per allocation

2 TB per user

4 TB per user, 6 TB per allocation

1 TB per user

Max Walltime

144 hr

144 hr

2 hr

144 hr

144 hr

2 hr

Max Core-Hours

73,728 per user

10,368 per user

256 per user

9,216 per user

20,736 per user

168 per user

Notes:

  • Shared node access: more than one job can run on a node

  • The micro-architecture on the V100 nodes is newer than (and distinct from) the Broadwell nodes. For best performance and compatibility, programs that are to run on V100 nodes should be compiled on a V100 node. Note that the login nodes are Broadwell nodes, so compilation on a V100 node should be done as part of the batch job, or during an interactive job on a V100 node (see below).

Access

Cascades can be accessed via one of the two login nodes:

  • cascades1.arc.vt.edu

  • cascades2.arc.vt.edu

Users may also use Open OnDemand to access the cluster.

Job Submission

Access to all compute nodes is controlled via the Slurm resource manager; see the Slurm documentation for additional usage information. Example resource requests on Cascades include:

#Request exclusive access to all resources on 2 nodes 
#SBATCH --nodes=2 
#SBATCH --exclusive

#Request 4 cores (on any number of nodes)
#SBATCH --ntasks=4

#Request 2 nodes with 12 tasks running on each
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=12

#Request 12 tasks with 20GB memory per core
#SBATCH --ntasks=12 
#SBATCH --mem-per-cpu=20G