Frequently Asked Questions
Why can’t I log in?
Log in problems can occur for a number of reasons. If you cannot log into one of ARC’s systems, please check the following:
Is your PID password expired? Try logging into onecampus.vt.edu. If you cannot log in there, then your PID password has likely expired and needs to be changed. (Contact 4Help for help with this issue.)
Are you on-campus? If you are not on-campus, you will need to connect to the Virginia Tech VPN in order to access ARC’s systems.
Is the hostname correct? Please check the name of the login node(s) for the system you are trying to access. For example, for login to TinkerCliffs, the hostname is not
tinkercliffs.arc.vt.edubut rathertinkercliffs1.arc.vt.eduortinkercliffs2.arc.vt.edu.Do you have an account? You must request an account on a system before you can log in.
Is there a maintenance outage? ARC systems are occassionally taken offline for maintenance purposes. Users are typically notified via email well ahead of maintenance outages.
If you have checked all of the above and are still not sure why you cannot log in, please submit a help ticket.
How much does it cost to use ARC’s systems?
ARC’s systems are free to use, within limits. This means that Virginia Tech researchers can simply request an account to get access and run. Usage beyond fairly restrictive personal limits does require an approved allocation requested by a faculty member or project principal investigator; this requires some basic information to be provided, but getting an allocation does not require monetary payment of any kind. Researchers who need access to more resources beyond what we provide for free or who would like to purchase dedicated hardware can do so through our Cost Center or Investment programs. More information on how to get started with ARC is here.
Why is my job not starting?
The squeue command shows the running and pending jobs for a user, and provides the reason a pending job isn’t starting. Jobs run by other users will not be shown in the squeue output. To only show information for a particular job use squeue -j <jobid>. There are limits that apply per job, per user, and per account to ensure a fair utilization of reources among all users (see the Quality of Service (QoS) of each partition of the clusters). Consult all Slurm job reason codes which include some of the most common reasons:
Reason |
Meaning |
Recommendations |
|---|---|---|
|
These two are the most common reasons for a job being pending (PD). |
No action needed. The job will start as soon as resources become available. |
|
The CPU request exceeds the per-user limit for the requested QoS. |
Revise the total number of CPUs in jobs run by your user. |
|
The memory request exceeds the per-user limit for the requested QoS. |
Revise the total amount of memory in jobs run by your user. |
|
The GRES request for GPUs exceeds the per-user limit for the requested QoS. |
Revise the total number of GPUs in jobs run by your user. |
|
The CPU request exceeds the per-account limit on the job’s QoS. |
Revise the total number of CPUs in jobs run in the account allocation. |
|
The memory request exceeds the per-account limit on the job’s QoS. |
Revise the total amount of memory in jobs run in the account allocation. |
|
The GRES request for GPUs exceeds the per-account limit on the job’s QoS. |
Revise the total number of GPUs in jobs run in the account allocation. |
|
The wall time request of the job exceeds the wall time limit of the QoS. |
Reduce the wall time within the QoS limits (1 day for short, 7 days for base (default), 14 days for long). |
|
The account allocation of the job has exceeded its resource limits (e.g., in the free tier). |
Acquire additional Service Units via the Cost Center, wait for the monthly renewal of the free tier, or run in preemtable partitions. |
Why can’t I run on the login node?
One of the most common beginner mistakes on compute clusters is to log into the cluster and then immediately start running a computation. When you log into a cluster, you land on a login node. Login nodes are individual computers that represent a very small segment of the overall cluster and, crucially, are shared by many of the users who are logged into the cluster at a given time. So while basic tasks (editing files, checking jobs, perhaps making simple plots or compiling software) are fine to do on the login nodes, when you run a computationally-intensive task on the login node, you are adversely impacting other users (since the node is shared) while getting worse performance for yourself (by not using the bulk of the cluster). You should therefore submit your computationally intensive tasks to compute nodes by submitting a job to the scheduler. See here for documentation about job submission; we also have a video tutorial that will walk you through the process in a few minutes. Users who run problematic programs on the login node can have those tasks killed without warning. Users who repeatedly violate this policy arc subject to having their ARC account suspended (see Acceptable Use Policy).
When will my job start?
Adding the --start flag to squeue will provide the system’s best guess as to when the job will start, or give a reason for why the job will not start in the NODELIST(REASON) column. If no estimated start time is provided, please see Why is my job not starting? for more information.
How do I submit an interactive job?
A user can request an interactive session on a compute node (e.g., for debugging purposes), using interact, a wrapper script around srun. By default, this script will request one core (with one GPU on GPU clusters) for one hour on a default partition. An allocation must be provided:
interact -A yourallocation
The request can be customized with standard job submission flags used by srun or sbatch. Examples include:
Request two hours:
interact -A yourallocation -t 2:00:00
Request two hours on the
normal_qpartition:interact -A yourallocation -t 2:00:00 -p normal_q
Request two hours on one core and one GPU on Falcon’s
l40s_normal_q:interact -A yourallocation -t 2:00:00 -p l40s_normal_q -n 1 --gres=gpu:1
The flags for requesting resources may vary from system to system; please see the documentation for the system that you want to use.
Once the job has been submitted, the system may print out some information about the defaults that interact has chosen. Once the resources requested are available, you will then get a prompt on a compute node. You can issue commands on the compute node as you would on the login node or any other system. To exit the interactive session, simply type exit.
Note: As with any other job, if all resources on the requested queue are being used by running jobs at the time an interactive job is submitted, it may take some time for the interactive job to start.
Important: All CPU cores, memory, and GPUs allocated to your job will remain attached until you terminate it with exit. Idle interactive sessions consume resources that other users can’t use. Do not leave interactive sessions idle.
How can I interpret Slurm billing reports for compute usage?
Understanding ARC accounting is important to best utilize the free tier. Every PI is allocated 1M Service Units (SUs) per month in the free tier. Jobs run on the cluster consume service units based on the amount and type of resources allocated, and the length of the job. For example, one job requesting 4 CPU cores, 16 GB of memory, and 1 A100 GPU will consume 4 cores x 0.01 SUs/core + 16 GB * 0.0625 SUs/GB + 1 * 100 SUs/A100 GPU = 105 SU/s per hour. Users can use sacct -j <jobid> -X -o jobID,partition%20,AllocTREs%70 to understand the SUs consumed by a given job:
[user@tinkercliffs1 ~]$ sacct -j 294 -X -o jobID,partition%20,AllocTREs%70
JobID Partition AllocTRES
------------ -------------------- ----------------------------------------------------------------------
294 a100_normal_q billing=105,cpu=4,gres/gpu:a100=1,gres/gpu=1,mem=16G,node=1
Users can view the TRESBillingWeights for a given partition using scontrol show partition <partition_name> | grep TRESBillingWeights
[user@tinkercliffs1 ~]$ scontrol show partition a100_normal_q | grep TRESBillingWeights
TRESBillingWeights=CPU=1.0,Mem=0.0625G,GRES/gpu=100.0
You can also estimate the SUs consumed by a job using this calculator based on the following table.
Resource |
SU per hour |
|---|---|
1 CPU core (default) |
1 |
1 CPU core (Owl) |
1.5 |
1 GB RAM |
0.0625 |
1 H200 GPU |
150 |
1 A100 GPU |
100 |
1 L40s GPU |
75 |
1 A30 GPU |
75 |
1 V100 GPU |
50 |
1 T4 GPU |
25 |
How do I change a job’s stack size limit?
If your MPI code needs higher stack sizes then you may specify the stack size in the command that you specify to MPI. For example:
mpirun -bind-to-core -np $SLURM_NTASKS /bin/bash -c ulimit -s unlimited; ./your_program
How do I check my job’s resource usage?
The jobload command will report core and memory usage for each node of a given job. Example output is:
[user@tinkercliffs2 04/06 09:21:13 ~]$ jobload 129722
Basic job information:
JOBID PARTITION NAME ACCOUNT USER STATE TIME TIME_LIMIT NODES NODELIST(REASON)
129722 normal_q tinkercliffs someaccount someuser RUNNING 43:43 8:00:00 2 tc[082-083]
Job is running on nodes: tc082 tc083
Node utilization is:
node cores load pct mem used pct
tc082 128 128.0 100.0 251.7GB 182.1GB 72.3
tc083 128 47.9 37.4 251.7GB 187.2GB 74.3
This TinkerCliffs job is using all 128 cores on one node but only 48 cores on the second node. In this case, we know that the job has requested two full nodes, so some optimization may be in order to ensure that they’re using all of the requested resources. The job is, however, using 70-75% memory on both nodes, so the resource request may be intentional. If more information is required about a given node, the command scontrol show node tc083 can provide it.
How can I monitor GPU utilization during my job?
The nvidia-smi command with no other options diplays this information but prints to standard output (console or output file) and only once upon invocation. There are many options which can be added to tap into lots of extended functionality of this tool.
Add a line like this to the batch script prior to starting a GPU workload:
nvidia-smi --query-gpu=timestamp,name,pci.bus_id,driver_version,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memy.used --format=csv -l 3 > $SLURM_JOBID.gpu.log &
The & causes the query to run in the background and keep running until the job ends or this process is manually killed. The > $SLURM_JOBID.gpu.log causes the output to be redirected to a file whose name is the numerical job id followed by .gpu.log.
The -l 3 is for the repeating polling interval. From the nvidia-smi manual:
-l SEC, --loop=SEC
Continuously report query data at the specified interval, rather than the default of just once.
For details on query options: nvidia-smi --help-query-gpu
Output from nvidia-smi run as above looks like this for a 2-GPU job (notice the different GPU identifier strings):
2021/10/29 16:36:30.047, A100-SXM-80GB, 00000000:CB:00.0, 460.73.01, 41, 0 %, 0 %, 81251 MiB, 81248 MiB, 3 MiB
2021/10/29 16:36:33.048, A100-SXM-80GB, 00000000:07:00.0, 460.73.01, 58, 16 %, 4 %, 81251 MiB, 66511 MiB, 14740 MiB
2021/10/29 16:36:33.053, A100-SXM-80GB, 00000000:CB:00.0, 460.73.01, 41, 0 %, 0 %, 81251 MiB, 81248 MiB, 3 MiB
2021/10/29 16:36:36.054, A100-SXM-80GB, 00000000:07:00.0, 460.73.01, 65, 98 %, 15 %, 81251 MiB, 66571 MiB, 14680 MiB
2021/10/29 16:36:36.055, A100-SXM-80GB, 00000000:CB:00.0, 460.73.01, 41, 0 %, 0 %, 81251 MiB, 81248 MiB, 3 MiB
2021/10/29 16:36:39.055, A100-SXM-80GB, 00000000:07:00.0, 460.73.01, 67, 100 %, 36 %, 81251 MiB, 66571 MiB, 14680 MiB
2021/10/29 16:36:39.056, A100-SXM-80GB, 00000000:CB:00.0, 460.73.01, 41, 0 %, 0 %, 81251 MiB, 81248 MiB, 3 MiB
2021/10/29 16:36:42.057, A100-SXM-80GB, 00000000:07:00.0, 460.73.01, 54, 10 %, 2 %, 81251 MiB, 66571 MiB, 14680 MiB
2021/10/29 16:36:42.058, A100-SXM-80GB, 00000000:CB:00.0, 460.73.01, 41, 0 %, 0 %, 81251 MiB, 81248 MiB, 3 MiB
2021/10/29 16:36:45.059, A100-SXM-80GB, 00000000:07:00.0, 460.73.01, 54, 0 %, 0 %, 81251 MiB, 66571 MiB, 14680 MiB
2021/10/29 16:36:45.060, A100-SXM-80GB, 00000000:CB:00.0, 460.73.01, 41, 0 %, 0 %, 81251 MiB, 81248 MiB, 3 MiB
2021/10/29 16:36:48.060, A100-SXM-80GB, 00000000:07:00.0, 460.73.01, 68, 100 %, 26 %, 81251 MiB, 66571 MiB, 14680 MiB
2021/10/29 16:36:48.061, A100-SXM-80GB, 00000000:CB:00.0, 460.73.01, 41, 0 %, 0 %, 81251 MiB, 81248 MiB, 3 MiB
2021/10/29 16:36:51.062, A100-SXM-80GB, 00000000:07:00.0, 460.73.01, 52, 20 %, 3 %, 81251 MiB, 66571 MiB, 14680 MiB
2021/10/29 16:36:51.063, A100-SXM-80GB, 00000000:CB:00.0, 460.73.01, 41, 0 %, 0 %, 81251 MiB, 81248 MiB, 3 MiB
2021/10/29 16:36:54.064, A100-SXM-80GB, 00000000:07:00.0, 460.73.01, 52, 0 %, 0 %, 81251 MiB, 66571 MiB, 14680 MiB
You can monitor the utilization information in near-real-time from a login node by navigating to the output directory for the job and using tail to follow the output with tail -f <jobid>.gpu.log. The CSV formatting makes it easy to analyze or generate graphics with other tools such as python, R, or matlab.
I need a software package for my research. Can you install it for me?
Yes! If you need a software package that can run on GNU/Linux you can submit a ticket and we will install it. Licensed software may require additional steps depending on the type of license (individual, group, or university-wide license). ARC staff install packages centrally via software modules to help all users easy access to the most common applications.
What does a “Disk quota exceeded” error mean?
This typically means that the quota of one of your storage locations has been exceeded. You will need to reduce the space consumed in order to run jobs successfully again. You can run the command quota to get a report of your storage utilization.
Important: the report generated by quota is not in real time. Therefore, any files you delete will not reflect new free space in quota immediately.
What does a “Detected 1 oom-kill event(s)” error mean?
If your job fails with an error like
slurmstepd: error: Detected 1 oom-kill event(s)
then your job triggered Linux’s Out of Memory (OOM) Killer process. This means that it tried to use more memory than allocated to the job. The seff command (Slurm job efficiency) also provides some information on resource utilization:
[user@tinkercliffs1 ~]$ seff 1447
Job ID: 1447
Cluster: tinkercliffs
User/Group: someuser/someuser
State: OUT_OF_MEMORY (exit code 0)
Nodes: 2
Cores per node: 32
CPU Utilized: 02:43:59
CPU Efficiency: 1.56% of 7-07:21:36 core-walltime
Job Wall-clock time: 02:44:24
Memory Utilized: 174.83 GB
Memory Efficiency: 49.11% of 356.00 GB
If your job is requesting an explict amount of memory with (e.g. --mem 16G) then increase the amount of memory allocated to the job. Recommendation: increment the amount of memory progressively until sufficient to run without error. Excessive memory allocations will waste memory and will make your job to sit in the queue for longer. If your job is not requesting an explicit amount of memory then Slurm is assigning a default amount of memory per CPU allocated. In this case, you can enforce the explicit amount of memory (recommended) or increase the number of CPU cores to force the system to assign to your job a larger memory amount (not recommended).
Why are basic commands like sbatch not recognized?
If basic commands like sbatch are not recognized, it is often because these default modules have been removed (e.g., via module purge). Please run module reset to restore the default $MODULEPATH.
How do I add a user to an allocation?
To add a user to an existing allocation, the PI must follow these steps:
Go to ColdFront.
You will see a list of your Projects. Click on the one you want to modify.
Scroll down to Users and select Add Users.
Under Search String enter the user’s PID (or a list of PIDs) and click Search.
Scroll down, select the user whom you want to add, and click Add Selected Users to Project.
The page will refresh and the user’s PID should be included in the Users table. They are now added to the project and its associated allocations.
Note: It may take up to 1 hour for the clusters to reflect changes made on ColdFront.
PIs are solely responsible of adding and removing users from their allocations. This is important for controlling access to data, especially if the data has access restrictions.
How do I attach to my process for debugging?
Debuggers like gdb make software development much more efficient. You may simply ssh to the compute node where the process is running, look up the process ID (e.g., with top or ps), and then attach to it.
How can I submit a job that depends on the completion of another job?
Sometimes it may be useful to split one large computation into multiple jobs (e.g. due to queue limits), but submit those jobs all at once. Jobs can be made dependent on each other using the --dependency=after:job_id flag to sbatch. Additional dependency options can be found in the documentation for sbatch. For example, here we submit three jobs, each of which depends on the preceding one:
[user@tinkercliffs2 ~]$ sbatch test.sh
Submitted batch job 126448
[user@tinkercliffs2 ~]$ sbatch --dependency=after:126448 test.sh
Submitted batch job 126449
[user@tinkercliffs2 ~]$ sbatch --dependency=after:126449 test.sh
Submitted batch job 126450
The first job starts right away, but the second doesn’t start until the first one finishes and the third job doesn’t start until the second one finishes. This allows the user to split their job up into multiple pieces, submit them all right away, and then just monitor them as they run one after the other to completion.
How can I run multiple serial tasks inside one job?
Users with serial (sequential) programs may want to package multiple serial tasks into a single job submitted to the scheduler. This can be done with third-party tools (GNU parallel is a good one) or using a loop within the job submission script. (A similar structure can be used to run multiple short, parallel tasks inside a job.) The basic structure is to loop through the number of tasks using while or for, start the task in the background using the & operator, and then use the wait command to wait for the tasks to finish:
# Define variables
numtasks=16
np=1
# Loop through numtasks tasks
while [ $np -le $numtasks ]
do
# Run the task in the background with input and output depending on the variable np
./a.out $np > $np.out &
# Increment task counter
np=$((np+1))
done
# Wait for all of the tasks to finish
wait
Please note that the above structure will only work within a single node. To ensure that the same program (with the same inputs) isn’t being run multiple times, users should make sure that the loop variable (np, above) is used to specify input files or parameters.
How can I run multiple short, parallel tasks inside one job?
Sometimes users have a parallel application that runs quickly, but that they need to run many times. In this case, it may be useful to package multiple parallel runs into a single job. This can be done using a loop within the job submission script. An example structure:
# Specify the list of tasks
tasklist=task1 task2 task3
# Loop through the tasks
for tsk in $tasklist; do
# run the task $tsk
mpirun -np $SLURM_NTASKS ./a.out $tsk
done
To ensure that the same program (with the same inputs) isn’t being run multiple times, users should make sure that the loop variable (tsk, above) is used to specify input files or parameters. Note that, unlike when running multiple serial tasks at once, in this case each task will not start until the previous one has finished.