Monitoring Your Work on Compute Nodes
Monitoring means:
Identifying ALL of your jobs or a PARTICULAR job
Inspecting a PARTICULAR RUNNING slurm job.
Every single running Slurm (and OOD) job that you have should be inspected each day, morning, noon, and night, to ensure that it is running properly.
When any user wastes resources, everyone suffers.
We need to look out for ourselves and for others by monitoring our jobs.
Identifying Your Slurm Jobs on a Cluster
Video Number |
Video Name and Link |
Description |
|---|---|---|
1 |
How to identify slurm jobs (on compute nodes) via the command line |
How to use Slurm commands |
2 |
For OOD jobs, how to get the slurm job ID. |
Inspecting Your Running Slurm Jobs on a Cluster
From the above subsection, you have the Slurm Job ID for your job of interest.
In the previous subsection, you can identify queued, running, and if you issue the command just right, you can even see when the process of shutting down a job occurs. But this shut down process happens very quickly. Here, in this subsection, we are interested in running processes.
You may also need the compute node names and GPU indexes (if your job uses GPUs), which can be obtained from scontrol show job --details <slurm_job_id>
in the video above.
You can view these videos in any order, but some of them build on earlier videos, and although each video is standalone, you may get more out of later videos if you have watched the previous ones.
Video Number |
Video Name and Link |
Description |
|---|---|---|
1 |
How to use the command |
|
2 |
How to use the command |
|
3 |
How to use Grafana to inspect a particular job. Particularly for detailed GPU utilization. |
|
4 |
How to directly zero-in on your job within Grafana dashboards |
How to use |
5 |
How to ssh onto a compute node that you are running on and inspect job |
How to use |