Parallelization with Slurm and GNU parallel
Setting up parallelization at the right level is crucial for efficient computing. Depending on your workflow, different tools are avaliable for parallelizing across nodes on cluster. GNU parallel is a great tool for “easily parallelizable” workflows (i.e. don’t require message passing) on Linux machines. Such workflows typically involve launching independent tasks with a combinatoric set of parameters; a common example of this use case is hyperparameter tuning. Other packages may be available in various programming languages, but orchestrating the resource allocation alongside the top-level conductor, Slurm, can be challenging. Therefore, we are providing some examples of best practices for allocating, utilizing, and scaling up resources for computing in parallel over a grid of arguments.
GNU Parallel in a single job
GNU Parallel is a great option because it is easily coupled with slurm to launch multiple instances of a program as one job in parallel. There are advantages for launching one sbatch job over many, including reducing overhead and improving runtime as well as averting limits on number of queued and running jobs.
R example with big data
Let’s say we have a working script in R named test_args.R that takes in some arguments:
args = commandArgs(trailingOnly=TRUE)
fmt_str = paste("Rello, from ", Sys.info()["nodename"], "... Arguments: ")
for (a in args) {fmt_str <- paste(fmt_str, " ", a)}
print(fmt_str)
This script simply prints the arguments (and the hostname so we can confirm that both nodes are being utilized), but let’s imagine that it loads our data and trains our model. Before even thinking about scaling up for hyperparameter tuning, we need to understand what resources are required for a single instance.
Examining utilization of a single instance
To begin developing our workflow, we will run a single instance of our program as a batch job on the normal_q partition on the tinkercliffs cluster. First, we set our sbatch parameters. With 8 cpus per task, and default 1 node and 1 task per node, we are telling slurm to how many resources we need overall and how to dedicate a certain amount of resources to a single task. Next, we’ll print some job information for logging. Then, we load R as a module. Finally, on the last line, we run our R program. Below is the script hello_R.job to run a single instance of our program:
#!/bin/bash
#SBATCH -A XXX
#SBATCH -p normal_q
#SBATCH --cpus-per-task=8
#SBATCH -J Rello
#SBATCH -D MyProjects/parallel_R
#SBATCH -o ./job_outputs/slurm-%j-%x.out
echo "date: `date`"
echo "hostname: $HOSTNAME"; echo
echo -e "\nChecking job details for CPU_IDs..."
scontrol show job --details $SLURM_JOB_ID
module load apps site/tinkercliffs/easybuild/setup
module load R/4.2.2-foss-2022b
Rscript test_args.R param1 param2 param3
Mind the directory structure. I have the hello_R.job file in the ~/MyProjects/parallel_R directory. With the -D sbatch parameter I am setting the working directory for the job, so I will submit the job from my $HOME directory denoted ~ in the command prompt. The job submission from the login node looks like this:
[slima2@tinkercliffs1 ~]$ sbatch MyProjects/parallel_R/hello_R.job
Submitted batch job 2518599
Let’s examine our memory utilization with seff <jobid> to determine if we requested enough resources. We see that a single execution of our toy script utilized only 2MB, but let’s pretend that we saw 10-12G of RAM is required to load our large data and train the model. This would tell us that 8 cpus per task is sufficient for our workload.
[slima2@tinkercliffs1 ~]$ seff 2518599
Job ID: 2518599
Cluster: tinkercliffs
User/Group: slima2/slima2
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 8
CPU Utilized: 00:00:03
CPU Efficiency: 9.38% of 00:00:32 core-walltime
Job Wall-clock time: 00:00:04
Memory Utilized: 2.21 MB
Memory Efficiency: 0.01% of 15.00 GB
Knowing this memory demand, we can tell slurm what resources we need and how to dedicate the right amount to each task in a parallel setup. 8 CPU cores will be associated with 15GB since the default mem-per-cpu is ~2GB (an even split of the 256GB across all 128 cores per node as shown in this table). Our slurm configured maximum for mem-per-cpu is ~2GB. If you increase this, you will be allocated more cpus and you will be charged accordingly. From the sbatch manual:
Note that if the job’s –mem-per-cpu value exceeds the configured MaxMemPerCPU, then the user’s limit will be treated as a memory limit per task; –mem-per-cpu will be reduced to a value no larger than MaxMemPerCPU; –cpus-per-task will be set and the value of –cpus-per-task multiplied by the new –mem-per-cpu value will equal the original –mem-per-cpu value specified by the user.
Note
Terminology around cluster architecture is important here. In slurm verbiage, a cpu is a core of a multicore node. “CPU”, “core”, “chip” and “processor” refer to the same hardware component. These terms can also apply to hardware on a local desktop machine. In HPC and research computing, a “node” in a cluster is a connected set of processors; a “partition” is a connected set of nodes. We publish the type of cpus (“Chip”) available on each partition for each cluster (e.g. Tinkercliffs and Infer) along with some details about memory and network connection. The different types of chips can accomodate different memory and processing loads. Which resources and how many are the key factors for optimizing runtime efficiency.
A digression on slurm accounting
Let’s check that we got the resources we expected. Sacct displays useful information about requested and allocated resources. We can use it with various flags such as -u for user name, -j for the system-wide unique job ID, -X for less about individual job steps, -o for the fields to print, and “%xx” for the number of characters of the field to print. You can run the following command to confirm your requested resources and allocated resources:
[slima2@tinkercliffs1 ~]$ sacct -u slima2 -j 2518599 -X -o jobid,jobname,reqtres%45,alloctres%45,state
JobID JobName ReqTRES AllocTRES State
------------ ---------- --------------------------------------------- --------------------------------------------- ----------
2518599 Rello billing=8,cpu=8,mem=15G,node=1 billing=8,cpu=8,mem=15G,node=1 COMPLETED
Here, we requested 8 cpus on a single node which gave us 15G of memory and we were indeed allocated that. A common case that may lead to unexpected results is the use of the –mem or –mem-per-cpu sbatch parameters. Let’s try another set of sbatch parameters in a slurm job script we’ll call RuhRoh.job:
#!/bin/bash
#SBATCH -A XXX
#SBATCH -p normal_q
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=10G
#SBATCH -J RuhRoh
#SBATCH -D MyProjects/parallel_R
#SBATCH -o ./job_outputs/slurm-%j-%x.out
echo "date: `date`"
echo "hostname: $HOSTNAME"; echo
echo -e "\nChecking job details for CPU_IDs..."
scontrol show job --details $SLURM_JOB_ID
module load apps site/tinkercliffs/easybuild/setup
module load R/4.2.2-foss-2022b
Rscript test_args.R param1 param2 param3
When we examine the accounting of our job, we see a discrepancy between the requested vs allocated resources. Even though we specified a single cpu in our request, we were actually allocated and billed 6 cpus to accomodate our memory request. For this reason, we recommend specifying the number of cpus that is associated with your memory requirement, rather than using the –mem flags, in order to avoid unexpected results.
[slima2@tinkercliffs1 ~]$ sacct -u slima2 -j 2597516 -X -o jobid,jobname,reqtres%45,alloctres%45,state
JobID JobName ReqTRES AllocTRES State
------------ ---------- --------------------------------------------- --------------------------------------------- ----------
2597516 RuhRoh billing=1,cpu=1,mem=10G,node=1 billing=6,cpu=6,mem=10242M,node=1 COMPLETED
Now that we have a deeper background on how slurm allocates resources, let’s get back to our parallelization effort…
Defining our parallelization scheme
We want to tune 3 hyperparameters each with different settings for a total of 512 configurations, and we want to speed up our gridsearch by parallelization. Let’s set our sbatch parameters so that each task will be allocated 8 cpu cores and each node can run 16 tasks, for a total of 128 cores per node (the entire node in this case). Let’s request 2 nodes for a total of 480GB of RAM (=15GBpertask*16taskspernode*2nodes or ~2GB*128cores*2nodes). To actually use all of these requested resources efficiently, we will use srun and GNU parallel to properly dedicate memory and processing from 8 cpus for each task.
We have two options for setting this up: one where a single srun calls parallel and one where parallel calls srun multiple times. The pros and cons of each setup come from the strengths and limitations of the tools srun and parallel in the context of the application.
Tool |
Pros |
Cons |
|---|---|---|
Slurm srun |
binding resources to tasks (CPU/GPU/memory) |
higher overhead for each task startup |
GNU parallel |
load balancing (ie. feeding N sets of parameters on P processors where N >> P and keeping all processors busy) |
difficult to do task binding to resources |
Ultimately, our goals are efficiency and usability. Different workflows have different demands that may make one setup a better choice than another. Below is a list of some considerations when these overall goals in mind from an HPC perspective.
Goal |
Considerations |
|---|---|
efficient usage of resources ~= minimal time to solution |
1. reduce overhead where possible (minimize srun tasks relative to parallel) |
ease of orchestration |
1. easy to understand chunking of the input domain(s) |
Srun calling parallel (srun ... parallel ...)
In this setup, we will have a single srun step. Our batch job shell script multinode_par.job looks like this:
#!/bin/bash
#SBATCH -A XXX
#SBATCH -p normal_q
#SBATCH -N 2
#SBATCH -c 8
#SBATCH --ntasks-per-node=16
#SBATCH -J parR
#SBATCH -D MyProjects/parallel_R
#SBATCH -o ./job_outputs/slurm-%j-%x.out
echo "date: `date`"
echo "hostname: $HOSTNAME"; echo
echo -e "\nChecking job details for CPU_IDs..."
scontrol show job --details $SLURM_JOB_ID
# Generate configurations and write commands to file
param1=$(seq -f "alpha-%g" 4)
param2=$(seq -f "beta-%g" 8)
param3=$(seq -f "gamma-%g" 16)
CMD_FOUT=commands.txt
parallel --dry-run Rscript test_args.R {} ::: $param1 ::: $param2 ::: $param3 > $CMD_FOUT
module load R/4.2.2-foss-2022b
# run parallel for each node
driver_fn () {
echo "$SLURM_NODEID"
cat $CMD_FOUT | \
awk -v NNODE="$SLURM_NNODES" -v NODEID="$SLURM_NODEID" 'NR % NNODE == NODEID' | \
parallel -j $SLURM_NTASKS_PER_NODE {}
}
export -f driver_fn
# the script will be executed ${SLURM_NTASKS} times
echo $SLURM_NTASKS
srun --ntasks=$SLURM_JOB_NUM_NODES bash -c "$(declare -p CMD_FOUT); driver_fn"
We use the --dry-run flag to write all arguments to a file called commands.txt. This will allow us to divide the total number of tasks by the number of nodes. We specify ${SLURM_JOB_NUM_NODES} parallel workers (tasks) in srun, executing our driver shell script with multiple cores per task and multiple tasks per node. Our driver bash function driver_fn will call our R script ${SLURM_NTASKS} times for each node. For dividing the tasks across nodes as mentioned, we use an awk command, otherwise duplication will occur.
Examining our resource utilization of the job after successfully completing, we can see that our job indeed was allocated 128 cores per node, 2 nodes, and 480GB of memory as expected.
[slima2@tinkercliffs1 ~]$ seff 2627276
Job ID: 2627276
Cluster: tinkercliffs
User/Group: slima2/slima2
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 128
CPU Utilized: 00:02:43
CPU Efficiency: 1.25% of 03:37:36 core-walltime
Job Wall-clock time: 00:00:51
Memory Utilized: 119.06 MB (estimated maximum)
Memory Efficiency: 0.02% of 480.00 GB (1.88 GB/core)
If we change the number of nodes to 3, we would expect an even faster walltime with more resources utilized:
[slima2@tinkercliffs1 ~]$ seff 2560737
Job ID: 2560737
Cluster: tinkercliffs
User/Group: slima2/slima2
State: COMPLETED (exit code 0)
Nodes: 3
Cores per node: 128
CPU Utilized: 00:02:57
CPU Efficiency: 1.65% of 02:59:12 core-walltime
Job Wall-clock time: 00:00:28
Memory Utilized: 11.30 MB (estimated maximum)
Memory Efficiency: 0.00% of 720.00 GB (1.88 GB/core)
Requesting more resources will result in longer waiting times in the queue, so the tradeoff between queue wait-time and job walltime should be weighed on a case by case basis. Different setups will be more or less optimal for different workflows. It’s important to understand as a baseline how your program works on a single node before scaling up to multiple nodes.
Parallel calling srun (parallel ... srun ...)
Let’s say we have the same working script test_args.R with the same arguments. In this setup, we will use parallel to launch 32 simulataneous sruns (=16tasks*2nodes) with the -j flag and the $SLURM_NTASKS variable. Below is our job script for this setup par_multistep.job:
#!/bin/bash
#SBATCH -A XXX
#SBATCH -p normal_q
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=8
#SBATCH -J parR
#SBATCH -D MyProjects/parallel_R
#SBATCH -o ./job_outputs/slurm-%j-%x.out
date
param1=$(seq -f "alpha-%g" 4)
param2=$(seq -f "beta-%g" 8)
param3=$(seq -f "gamma-%g" 16)
module load apps site/tinkercliffs/easybuild/setup
module load R/4.2.2-foss-2022b
parallel -j $SLURM_NTASKS srun --nodes=1 --ntasks=1 --cpus-per-task=8 Rscript test_args.R {1} {2} {3} ::: $param1 ::: $param2 ::: $param3
Examining our resource utilization, we can see we have the same number of cpus and memory as before with the same #SBATCH parameters.
[slima2@tinkercliffs1 ~]$ seff 2560745
Job ID: 2560745
Cluster: tinkercliffs
User/Group: slima2/slima2
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 128
CPU Utilized: 00:03:50
CPU Efficiency: 2.99% of 02:08:00 core-walltime
Job Wall-clock time: 00:00:30
Memory Utilized: 3.74 MB
Memory Efficiency: 0.00% of 480.00 GB
[slima2@tinkercliffs1 ~]$ seff 2560771
Job ID: 2560771
Cluster: tinkercliffs
User/Group: slima2/slima2
State: COMPLETED (exit code 0)
Nodes: 3
Cores per node: 128
CPU Utilized: 00:03:57
CPU Efficiency: 2.94% of 02:14:24 core-walltime
Job Wall-clock time: 00:00:21
Memory Utilized: 3.74 MB
Memory Efficiency: 0.00% of 720.00 GB
Besides the difference in syntax in how we are passing arguments to the parallel command, many more srun calls will be made which may put us at risk for hitting step number quotas. For completeness, we show below the difference in the number of steps in the two setups:
[slima2@tinkercliffs1 ~]$ sacct -j 2627276
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
2627276 parR normal_q arcadm 256 COMPLETED 0:0
2627276.bat+ batch arcadm 128 COMPLETED 0:0
2627276.ext+ extern arcadm 256 COMPLETED 0:0
2627276.0 bash arcadm 256 COMPLETED 0:0
[slima2@tinkercliffs1 ~]$ sacct -j 2560745
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
2560745 parR normal_q arcadm 256 COMPLETED 0:0
2560745.bat+ batch arcadm 128 COMPLETED 0:0
2560745.ext+ extern arcadm 256 COMPLETED 0:0
2560745.0 Rscript arcadm 8 COMPLETED 0:0
2560745.1 Rscript arcadm 8 COMPLETED 0:0
2560745.2 Rscript arcadm 8 COMPLETED 0:0
2560745.3 Rscript arcadm 8 COMPLETED 0:0
2560745.4 Rscript arcadm 8 COMPLETED 0:0
2560745.5 Rscript arcadm 8 COMPLETED 0:0
2560745.6 Rscript arcadm 8 COMPLETED 0:0
2560745.7 Rscript arcadm 8 COMPLETED 0:0
2560745.8 Rscript arcadm 8 COMPLETED 0:0
2560745.9 Rscript arcadm 8 COMPLETED 0:0
2560745.10 Rscript arcadm 8 COMPLETED 0:0
2560745.11 Rscript arcadm 8 COMPLETED 0:0
2560745.12 Rscript arcadm 8 COMPLETED 0:0
2560745.13 Rscript arcadm 8 COMPLETED 0:0
2560745.14 Rscript arcadm 8 COMPLETED 0:0
2560745.15 Rscript arcadm 8 COMPLETED 0:0
2560745.16 Rscript arcadm 8 COMPLETED 0:0
2560745.17 Rscript arcadm 8 COMPLETED 0:0
2560745.18 Rscript arcadm 8 COMPLETED 0:0
2560745.19 Rscript arcadm 8 COMPLETED 0:0
2560745.20 Rscript arcadm 8 COMPLETED 0:0
2560745.21 Rscript arcadm 8 COMPLETED 0:0
2560745.22 Rscript arcadm 8 COMPLETED 0:0
2560745.23 Rscript arcadm 8 COMPLETED 0:0
2560745.24 Rscript arcadm 8 COMPLETED 0:0
2560745.25 Rscript arcadm 8 COMPLETED 0:0
2560745.26 Rscript arcadm 8 COMPLETED 0:0
2560745.27 Rscript arcadm 8 COMPLETED 0:0
2560745.28 Rscript arcadm 8 COMPLETED 0:0
2560745.29 Rscript arcadm 8 COMPLETED 0:0
2560745.30 Rscript arcadm 8 COMPLETED 0:0
2560745.31 Rscript arcadm 8 COMPLETED 0:0
2560745.32 Rscript arcadm 8 COMPLETED 0:0
2560745.33 Rscript arcadm 8 COMPLETED 0:0
2560745.34 Rscript arcadm 8 COMPLETED 0:0
2560745.35 Rscript arcadm 8 COMPLETED 0:0
2560745.36 Rscript arcadm 8 COMPLETED 0:0
2560745.37 Rscript arcadm 8 COMPLETED 0:0
2560745.38 Rscript arcadm 8 COMPLETED 0:0
2560745.39 Rscript arcadm 8 COMPLETED 0:0
2560745.40 Rscript arcadm 8 COMPLETED 0:0
2560745.41 Rscript arcadm 8 COMPLETED 0:0
2560745.42 Rscript arcadm 8 COMPLETED 0:0
2560745.43 Rscript arcadm 8 COMPLETED 0:0
2560745.44 Rscript arcadm 8 COMPLETED 0:0
2560745.45 Rscript arcadm 8 COMPLETED 0:0
2560745.46 Rscript arcadm 8 COMPLETED 0:0
2560745.47 Rscript arcadm 8 COMPLETED 0:0
2560745.48 Rscript arcadm 8 COMPLETED 0:0
2560745.49 Rscript arcadm 8 COMPLETED 0:0
2560745.50 Rscript arcadm 8 COMPLETED 0:0
2560745.51 Rscript arcadm 8 COMPLETED 0:0
2560745.52 Rscript arcadm 8 COMPLETED 0:0
2560745.53 Rscript arcadm 8 COMPLETED 0:0
2560745.54 Rscript arcadm 8 COMPLETED 0:0
2560745.55 Rscript arcadm 8 COMPLETED 0:0
2560745.56 Rscript arcadm 8 COMPLETED 0:0
2560745.57 Rscript arcadm 8 COMPLETED 0:0
2560745.58 Rscript arcadm 8 COMPLETED 0:0
2560745.59 Rscript arcadm 8 COMPLETED 0:0
2560745.60 Rscript arcadm 8 COMPLETED 0:0
2560745.61 Rscript arcadm 8 COMPLETED 0:0
2560745.62 Rscript arcadm 8 COMPLETED 0:0
2560745.63 Rscript arcadm 8 COMPLETED 0:0
2560745.64 Rscript arcadm 8 COMPLETED 0:0
2560745.65 Rscript arcadm 8 COMPLETED 0:0
2560745.66 Rscript arcadm 8 COMPLETED 0:0
2560745.67 Rscript arcadm 8 COMPLETED 0:0
2560745.68 Rscript arcadm 8 COMPLETED 0:0
2560745.69 Rscript arcadm 8 COMPLETED 0:0
2560745.70 Rscript arcadm 8 COMPLETED 0:0
2560745.71 Rscript arcadm 8 COMPLETED 0:0
2560745.72 Rscript arcadm 8 COMPLETED 0:0
2560745.73 Rscript arcadm 8 COMPLETED 0:0
2560745.74 Rscript arcadm 8 COMPLETED 0:0
2560745.75 Rscript arcadm 8 COMPLETED 0:0
2560745.76 Rscript arcadm 8 COMPLETED 0:0
2560745.77 Rscript arcadm 8 COMPLETED 0:0
2560745.78 Rscript arcadm 8 COMPLETED 0:0
2560745.79 Rscript arcadm 8 COMPLETED 0:0
2560745.80 Rscript arcadm 8 COMPLETED 0:0
2560745.81 Rscript arcadm 8 COMPLETED 0:0
2560745.82 Rscript arcadm 8 COMPLETED 0:0
2560745.83 Rscript arcadm 8 COMPLETED 0:0
2560745.84 Rscript arcadm 8 COMPLETED 0:0
2560745.85 Rscript arcadm 8 COMPLETED 0:0
2560745.86 Rscript arcadm 8 COMPLETED 0:0
2560745.87 Rscript arcadm 8 COMPLETED 0:0
2560745.88 Rscript arcadm 8 COMPLETED 0:0
2560745.89 Rscript arcadm 8 COMPLETED 0:0
2560745.90 Rscript arcadm 8 COMPLETED 0:0
2560745.91 Rscript arcadm 8 COMPLETED 0:0
2560745.92 Rscript arcadm 8 COMPLETED 0:0
2560745.93 Rscript arcadm 8 COMPLETED 0:0
2560745.94 Rscript arcadm 8 COMPLETED 0:0
2560745.95 Rscript arcadm 8 COMPLETED 0:0
2560745.96 Rscript arcadm 8 COMPLETED 0:0
2560745.97 Rscript arcadm 8 COMPLETED 0:0
2560745.98 Rscript arcadm 8 COMPLETED 0:0
2560745.99 Rscript arcadm 8 COMPLETED 0:0
2560745.100 Rscript arcadm 8 COMPLETED 0:0
2560745.101 Rscript arcadm 8 COMPLETED 0:0
2560745.102 Rscript arcadm 8 COMPLETED 0:0
2560745.103 Rscript arcadm 8 COMPLETED 0:0
2560745.104 Rscript arcadm 8 COMPLETED 0:0
2560745.105 Rscript arcadm 8 COMPLETED 0:0
2560745.106 Rscript arcadm 8 COMPLETED 0:0
2560745.107 Rscript arcadm 8 COMPLETED 0:0
2560745.108 Rscript arcadm 8 COMPLETED 0:0
2560745.109 Rscript arcadm 8 COMPLETED 0:0
2560745.110 Rscript arcadm 8 COMPLETED 0:0
2560745.111 Rscript arcadm 8 COMPLETED 0:0
2560745.112 Rscript arcadm 8 COMPLETED 0:0
2560745.113 Rscript arcadm 8 COMPLETED 0:0
2560745.114 Rscript arcadm 8 COMPLETED 0:0
2560745.115 Rscript arcadm 8 COMPLETED 0:0
2560745.116 Rscript arcadm 8 COMPLETED 0:0
2560745.117 Rscript arcadm 8 COMPLETED 0:0
2560745.118 Rscript arcadm 8 COMPLETED 0:0
2560745.119 Rscript arcadm 8 COMPLETED 0:0
2560745.120 Rscript arcadm 8 COMPLETED 0:0
2560745.121 Rscript arcadm 8 COMPLETED 0:0
2560745.122 Rscript arcadm 8 COMPLETED 0:0
2560745.123 Rscript arcadm 8 COMPLETED 0:0
2560745.124 Rscript arcadm 8 COMPLETED 0:0
2560745.125 Rscript arcadm 8 COMPLETED 0:0
2560745.126 Rscript arcadm 8 COMPLETED 0:0
2560745.127 Rscript arcadm 8 COMPLETED 0:0
2560745.128 Rscript arcadm 8 COMPLETED 0:0
2560745.129 Rscript arcadm 8 COMPLETED 0:0
2560745.130 Rscript arcadm 8 COMPLETED 0:0
2560745.131 Rscript arcadm 8 COMPLETED 0:0
2560745.132 Rscript arcadm 8 COMPLETED 0:0
2560745.133 Rscript arcadm 8 COMPLETED 0:0
2560745.134 Rscript arcadm 8 COMPLETED 0:0
2560745.135 Rscript arcadm 8 COMPLETED 0:0
2560745.136 Rscript arcadm 8 COMPLETED 0:0
2560745.137 Rscript arcadm 8 COMPLETED 0:0
2560745.138 Rscript arcadm 8 COMPLETED 0:0
2560745.139 Rscript arcadm 8 COMPLETED 0:0
2560745.140 Rscript arcadm 8 COMPLETED 0:0
2560745.141 Rscript arcadm 8 COMPLETED 0:0
2560745.142 Rscript arcadm 8 COMPLETED 0:0
2560745.143 Rscript arcadm 8 COMPLETED 0:0
2560745.144 Rscript arcadm 8 COMPLETED 0:0
2560745.145 Rscript arcadm 8 COMPLETED 0:0
2560745.146 Rscript arcadm 8 COMPLETED 0:0
2560745.147 Rscript arcadm 8 COMPLETED 0:0
2560745.148 Rscript arcadm 8 COMPLETED 0:0
2560745.149 Rscript arcadm 8 COMPLETED 0:0
2560745.150 Rscript arcadm 8 COMPLETED 0:0
2560745.151 Rscript arcadm 8 COMPLETED 0:0
2560745.152 Rscript arcadm 8 COMPLETED 0:0
2560745.153 Rscript arcadm 8 COMPLETED 0:0
2560745.154 Rscript arcadm 8 COMPLETED 0:0
2560745.155 Rscript arcadm 8 COMPLETED 0:0
2560745.156 Rscript arcadm 8 COMPLETED 0:0
2560745.157 Rscript arcadm 8 COMPLETED 0:0
2560745.158 Rscript arcadm 8 COMPLETED 0:0
2560745.159 Rscript arcadm 8 COMPLETED 0:0
2560745.160 Rscript arcadm 8 COMPLETED 0:0
2560745.161 Rscript arcadm 8 COMPLETED 0:0
2560745.162 Rscript arcadm 8 COMPLETED 0:0
2560745.163 Rscript arcadm 8 COMPLETED 0:0
2560745.164 Rscript arcadm 8 COMPLETED 0:0
2560745.165 Rscript arcadm 8 COMPLETED 0:0
2560745.166 Rscript arcadm 8 COMPLETED 0:0
2560745.167 Rscript arcadm 8 COMPLETED 0:0
2560745.168 Rscript arcadm 8 COMPLETED 0:0
2560745.169 Rscript arcadm 8 COMPLETED 0:0
2560745.170 Rscript arcadm 8 COMPLETED 0:0
2560745.171 Rscript arcadm 8 COMPLETED 0:0
2560745.172 Rscript arcadm 8 COMPLETED 0:0
2560745.173 Rscript arcadm 8 COMPLETED 0:0
2560745.174 Rscript arcadm 8 COMPLETED 0:0
2560745.175 Rscript arcadm 8 COMPLETED 0:0
2560745.176 Rscript arcadm 8 COMPLETED 0:0
2560745.177 Rscript arcadm 8 COMPLETED 0:0
2560745.178 Rscript arcadm 8 COMPLETED 0:0
2560745.179 Rscript arcadm 8 COMPLETED 0:0
2560745.180 Rscript arcadm 8 COMPLETED 0:0
2560745.181 Rscript arcadm 8 COMPLETED 0:0
2560745.182 Rscript arcadm 8 COMPLETED 0:0
2560745.183 Rscript arcadm 8 COMPLETED 0:0
2560745.184 Rscript arcadm 8 COMPLETED 0:0
2560745.185 Rscript arcadm 8 COMPLETED 0:0
2560745.186 Rscript arcadm 8 COMPLETED 0:0
2560745.187 Rscript arcadm 8 COMPLETED 0:0
2560745.188 Rscript arcadm 8 COMPLETED 0:0
2560745.189 Rscript arcadm 8 COMPLETED 0:0
2560745.190 Rscript arcadm 8 COMPLETED 0:0
2560745.191 Rscript arcadm 8 COMPLETED 0:0
2560745.192 Rscript arcadm 8 COMPLETED 0:0
2560745.193 Rscript arcadm 8 COMPLETED 0:0
2560745.194 Rscript arcadm 8 COMPLETED 0:0
2560745.195 Rscript arcadm 8 COMPLETED 0:0
2560745.196 Rscript arcadm 8 COMPLETED 0:0
2560745.197 Rscript arcadm 8 COMPLETED 0:0
2560745.198 Rscript arcadm 8 COMPLETED 0:0
2560745.199 Rscript arcadm 8 COMPLETED 0:0
2560745.200 Rscript arcadm 8 COMPLETED 0:0
2560745.201 Rscript arcadm 8 COMPLETED 0:0
2560745.202 Rscript arcadm 8 COMPLETED 0:0
2560745.203 Rscript arcadm 8 COMPLETED 0:0
2560745.204 Rscript arcadm 8 COMPLETED 0:0
2560745.205 Rscript arcadm 8 COMPLETED 0:0
2560745.206 Rscript arcadm 8 COMPLETED 0:0
2560745.207 Rscript arcadm 8 COMPLETED 0:0
2560745.208 Rscript arcadm 8 COMPLETED 0:0
2560745.209 Rscript arcadm 8 COMPLETED 0:0
2560745.210 Rscript arcadm 8 COMPLETED 0:0
2560745.211 Rscript arcadm 8 COMPLETED 0:0
2560745.212 Rscript arcadm 8 COMPLETED 0:0
2560745.213 Rscript arcadm 8 COMPLETED 0:0
2560745.214 Rscript arcadm 8 COMPLETED 0:0
2560745.215 Rscript arcadm 8 COMPLETED 0:0
2560745.216 Rscript arcadm 8 COMPLETED 0:0
2560745.217 Rscript arcadm 8 COMPLETED 0:0
2560745.218 Rscript arcadm 8 COMPLETED 0:0
2560745.219 Rscript arcadm 8 COMPLETED 0:0
2560745.220 Rscript arcadm 8 COMPLETED 0:0
2560745.221 Rscript arcadm 8 COMPLETED 0:0
2560745.222 Rscript arcadm 8 COMPLETED 0:0
2560745.223 Rscript arcadm 8 COMPLETED 0:0
2560745.224 Rscript arcadm 8 COMPLETED 0:0
2560745.225 Rscript arcadm 8 COMPLETED 0:0
2560745.226 Rscript arcadm 8 COMPLETED 0:0
2560745.227 Rscript arcadm 8 COMPLETED 0:0
2560745.228 Rscript arcadm 8 COMPLETED 0:0
2560745.229 Rscript arcadm 8 COMPLETED 0:0
2560745.230 Rscript arcadm 8 COMPLETED 0:0
2560745.231 Rscript arcadm 8 COMPLETED 0:0
2560745.232 Rscript arcadm 8 COMPLETED 0:0
2560745.233 Rscript arcadm 8 COMPLETED 0:0
2560745.234 Rscript arcadm 8 COMPLETED 0:0
2560745.235 Rscript arcadm 8 COMPLETED 0:0
2560745.236 Rscript arcadm 8 COMPLETED 0:0
2560745.237 Rscript arcadm 8 COMPLETED 0:0
2560745.238 Rscript arcadm 8 COMPLETED 0:0
2560745.239 Rscript arcadm 8 COMPLETED 0:0
2560745.240 Rscript arcadm 8 COMPLETED 0:0
2560745.241 Rscript arcadm 8 COMPLETED 0:0
2560745.242 Rscript arcadm 8 COMPLETED 0:0
2560745.243 Rscript arcadm 8 COMPLETED 0:0
2560745.244 Rscript arcadm 8 COMPLETED 0:0
2560745.245 Rscript arcadm 8 COMPLETED 0:0
2560745.246 Rscript arcadm 8 COMPLETED 0:0
2560745.247 Rscript arcadm 8 COMPLETED 0:0
2560745.248 Rscript arcadm 8 COMPLETED 0:0
2560745.249 Rscript arcadm 8 COMPLETED 0:0
2560745.250 Rscript arcadm 8 COMPLETED 0:0
2560745.251 Rscript arcadm 8 COMPLETED 0:0
2560745.252 Rscript arcadm 8 COMPLETED 0:0
2560745.253 Rscript arcadm 8 COMPLETED 0:0
2560745.254 Rscript arcadm 8 COMPLETED 0:0
2560745.255 Rscript arcadm 8 COMPLETED 0:0
2560745.256 Rscript arcadm 8 COMPLETED 0:0
2560745.257 Rscript arcadm 8 COMPLETED 0:0
2560745.258 Rscript arcadm 8 COMPLETED 0:0
2560745.259 Rscript arcadm 8 COMPLETED 0:0
2560745.260 Rscript arcadm 8 COMPLETED 0:0
2560745.261 Rscript arcadm 8 COMPLETED 0:0
2560745.262 Rscript arcadm 8 COMPLETED 0:0
2560745.263 Rscript arcadm 8 COMPLETED 0:0
2560745.264 Rscript arcadm 8 COMPLETED 0:0
2560745.265 Rscript arcadm 8 COMPLETED 0:0
2560745.266 Rscript arcadm 8 COMPLETED 0:0
2560745.267 Rscript arcadm 8 COMPLETED 0:0
2560745.268 Rscript arcadm 8 COMPLETED 0:0
2560745.269 Rscript arcadm 8 COMPLETED 0:0
2560745.270 Rscript arcadm 8 COMPLETED 0:0
2560745.271 Rscript arcadm 8 COMPLETED 0:0
2560745.272 Rscript arcadm 8 COMPLETED 0:0
2560745.273 Rscript arcadm 8 COMPLETED 0:0
2560745.274 Rscript arcadm 8 COMPLETED 0:0
2560745.275 Rscript arcadm 8 COMPLETED 0:0
2560745.276 Rscript arcadm 8 COMPLETED 0:0
2560745.277 Rscript arcadm 8 COMPLETED 0:0
2560745.278 Rscript arcadm 8 COMPLETED 0:0
2560745.279 Rscript arcadm 8 COMPLETED 0:0
2560745.280 Rscript arcadm 8 COMPLETED 0:0
2560745.281 Rscript arcadm 8 COMPLETED 0:0
2560745.282 Rscript arcadm 8 COMPLETED 0:0
2560745.283 Rscript arcadm 8 COMPLETED 0:0
2560745.284 Rscript arcadm 8 COMPLETED 0:0
2560745.285 Rscript arcadm 8 COMPLETED 0:0
2560745.286 Rscript arcadm 8 COMPLETED 0:0
2560745.287 Rscript arcadm 8 COMPLETED 0:0
2560745.288 Rscript arcadm 8 COMPLETED 0:0
2560745.289 Rscript arcadm 8 COMPLETED 0:0
2560745.290 Rscript arcadm 8 COMPLETED 0:0
2560745.291 Rscript arcadm 8 COMPLETED 0:0
2560745.292 Rscript arcadm 8 COMPLETED 0:0
2560745.293 Rscript arcadm 8 COMPLETED 0:0
2560745.294 Rscript arcadm 8 COMPLETED 0:0
2560745.295 Rscript arcadm 8 COMPLETED 0:0
2560745.296 Rscript arcadm 8 COMPLETED 0:0
2560745.297 Rscript arcadm 8 COMPLETED 0:0
2560745.298 Rscript arcadm 8 COMPLETED 0:0
2560745.299 Rscript arcadm 8 COMPLETED 0:0
2560745.300 Rscript arcadm 8 COMPLETED 0:0
2560745.301 Rscript arcadm 8 COMPLETED 0:0
2560745.302 Rscript arcadm 8 COMPLETED 0:0
2560745.303 Rscript arcadm 8 COMPLETED 0:0
2560745.304 Rscript arcadm 8 COMPLETED 0:0
2560745.305 Rscript arcadm 8 COMPLETED 0:0
2560745.306 Rscript arcadm 8 COMPLETED 0:0
2560745.307 Rscript arcadm 8 COMPLETED 0:0
2560745.308 Rscript arcadm 8 COMPLETED 0:0
2560745.309 Rscript arcadm 8 COMPLETED 0:0
2560745.310 Rscript arcadm 8 COMPLETED 0:0
2560745.311 Rscript arcadm 8 COMPLETED 0:0
2560745.312 Rscript arcadm 8 COMPLETED 0:0
2560745.313 Rscript arcadm 8 COMPLETED 0:0
2560745.314 Rscript arcadm 8 COMPLETED 0:0
2560745.315 Rscript arcadm 8 COMPLETED 0:0
2560745.316 Rscript arcadm 8 COMPLETED 0:0
2560745.317 Rscript arcadm 8 COMPLETED 0:0
2560745.318 Rscript arcadm 8 COMPLETED 0:0
2560745.319 Rscript arcadm 8 COMPLETED 0:0
2560745.320 Rscript arcadm 8 COMPLETED 0:0
2560745.321 Rscript arcadm 8 COMPLETED 0:0
2560745.322 Rscript arcadm 8 COMPLETED 0:0
2560745.323 Rscript arcadm 8 COMPLETED 0:0
2560745.324 Rscript arcadm 8 COMPLETED 0:0
2560745.325 Rscript arcadm 8 COMPLETED 0:0
2560745.326 Rscript arcadm 8 COMPLETED 0:0
2560745.327 Rscript arcadm 8 COMPLETED 0:0
2560745.328 Rscript arcadm 8 COMPLETED 0:0
2560745.329 Rscript arcadm 8 COMPLETED 0:0
2560745.330 Rscript arcadm 8 COMPLETED 0:0
2560745.331 Rscript arcadm 8 COMPLETED 0:0
2560745.332 Rscript arcadm 8 COMPLETED 0:0
2560745.333 Rscript arcadm 8 COMPLETED 0:0
2560745.334 Rscript arcadm 8 COMPLETED 0:0
2560745.335 Rscript arcadm 8 COMPLETED 0:0
2560745.336 Rscript arcadm 8 COMPLETED 0:0
2560745.337 Rscript arcadm 8 COMPLETED 0:0
2560745.338 Rscript arcadm 8 COMPLETED 0:0
2560745.339 Rscript arcadm 8 COMPLETED 0:0
2560745.340 Rscript arcadm 8 COMPLETED 0:0
2560745.341 Rscript arcadm 8 COMPLETED 0:0
2560745.342 Rscript arcadm 8 COMPLETED 0:0
2560745.343 Rscript arcadm 8 COMPLETED 0:0
2560745.344 Rscript arcadm 8 COMPLETED 0:0
2560745.345 Rscript arcadm 8 COMPLETED 0:0
2560745.346 Rscript arcadm 8 COMPLETED 0:0
2560745.347 Rscript arcadm 8 COMPLETED 0:0
2560745.348 Rscript arcadm 8 COMPLETED 0:0
2560745.349 Rscript arcadm 8 COMPLETED 0:0
2560745.350 Rscript arcadm 8 COMPLETED 0:0
2560745.351 Rscript arcadm 8 COMPLETED 0:0
2560745.352 Rscript arcadm 8 COMPLETED 0:0
2560745.353 Rscript arcadm 8 COMPLETED 0:0
2560745.354 Rscript arcadm 8 COMPLETED 0:0
2560745.355 Rscript arcadm 8 COMPLETED 0:0
2560745.356 Rscript arcadm 8 COMPLETED 0:0
2560745.357 Rscript arcadm 8 COMPLETED 0:0
2560745.358 Rscript arcadm 8 COMPLETED 0:0
2560745.359 Rscript arcadm 8 COMPLETED 0:0
2560745.360 Rscript arcadm 8 COMPLETED 0:0
2560745.361 Rscript arcadm 8 COMPLETED 0:0
2560745.362 Rscript arcadm 8 COMPLETED 0:0
2560745.363 Rscript arcadm 8 COMPLETED 0:0
2560745.364 Rscript arcadm 8 COMPLETED 0:0
2560745.365 Rscript arcadm 8 COMPLETED 0:0
2560745.366 Rscript arcadm 8 COMPLETED 0:0
2560745.367 Rscript arcadm 8 COMPLETED 0:0
2560745.368 Rscript arcadm 8 COMPLETED 0:0
2560745.369 Rscript arcadm 8 COMPLETED 0:0
2560745.370 Rscript arcadm 8 COMPLETED 0:0
2560745.371 Rscript arcadm 8 COMPLETED 0:0
2560745.372 Rscript arcadm 8 COMPLETED 0:0
2560745.373 Rscript arcadm 8 COMPLETED 0:0
2560745.374 Rscript arcadm 8 COMPLETED 0:0
2560745.375 Rscript arcadm 8 COMPLETED 0:0
2560745.376 Rscript arcadm 8 COMPLETED 0:0
2560745.377 Rscript arcadm 8 COMPLETED 0:0
2560745.378 Rscript arcadm 8 COMPLETED 0:0
2560745.379 Rscript arcadm 8 COMPLETED 0:0
2560745.380 Rscript arcadm 8 COMPLETED 0:0
2560745.381 Rscript arcadm 8 COMPLETED 0:0
2560745.382 Rscript arcadm 8 COMPLETED 0:0
2560745.383 Rscript arcadm 8 COMPLETED 0:0
2560745.384 Rscript arcadm 8 COMPLETED 0:0
2560745.385 Rscript arcadm 8 COMPLETED 0:0
2560745.386 Rscript arcadm 8 COMPLETED 0:0
2560745.387 Rscript arcadm 8 COMPLETED 0:0
2560745.388 Rscript arcadm 8 COMPLETED 0:0
2560745.389 Rscript arcadm 8 COMPLETED 0:0
2560745.390 Rscript arcadm 8 COMPLETED 0:0
2560745.391 Rscript arcadm 8 COMPLETED 0:0
2560745.392 Rscript arcadm 8 COMPLETED 0:0
2560745.393 Rscript arcadm 8 COMPLETED 0:0
2560745.394 Rscript arcadm 8 COMPLETED 0:0
2560745.395 Rscript arcadm 8 COMPLETED 0:0
2560745.396 Rscript arcadm 8 COMPLETED 0:0
2560745.397 Rscript arcadm 8 COMPLETED 0:0
2560745.398 Rscript arcadm 8 COMPLETED 0:0
2560745.399 Rscript arcadm 8 COMPLETED 0:0
2560745.400 Rscript arcadm 8 COMPLETED 0:0
2560745.401 Rscript arcadm 8 COMPLETED 0:0
2560745.402 Rscript arcadm 8 COMPLETED 0:0
2560745.403 Rscript arcadm 8 COMPLETED 0:0
2560745.404 Rscript arcadm 8 COMPLETED 0:0
2560745.405 Rscript arcadm 8 COMPLETED 0:0
2560745.406 Rscript arcadm 8 COMPLETED 0:0
2560745.407 Rscript arcadm 8 COMPLETED 0:0
2560745.408 Rscript arcadm 8 COMPLETED 0:0
2560745.409 Rscript arcadm 8 COMPLETED 0:0
2560745.410 Rscript arcadm 8 COMPLETED 0:0
2560745.411 Rscript arcadm 8 COMPLETED 0:0
2560745.412 Rscript arcadm 8 COMPLETED 0:0
2560745.413 Rscript arcadm 8 COMPLETED 0:0
2560745.414 Rscript arcadm 8 COMPLETED 0:0
2560745.415 Rscript arcadm 8 COMPLETED 0:0
2560745.416 Rscript arcadm 8 COMPLETED 0:0
2560745.417 Rscript arcadm 8 COMPLETED 0:0
2560745.418 Rscript arcadm 8 COMPLETED 0:0
2560745.419 Rscript arcadm 8 COMPLETED 0:0
2560745.420 Rscript arcadm 8 COMPLETED 0:0
2560745.421 Rscript arcadm 8 COMPLETED 0:0
2560745.422 Rscript arcadm 8 COMPLETED 0:0
2560745.423 Rscript arcadm 8 COMPLETED 0:0
2560745.424 Rscript arcadm 8 COMPLETED 0:0
2560745.425 Rscript arcadm 8 COMPLETED 0:0
2560745.426 Rscript arcadm 8 COMPLETED 0:0
2560745.427 Rscript arcadm 8 COMPLETED 0:0
2560745.428 Rscript arcadm 8 COMPLETED 0:0
2560745.429 Rscript arcadm 8 COMPLETED 0:0
2560745.430 Rscript arcadm 8 COMPLETED 0:0
2560745.431 Rscript arcadm 8 COMPLETED 0:0
2560745.432 Rscript arcadm 8 COMPLETED 0:0
2560745.433 Rscript arcadm 8 COMPLETED 0:0
2560745.434 Rscript arcadm 8 COMPLETED 0:0
2560745.435 Rscript arcadm 8 COMPLETED 0:0
2560745.436 Rscript arcadm 8 COMPLETED 0:0
2560745.437 Rscript arcadm 8 COMPLETED 0:0
2560745.438 Rscript arcadm 8 COMPLETED 0:0
2560745.439 Rscript arcadm 8 COMPLETED 0:0
2560745.440 Rscript arcadm 8 COMPLETED 0:0
2560745.441 Rscript arcadm 8 COMPLETED 0:0
2560745.442 Rscript arcadm 8 COMPLETED 0:0
2560745.443 Rscript arcadm 8 COMPLETED 0:0
2560745.444 Rscript arcadm 8 COMPLETED 0:0
2560745.445 Rscript arcadm 8 COMPLETED 0:0
2560745.446 Rscript arcadm 8 COMPLETED 0:0
2560745.447 Rscript arcadm 8 COMPLETED 0:0
2560745.448 Rscript arcadm 8 COMPLETED 0:0
2560745.449 Rscript arcadm 8 COMPLETED 0:0
2560745.450 Rscript arcadm 8 COMPLETED 0:0
2560745.451 Rscript arcadm 8 COMPLETED 0:0
2560745.452 Rscript arcadm 8 COMPLETED 0:0
2560745.453 Rscript arcadm 8 COMPLETED 0:0
2560745.454 Rscript arcadm 8 COMPLETED 0:0
2560745.455 Rscript arcadm 8 COMPLETED 0:0
2560745.456 Rscript arcadm 8 COMPLETED 0:0
2560745.457 Rscript arcadm 8 COMPLETED 0:0
2560745.458 Rscript arcadm 8 COMPLETED 0:0
2560745.459 Rscript arcadm 8 COMPLETED 0:0
2560745.460 Rscript arcadm 8 COMPLETED 0:0
2560745.461 Rscript arcadm 8 COMPLETED 0:0
2560745.462 Rscript arcadm 8 COMPLETED 0:0
2560745.463 Rscript arcadm 8 COMPLETED 0:0
2560745.464 Rscript arcadm 8 COMPLETED 0:0
2560745.465 Rscript arcadm 8 COMPLETED 0:0
2560745.466 Rscript arcadm 8 COMPLETED 0:0
2560745.467 Rscript arcadm 8 COMPLETED 0:0
2560745.468 Rscript arcadm 8 COMPLETED 0:0
2560745.469 Rscript arcadm 8 COMPLETED 0:0
2560745.470 Rscript arcadm 8 COMPLETED 0:0
2560745.471 Rscript arcadm 8 COMPLETED 0:0
2560745.472 Rscript arcadm 8 COMPLETED 0:0
2560745.473 Rscript arcadm 8 COMPLETED 0:0
2560745.474 Rscript arcadm 8 COMPLETED 0:0
2560745.475 Rscript arcadm 8 COMPLETED 0:0
2560745.476 Rscript arcadm 8 COMPLETED 0:0
2560745.477 Rscript arcadm 8 COMPLETED 0:0
2560745.478 Rscript arcadm 8 COMPLETED 0:0
2560745.479 Rscript arcadm 8 COMPLETED 0:0
2560745.480 Rscript arcadm 8 COMPLETED 0:0
2560745.481 Rscript arcadm 8 COMPLETED 0:0
2560745.482 Rscript arcadm 8 COMPLETED 0:0
2560745.483 Rscript arcadm 8 COMPLETED 0:0
2560745.484 Rscript arcadm 8 COMPLETED 0:0
2560745.485 Rscript arcadm 8 COMPLETED 0:0
2560745.486 Rscript arcadm 8 COMPLETED 0:0
2560745.487 Rscript arcadm 8 COMPLETED 0:0
2560745.488 Rscript arcadm 8 COMPLETED 0:0
2560745.489 Rscript arcadm 8 COMPLETED 0:0
2560745.490 Rscript arcadm 8 COMPLETED 0:0
2560745.491 Rscript arcadm 8 COMPLETED 0:0
2560745.492 Rscript arcadm 8 COMPLETED 0:0
2560745.493 Rscript arcadm 8 COMPLETED 0:0
2560745.494 Rscript arcadm 8 COMPLETED 0:0
2560745.495 Rscript arcadm 8 COMPLETED 0:0
2560745.496 Rscript arcadm 8 COMPLETED 0:0
2560745.497 Rscript arcadm 8 COMPLETED 0:0
2560745.498 Rscript arcadm 8 COMPLETED 0:0
2560745.499 Rscript arcadm 8 COMPLETED 0:0
2560745.500 Rscript arcadm 8 COMPLETED 0:0
2560745.501 Rscript arcadm 8 COMPLETED 0:0
2560745.502 Rscript arcadm 8 COMPLETED 0:0
2560745.503 Rscript arcadm 8 COMPLETED 0:0
2560745.504 Rscript arcadm 8 COMPLETED 0:0
2560745.505 Rscript arcadm 8 COMPLETED 0:0
2560745.506 Rscript arcadm 8 COMPLETED 0:0
2560745.507 Rscript arcadm 8 COMPLETED 0:0
2560745.508 Rscript arcadm 8 COMPLETED 0:0
2560745.509 Rscript arcadm 8 COMPLETED 0:0
2560745.510 Rscript arcadm 8 COMPLETED 0:0
2560745.511 Rscript arcadm 8 COMPLETED 0:0
Max number of srun steps
In order to manage fairness of this valuable shared resource, there are limits on requests. One relevant limit here is the number of srun calls (aka steps):
[slima2@tinkercliffs1 ~]$ scontrol show config | grep -i step
InteractiveStepOptions = --interactive --preserve-env --pty $SHELL
MaxStepCount = 40000
UnkillableStepProgram = (null)
UnkillableStepTimeout = 180 sec
Therefore, a single job with the setup above will be limited to 40000 configurations, unless multiple configurations are passed to each call of the working script.
Python example with GPUs
Below is the working python script test_args.py that prints not only the arguments, but also the hostname which that particular instance of the program is running on:
import argparse
import os
if __name__ == "__main__":
print("Hello, testing args from... ", {os.uname()})
parser = argparse.ArgumentParser(prog='Parallel_Python', description='Read arguments for launching an instance of the program.', epilog='Example with header and configuration')
parser.add_argument("-t", "--header", help="config header")
parser.add_argument("-l", "--line", help="config line")
args = parser.parse_args()
H = args.header.split(',')
L = args.line.split(',')
fmt_str = "Arguments:: "
for h, l in zip(H, L):
fmt_str += h+': '+l+' '
print(fmt_str+'\n')
We will use a csv file config.csv shown below with 5 hyperparameter configurations, passing each line as a single argument to the python script:
Model_Name,alpha,gamma
"model1",1,1
"model2",1,2
"model3",2,1
"model4",2,2
"model5",2,3
Srun calling parallel (srun ... parallel ...)
The following example requests 2 t4 nodes on Infer with GPUs, and runs 2 tasks simultaneously (=1task*2nodes). Our batch job script multinode_par.job looks like this:
#!/bin/bash
#SBATCH -A XXX
#SBATCH -p t4_normal_q
#SBATCH -N 2
#SBATCH --gres=gpu:1
#SBATCH --ntasks-per-node=1
#SBATCH -J parPy
#SBATCH -D MyProjects/parallel_Py
#SBATCH -o ./job_outputs/slurm-%j-%x.out
echo "date: `date`"
echo "hostname: $HOSTNAME"; echo
echo -e "\nChecking job details for CPU_IDs..."
scontrol show job --details $SLURM_JOB_ID
# Generate configurations and write commands to file
configs=$(cat config.csv)
header=$(echo "$configs" | head -1)
lines=$(echo "$configs" | tail -n +2)
parallel --dry-run python test_args.py -t {1} -l {2} ::: $header ::: $lines > commands.txt
srun --ntasks=$SLURM_JOB_NUM_NODES --gres=gpu:1 bash par_driver.sh commands.txt
Our driver script par_driver.sh which calls the working script is the same as the driver_fn previous example:
#!/bin/bash
echo "$SLURM_NODEID"
cat $1 | \
awk -v NNODE="$SLURM_NNODES" -v NODEID="$SLURM_NODEID" 'NR % NNODE == NODEID' | \
parallel -j $SLURM_NTASKS_PER_NODE {}
Examining our resource utilization of the job after successfully completing, we can see that our job indeed was allocated 2 nodes as expected.
-bash-4.2$ seff 296285
Job ID: 296285
Cluster: infer
User/Group: slima2/slima2
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 1
CPU Utilized: 00:00:01
CPU Efficiency: 50.00% of 00:00:02 core-walltime
Job Wall-clock time: 00:00:01
Memory Utilized: 2.57 MB (estimated maximum)
Memory Efficiency: 0.02% of 11.12 GB (5.56 GB/core)
Parallel calling srun (parallel ... srun ...)
This setup requests the same amount of memory and processing resources, also with 2 tasks simultaneously:
#!/bin/bash
#SBATCH -A XXX
#SBATCH -p t4_normal_q
#SBATCH --nodes=2
#SBATCH --gres=gpu:1
#SBATCH --ntasks-per-node=1
#SBATCH -D MyProjectDirectory
#SBATCH -o ./job_outputs/slurm-%j-par.out
date
scontrol show job --details $SLURM_JOB_ID
configs=$(cat config.csv)
header=$(echo "$configs" | head -1)
lines=$(echo "$configs" | tail -n +2)
parallel -j $SLURM_NTASKS srun --nodes=1 --ntasks=1 --ntasks-per-node=1 --gres=gpu:1 python test_args.py -h {1} -l {2} ::: $header ::: $lines
With seff we can see that we were allocated 2 nodes, with a total of ~12GB of RAM:
-bash-4.2$ seff 293488
Job ID: 293488
Cluster: infer
User/Group: slima2/slima2
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 1
CPU Utilized: 00:00:00
CPU Efficiency: 0.00% of 00:00:02 core-walltime
Job Wall-clock time: 00:00:01
Memory Utilized: 1.15 MB
Memory Efficiency: 0.01% of 11.12 GB
After this batch job completes, we can check the IDs of our allocated resources and confirm that they line up with our requests as expected. Below is the expected output from this example showing that both nodes inf[007,012] are utilized for the job:
Mon Jul 15 10:30:02 EDT 2024
JobId=293488 JobName=par_py.job
UserId=slima2(22539809) GroupId=slima2(22539809) MCS_label=N/A
Priority=1210 Nice=0 Account=arcadm QOS=normal
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
DerivedExitCode=0:0
RunTime=00:00:00 TimeLimit=01:00:00 TimeMin=N/A
SubmitTime=2024-07-15T10:46:04 EligibleTime=2024-07-15T10:46:04
AccrueTime=2024-07-15T10:46:04
StartTime=2024-07-15T10:46:33 EndTime=2024-07-15T11:46:33 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-07-15T10:46:33 Scheduler=Backfill
Partition=t4_normal_q AllocNode:Sid=infer1:4373
ReqNodeList=(null) ExcNodeList=(null)
NodeList=inf[007,012]
BatchHost=inf007
NumNodes=2 NumCPUs=2 NumTasks=2 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
ReqTRES=cpu=2,mem=11392M,node=2,gres/gpu=2
AllocTRES=cpu=2,mem=11392M,node=2,gres/gpu=2,gres/gpu:turing=2
Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=*
JOB_GRES=gpu:turing:2
Nodes=inf007 CPU_IDs=0 Mem=5696 GRES=gpu:turing:1(IDX:0)
Nodes=inf012 CPU_IDs=20 Mem=5696 GRES=gpu:turing:1(IDX:0)
MinCPUsNode=1 MinMemoryCPU=5696M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/home/slima2/MyProjects/parallel_Py/par_py.job
WorkDir=/home/slima2/MyProjects/parallel_Py
StdErr=/home/slima2/MyProjects/parallel_Py/./job_outputs/slurm-293488-par.out
StdIn=/dev/null
StdOut=/home/slima2/MyProjects/parallel_Py/./job_outputs/slurm-293488-par.out
Power=
TresPerNode=gres:gpu:1
('Hello, testing args from... ', set([('Linux', 'inf007', '3.10.0-1160.90.1.el7.x86_64', '#1 SMP Thu May 4 15:21:22 UTC 2023', 'x86_64')]))
Arguments:: Model_Name: "model1" alpha: 1 gamma: 1
('Hello, testing args from... ', set([('Linux', 'inf012', '3.10.0-1160.90.1.el7.x86_64', '#1 SMP Thu May 4 15:21:22 UTC 2023', 'x86_64')]))
Arguments:: Model_Name: "model2" alpha: 1 gamma: 2
('Hello, testing args from... ', set([('Linux', 'inf007', '3.10.0-1160.90.1.el7.x86_64', '#1 SMP Thu May 4 15:21:22 UTC 2023', 'x86_64')]))
Arguments:: Model_Name: "model3" alpha: 2 gamma: 1
('Hello, testing args from... ', set([('Linux', 'inf012', '3.10.0-1160.90.1.el7.x86_64', '#1 SMP Thu May 4 15:21:22 UTC 2023', 'x86_64')]))
Arguments:: Model_Name: "model4" alpha: 2 gamma: 2
('Hello, testing args from... ', set([('Linux', 'inf007', '3.10.0-1160.90.1.el7.x86_64', '#1 SMP Thu May 4 15:21:22 UTC 2023', 'x86_64')]))
Arguments:: Model_Name: "model5" alpha: 2 gamma: 3
References
This documentation site was developed primarily in reference to material presented at PEARC24 conference.
https://github.com/ketancmaheshwari/pearc24tut
https://rcc-uchicago.github.io/user-guide/tutorials/kicp/#gnu-parallel
https://rcc-uchicago.github.io/user-guide/slurm/sbatch/#gnu-parallel
https://ulhpc-tutorials.readthedocs.io/en/latest/sequential/basics/
https://ulhpc-tutorials.readthedocs.io/en/latest/sequential/manytasks-manynodes/
https://docs-research-it.berkeley.edu/services/high-performance-computing/user-guide/running-your-jobs/gnu-parallel/
https://curc.readthedocs.io/en/latest/software/GNUParallel.html