Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
slurm [2022/09/13 09:59] vodrazka [ÚFAL Grid Engine (LRC)] |
slurm [2022/10/17 11:43] fucik [Interactive mode] |
| cpu-troja | 7x CPU | default partition | | | cpu-troja | 7x CPU | default partition | |
| gpu-troja | 6x GPU | features: gpuram48G,gpuram40G | | | gpu-troja | 6x GPU | features: gpuram48G,gpuram40G | |
| cpu-ms | 7x GPU | features: gpuram48G,gpuram24G | | | gpu-ms | 7x GPU | features: gpuram48G,gpuram24G | |
| |
| In order to submit a job you need to login to one of the head nodes: |
| |
| lrc1.ufal.hide.ms.mff.cuni.cz |
| lrc2.ufal.hide.ms.mff.cuni.cz |
===== Basic usage ===== | ===== Basic usage ===== |
| |
#SBATCH -N 2 # number of nodes (default 1) | #SBATCH -N 2 # number of nodes (default 1) |
#SBATCH --nodelist=node1,node2... # required node, or comma separated list of required nodes | #SBATCH --nodelist=node1,node2... # required node, or comma separated list of required nodes |
#SBATCH -c 4 # number of cores/threads per task (default 1) | #SBATCH --cpus-per-task=4 # number of cores/threads per task (default 1) |
#SBATCH --gres=gpu:1 # number of GPUs to request (default 0) | #SBATCH --gres=gpu:1 # number of GPUs to request (default 0) |
#SBATCH --mem=10G # request 10 gigabytes memory (per node, default depends on node) | #SBATCH --mem=10G # request 10 gigabytes memory (per node, default depends on node) |
</code> | </code> |
| |
| === CPU core allocation === |
| |
| The minimal computing resource in SLURM is one CPU core. However, CPU count advertised by SLURM corresponds to the number of CPU threads. |
| If you ask for 1 CPU core with <code>--cpus-per-task=1</code> SLURM will allocate all threads of 1 CPU core. |
| |
| For example ''dll-8gpu1'' will allocate 2 threads since its ThreadsPerCore=2: |
| |
| <code> |
| $> scontrol show node dll-8gpu1 |
| $ scontrol show node dll-8gpu1 |
| NodeName=dll-8gpu1 Arch=x86_64 CoresPerSocket=16 |
| CPUAlloc=0 CPUTot=64 CPULoad=0.05 // CPUAlloc - allocated threads, CPUTot - total threads |
| AvailableFeatures=gpuram24G |
| ActiveFeatures=gpuram24G |
| Gres=gpu:nvidia_a30:8(S:0-1) |
| NodeAddr=10.10.24.63 NodeHostName=dll-8gpu1 Version=21.08.8-2 |
| OS=Linux 5.15.35-1-pve #1 SMP PVE 5.15.35-3 (Wed, 11 May 2022 07:57:51 +0200) |
| RealMemory=515838 AllocMem=0 FreeMem=507650 Sockets=2 Boards=1 |
| CoreSpecCount=1 CPUSpecList=62-63 // CoreSpecCount - cores reserved for OS, CPUSpecList - list of threads reserved for system |
| State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A // ThreadsPerCore - count of threads for 1 CPU core |
| Partitions=gpu-ms |
| BootTime=2022-09-01T14:07:50 SlurmdStartTime=2022-09-02T13:54:05 |
| LastBusyTime=2022-10-02T20:17:09 |
| CfgTRES=cpu=64,mem=515838M,billing=64 |
| AllocTRES= |
| CapWatts=n/a |
| CurrentWatts=0 AveWatts=0 |
| ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s |
| </code> |
| |
| In the example above you can see comments at all lines relevant to CPU allocation. |
| |
| |
There are many more parameters available to use. For example: | There are many more parameters available to use. For example: |
| |
<code>srun -p cpu-troja --mem=64G --pty bash</code> | **To get an interactive CPU job with 64GB of reserved memory:** |
| <code>srun -p cpu-troja,cpu-ms --mem=64G --pty bash</code> |
| |
* ''-p cpu-troja'' explicitly requires partition ''cpu-troja''. If not specified slurm will use default partition. | * ''-p cpu-troja'' explicitly requires partition ''cpu-troja''. If not specified slurm will use default partition. |
* ''--mem=64G'' requires 64G of memory for the job | * ''--mem=64G'' requires 64G of memory for the job |
| |
| **To get interactive job with a single GPU of any kind:** |
| <code>srun -p gpu-troja,gpu-ms --gres=gpu:1 --pty bash</code> |
| * ''-p gpu-troja,gpu-ms'' require only nodes from these two partitions |
| * ''--gres=gpu:1'' requires 1 GPUs |
| |
<code>srun -p gpu-troja,gpu-ms --nodelist=tdll-3gpu1 --mem=64G --gres=gpu:2 --pty bash</code> | <code>srun -p gpu-troja,gpu-ms --nodelist=tdll-3gpu1 --mem=64G --gres=gpu:2 --pty bash</code> |
* ''--gres=gpu:2'' requires 2 GPUs | * ''--gres=gpu:2'' requires 2 GPUs |
| |
<code>srun -p gpu-troja --constraint="gpuram44G|gpuram39G" --mem=64G --gres=gpu:2 --pty bash</code> | <code>srun -p gpu-troja --constraint="gpuram48G|gpuram40G" --mem=64G --gres=gpu:2 --pty bash</code> |
* ''--constraint="gpuram44G|gpuram39G"'' only consider nodes that have either ''gpuram44G'' or ''gpuram39G'' feature defined | * ''--constraint="gpuram48G|gpuram40G"'' only consider nodes that have either ''gpuram48G'' or ''gpuram40G'' feature defined |
| |
To see all the available options type: | To see all the available options type: |