[ Skip to the content ]

Institute of Formal and Applied Linguistics Wiki

[ Back to the navigation ]

You are here: start » slurm

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
slurm [2023/03/15 17:30]
rosa [Batch mode] Rudolfs template 		slurm [2023/04/13 17:01]
dusek Priority
Line 2: Line 2:
  
 LRC (Linguistic Research Cluster) is the name of ÚFAL's computational grid/cluster. The cluster is built on top of [[https://slurm.schedmd.com/|SLURM]] and is using [[https://www.lustre.org/|Lustre]] for [[internal:linux-network#directory-structure|data storage]]. LRC (Linguistic Research Cluster) is the name of ÚFAL's computational grid/cluster. The cluster is built on top of [[https://slurm.schedmd.com/|SLURM]] and is using [[https://www.lustre.org/|Lustre]] for [[internal:linux-network#directory-structure|data storage]].
 +
 +See Milan Straka's intro to Slurm (and Spark if you want):
 +
 +  * https://lectures.ms.mff.cuni.cz/video/rec/npfl118/2223/npfl118-2223-winter-slurm.mp4
 +  * https://lectures.ms.mff.cuni.cz/video/rec/npfl118/2223/npfl118-2223-winter-spark.mp4
  
 Currently there are following partitions (queues) available for computing: Currently there are following partitions (queues) available for computing:
Line 33: Line 38:
 | dll-3gpu[1-5] | 64 | 2:16:2 | 128642 | gpuram48G gpu_cc8.6 | NVIDIA A40 | | dll-3gpu[1-5] | 64 | 2:16:2 | 128642 | gpuram48G gpu_cc8.6 | NVIDIA A40 |
 | dll-4gpu[1,2] | 40 | 2:10:2 | 187978 | gpuram24G gpu_cc8.6 | NVIDIA RTX 3090 | | dll-4gpu[1,2] | 40 | 2:10:2 | 187978 | gpuram24G gpu_cc8.6 | NVIDIA RTX 3090 |
 +| dll-4gpu3 | 62 | 1:32:2 | 515652 | gpuram48G gpu_cc8.9 | NVIDIA L40 |
 +| dll-4gpu4 | 30 | 1:16:2 | 257616 | gpuram48G gpu_cc8.6 | NVIDIA A40 |
 | dll-8gpu[1,2] | 64 | 2:16:2 | 515838 | gpuram24G gpu_cc8.0 | NVIDIA A30 | | dll-8gpu[1,2] | 64 | 2:16:2 | 515838 | gpuram24G gpu_cc8.0 | NVIDIA A30 |
 | dll-8gpu[3,4] | 32 | 2:8:2 | 257830 | gpuram16G gpu_cc8.6 | NVIDIA RTX A4000 | | dll-8gpu[3,4] | 32 | 2:8:2 | 257830 | gpuram16G gpu_cc8.6 | NVIDIA RTX A4000 |
Line 199: Line 206:
 In the example above you can see comments at all lines relevant to CPU allocation. In the example above you can see comments at all lines relevant to CPU allocation.
  
 +=== Priority ====
  
 +When running srun or sbatch, you can pass `-q high/normal/low/preempt-low`. These represent priorities 300/200/100/100, with `normal` (200) being the default. Furthermore, the `preempt-low` QOS is actually preemptible -- if there is a job with normal or high QOS, they can interrupt your `preempt-low` job.
  
 +The preemption has probably not been used by anyone yet; some documentation about it is on https://slurm.schedmd.com/preempt.html, we use the REQUEUE regime (so your job is killed, very likely with some signal, so you could monitor it and for example save a checkpoint; but currently I do not know any details), and then started again when there are resources.
  
 ==== Interactive mode ==== ==== Interactive mode ====
Line 237: Line 247:
  
 <code>man srun</code> <code>man srun</code>
 +
 +==== Basic commands on cluster machines ====
 +
 +  lspci
 +    # is any such hardware there?
 +  nvidia-smi
 +    # more details, incl. running processes on the GPU
 +    # nvidia-* are typically located in /usr/bin
 +  watch nvidia-smi
 +    # For monitoring GPU activity in a separate terminal (thanks to Jindrich Libovicky for this!)
 +    # You can also use nvidia-smi -l TIME
 +  nvcc --version
 +    # this should tell CUDA version
 +    # nvcc is typically installed in /usr/local/cuda/bin/
 +  theano-test
 +    # dela to vubec neco uzitecneho? :-)
 +    # theano-* are typically located in /usr/local/bin/
 +  /usr/local/cuda/samples/1_Utilities/deviceQuery/deviceQuery
 +    # shows CUDA capability etc.
 +  ssh dll1; ~popel/bin/gpu_allocations
 +    # who occupies which card on a given machine
 +    
 +
  
 ===== See also ===== ===== See also =====

[ Back to the navigation ] [ Back to the content ]