| Both sides previous revision
Previous revision
|
Next revision
Both sides next revision
|
gpu [2018/12/01 21:09]
popel [Rules] -pe smp now works with gpu |
gpu [2018/12/16 16:26]
straka [Rules] |
| ===== Rules ===== | ===== Rules ===== |
| * First, read [[internal:Linux network]] and [[:Grid]]. | * First, read [[internal:Linux network]] and [[:Grid]]. |
| * All the rules from [[:Grid]] apply, even more strictly than for CPU because there are too many GPU users and not as many GPUs available. So as a reminder: always use GPUs via ''qsub'' (or ''qrsh''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU. Don't forget to specify you RAM requirements with e.g. ''-l mem_free=8G,act_mem_free=8G,h_vmem=12G''. | * All the rules from [[:Grid]] apply, even more strictly than for CPU because there are too many GPU users and not as many GPUs available. So as a reminder: always use GPUs via ''qsub'' (or ''qrsh''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU. Don't forget to specify you RAM requirements with e.g. ''-l mem_free=8G,act_mem_free=8G,h_data=12G''. |
| | * **Note that you need to use ''h_data'' instead of ''h_vmem'' for GPU jobs.** CUDA driver allocates a lot of "unused" virtual memory (tens of GB per card), which is counted in ''h_vmem'', but not in ''h_data''. All usual allocations (''malloc'', ''new'', Python allocations) seem to be included in ''h_data''. |
| * Always specify the number of GPU cards (e.g. ''gpu=1''), the minimal Cuda capability you need (e.g. ''gpu_cc_min3.5=1'') and your GPU memory requirements (e.g. ''gpu_ram=2G''). Thus e.g. <code>qsub -q gpu-ms.q -l gpu=1,gpu_cc_min3.5=1,gpu_ram=2G</code> | * Always specify the number of GPU cards (e.g. ''gpu=1''), the minimal Cuda capability you need (e.g. ''gpu_cc_min3.5=1'') and your GPU memory requirements (e.g. ''gpu_ram=2G''). Thus e.g. <code>qsub -q gpu-ms.q -l gpu=1,gpu_cc_min3.5=1,gpu_ram=2G</code> |
| * If you need more than one GPU card (on a single machine), always require as many CPU cores (''-pe smp X'') as many GPU cards you need. E.g. <code>qsub -q gpu-ms.q -l gpu=4,gpu_cc_min3.5=1,gpu_ram=7G -pe smp 4</code> | * If you need more than one GPU card (on a single machine), always require as many CPU cores (''-pe smp X'') as many GPU cards you need. E.g. <code>qsub -q gpu-ms.q -l gpu=4,gpu_cc_min3.5=1,gpu_ram=7G -pe smp 4</code> |
