Both sides previous revision
Previous revision
Next revision
|
Previous revision
Next revision
Both sides next revision
|
gpu [2017/10/11 13:07] bojar people should not set CUDA_VISIBLE_DEVICES |
gpu [2017/11/23 14:15] bojar link to munin graphs |
All machines have CUDA8.0 and should support both Theano and TensorFlow. | All machines have CUDA8.0 and should support both Theano and TensorFlow. |
| |
=== Disk space === | [[https://ufaladm2.ufal.hide.ms.mff.cuni.cz/munin/ufal.hide.ms.mff.cuni.cz/lrc-headnode.ufal.hide.ms.mff.cuni.cz/index.html#dll|GPU usage rolling graphs]] |
All the GPU machines are at Malá Strana (not at Troja), so you should not use ''/lnet/tspec/work/'', but you should use: | |
- ''/lnet/spec/work/'' (alias ''/net/work/'') - Lustre disk space at Malá Strana | |
- ''/net/cluster/TMP'' - NFS hard disk for temporary files, so slower than Lustre for most tasks | |
- ''/net/cluster/SSD'' - also NFS, but faster then TMP because of SSD | |
- ''/COMP.TMP'' - local (for each machine) space for temporary files (use it instead of ''/tmp''; over-filling ''/COMP.TMP'' should not halt the system). | |
| |
=== Individual acquisitions: NVIDIA Academic Hardware Grants == | |
| |
There is an easy way to get one high-end GPU: [[https://developer.nvidia.com/academic_gpu_seeding|ask NVIDIA for an Academic Hardware Grant]]. All it takes is writing a short grant application (at most ~2 hrs of work from scratch; if you have a GAUK, ~15 minutes of copy-pasting). Due to the GPU housing issues (mainly rack space and cooling), Milan F. said we should request the Tesla-line cards (2017 check with Milan about this issue). If you want to have a look at an application, feel free to ask at hajicj@ufal.mff.cuni.cz :) | ===== Rules ===== |
| * First, read [[internal:Linux network]] and [[:Grid]]. |
Take care, however, to coordinate the grant applications a little, so that not too many arrive from UFAL within a short time: these grants are explicitly //not// intended to build GPU clusters, they are "seeding" grants meant for researchers to try out GPUs (and fall in love with them, and buy a cluster later). If you are planning to submit the hardware grant, have submitted one, or have already been awarded one, please add yourself here. | * All the rules from [[:Grid]] apply, even more strictly than for CPU because there are too many GPU users and not as many GPUs available. So as a reminder: always use GPUs via ''qsub'' (or ''qrsh''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU. Don't forget to specify you RAM requirements with e.g. ''-l mem_free=8G,act_mem_free=8G,h_vmem=12G''. |
| * Always specify the number of GPU cards (e.g. ''gpu=1''), the minimal Cuda capability you need (e.g. ''gpu_cc_min3.5=1'') and you GPU memory requirements (e.g. ''gpu_ram=2G''). Thus e.g. <code>qsub -q gpu.q -l gpu=1,gpu_cc_min3.5=1,gpu_ram=2G</code> |
Known NVIDIA Academic Hardware Grants: | * If you need more than one GPU card (on a single machine), always require as many CPU cores (''-pe cmp X'') as many GPU cards you need. E.g. <code>qsub -q gpu.q -l gpu=4,gpu_cc_min3.5=1,gpu_ram=7G -pe smp 4</code> |
| * For interactive jobs, you can use ''qrsh'', but make sure to end your job as soon as you don't need the GPU (so don't use qrsh for long training). **Warning: ''-pty yes bash'' is necessary**, otherwise the variable ''$CUDA_VISIBLE_DEVICES'' will not be set correctly. E.g. <code>qrsh -q gpu.q -l gpu=1,gpu_ram=2G -pty yes bash</code>In general: don't reserve a GPU (as described above) without actually using it for longer time. Ondřej Bojar has a script /home/bojar/tools/servers/watch_gpus for watching reserved but unused GPU on most machines which will e-mail you, but don't rely on in only. |
* Ondřej Plátek - granted (2015) | |
* Jan Hajič jr. - granted (early 2016) | |
| |
| |
| |
| |
===== How to use cluster ===== | ===== How to use cluster ===== |
| |
In this section will be explained how to use cluster properly. | |
| |
==== Set-up CUDA and CUDNN ==== | ==== Set-up CUDA and CUDNN ==== |
This environment have TensorFlow 1.0 and all necessary requirements for NeuralMonkey. | This environment have TensorFlow 1.0 and all necessary requirements for NeuralMonkey. |
| |
==== Using cluster ==== | ==== Pytorch Environment ==== |
| |
Rule number one, always use the GPU queue (never log in machine by ssh). Always use qsub or qsubmit with proper arguments. | If you want to use pytorch, there is a ready-made environment in |
| |
For testing and using the cluster interactively you can use qrsh (this should not be used for long running experiments since the console is not closed on the end of the experiment). Following command will assign you a GPU and creates interactive console. | /home/hajicj/anaconda3/envs/pytorch/bin |
| |
qrsh -q gpu.q -l gpu=1,gpu_ram=2G -pty yes bash | |
| |
For running experiments you must use qsub command: | It does rely on the CUDA and CuDNN setup above. |
| |
qsub -q gpu.q -l gpu=1,gpu_cc_min3.5=1,gpu_ram=2G WHAT_SHOULD_BE_RUN | ==== Using cluster ==== |
| |
Cleaner way to use cluster is with /home/bojar/tools/shell/qsubmit | As an alternative to ''qsub'', you can use /home/bojar/tools/shell/qsubmit |
| |
qsubmit --gpumem=2G --queue="gpu.q" WHAT_SHOULD_BE_RUN | qsubmit --gpumem=2G --queue="gpu.q" WHAT_SHOULD_BE_RUN |
GPU specs for those GPUs we have: | GPU specs for those GPUs we have: |
* [[http://www.nvidia.com/content/PDF/kepler/Tesla-K40-Active-Board-Spec-BD-06949-001_v03.pdf|Tesla K40c]] | * [[http://www.nvidia.com/content/PDF/kepler/Tesla-K40-Active-Board-Spec-BD-06949-001_v03.pdf|Tesla K40c]] |
| |
| ==== Individual acquisitions: NVIDIA Academic Hardware Grants ==== |
| |
| There is an easy way to get one high-end GPU: [[https://developer.nvidia.com/academic_gpu_seeding|ask NVIDIA for an Academic Hardware Grant]]. All it takes is writing a short grant application (at most ~2 hrs of work from scratch; if you have a GAUK, ~15 minutes of copy-pasting). Due to the GPU housing issues (mainly rack space and cooling), Milan F. said we should request the Tesla-line cards (2017 check with Milan about this issue). If you want to have a look at an application, feel free to ask at hajicj@ufal.mff.cuni.cz :) |
| |
| Take care, however, to coordinate the grant applications a little, so that not too many arrive from UFAL within a short time: these grants are explicitly //not// intended to build GPU clusters, they are "seeding" grants meant for researchers to try out GPUs (and fall in love with them, and buy a cluster later). If you are planning to submit the hardware grant, have submitted one, or have already been awarded one, please add yourself here. |
| |
| Known NVIDIA Academic Hardware Grants: |
| |
| * Ondřej Plátek - granted (2015) |
| * Jan Hajič jr. - granted (early 2016) |
| |
| |
| |