Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
gpu [2017/10/17 16:39] popel [Using cluster] |
gpu [2018/04/12 15:17] kocmanek [Performance tests] |
||
---|---|---|---|
Line 5: | Line 5: | ||
===== Servers with GPU units ===== | ===== Servers with GPU units ===== | ||
GPU cluster '' | GPU cluster '' | ||
- | + | | machine | GPU type | GPU driver version | |
- | | machine | + | | dll1 | |
- | | iridium | + | | dll2 | GeForce GTX 1080 | |
- | | titan-gpu | + | | dll3 | GeForce GTX 1080 Ti | 375.66 | |
- | | twister1; twister2; kronos | + | | dll4 | |
- | | dll1; dll2 | GeForce GTX 1080 | | + | | dll5 | GeForce GTX 1080 Ti | |
- | | titan | + | | dll6 | GeForce GTX 1080 Ti | |
- | | dll3; dll4; dll5 | GeForce GTX 1080 Ti | | + | | gpu | GeForce GTX TITAN Z | 381.22 | 3.5 | 2 | 6 | 31 | |
- | | dll6 | GeForce GTX 1080 Ti | | + | | iridium | Quadro K2000 | 367.48 | 3.0 | |
+ | | kronos | ||
+ | | titan | GeForce GTX 1080 | | ||
+ | | twister1 | Tesla K40c | 367.48 | | ||
+ | | twister2 | Quadro P5000 | 367.48 | 6.1 | 1 | 17 | 47 | | ||
Desktop machines: | Desktop machines: | ||
Line 22: | Line 26: | ||
Not used at the moment: GeForce GTX 570 (from twister2) | Not used at the moment: GeForce GTX 570 (from twister2) | ||
All machines have CUDA8.0 and should support both Theano and TensorFlow. | All machines have CUDA8.0 and should support both Theano and TensorFlow. | ||
+ | |||
+ | [[https:// | ||
+ | |||
===== Rules ===== | ===== Rules ===== | ||
* First, read [[internal: | * First, read [[internal: | ||
* All the rules from [[:Grid]] apply, even more strictly than for CPU because there are too many GPU users and not as many GPUs available. So as a reminder: always use GPUs via '' | * All the rules from [[:Grid]] apply, even more strictly than for CPU because there are too many GPU users and not as many GPUs available. So as a reminder: always use GPUs via '' | ||
- | * Always specify the number of GPU cards (e.g. '' | + | * Always specify the number of GPU cards (e.g. '' |
- | * If you need more than one GPU card, always require as many CPU cores as many GPU cards you need. E.g. < | + | * If you need more than one GPU card (on a single machine), always require as many CPU cores ('' |
- | * For interactive jobs, you can use '' | + | * For interactive jobs, you can use '' |
+ | * Note that the dll machines have typically 10 cards, but " | ||
===== How to use cluster ===== | ===== How to use cluster ===== | ||
Line 34: | Line 42: | ||
==== Set-up CUDA and CUDNN ==== | ==== Set-up CUDA and CUDNN ==== | ||
- | You can add following | + | You should |
CUDNN_version=6.0 | CUDNN_version=6.0 | ||
Line 47: | Line 55: | ||
export CPATH=$CUDA_DIR/ | export CPATH=$CUDA_DIR/ | ||
fi | fi | ||
+ | |||
+ | When not using Theano, just Tensorflow this can be simplified to '' | ||
+ | |||
+ | TensorFlow 1.5 precompiled binaries need CUDA 9.0, for this you need to | ||
+ | |||
+ | export LD_LIBRARY_PATH=/ | ||
+ | |||
+ | You also need to use '' | ||
+ | |||
+ | **Testing configuration (so far on twister2 only)** | ||
+ | |||
+ | Multiple versions of '' | ||
+ | System default version for both libraries is configured in ''/ | ||
+ | |||
+ | / | ||
+ | / | ||
+ | / | ||
+ | |||
+ | Actual version used depends on the link in ''/ | ||
+ | |||
+ | ls -l /opt | ||
+ | ... | ||
+ | lrwxrwxrwx 1 root root 8 dub 9 12:30 cuda -> cuda-9.0 | ||
+ | lrwxrwxrwx 1 root root 9 dub 9 12:32 cudnn -> cudnn-7.1 | ||
+ | ... | ||
+ | | ||
+ | This means that the system is using '' | ||
+ | |||
+ | If system default version does not work for you, you can set library path from your '' | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
==== TensorFlow Environment ==== | ==== TensorFlow Environment ==== | ||
Line 80: | Line 125: | ||
qsubmit --gpumem=2G --queue=" | qsubmit --gpumem=2G --queue=" | ||
| | ||
- | It is recommended to use priority -100 if you are not rushing for the results and don't need to leap over your colleagues jobs. | + | It is recommended to use priority |
==== Basic commands ==== | ==== Basic commands ==== | ||
Line 137: | Line 182: | ||
| GPU; Cuda capability | | GPU; Cuda capability | ||
- | | Tesla K40c; cc3.5 | 12 GB | | ||
| GeForce GTX 1080 Ti; cc6.1 | 11 GB | 00:55:56 | 2300 | dll5 | | | GeForce GTX 1080 Ti; cc6.1 | 11 GB | 00:55:56 | 2300 | dll5 | | ||
| GeForce GTX 1080; cc6.1 | 8 GB | 01:10:57 | 1700 | dll1 | | | GeForce GTX 1080; cc6.1 | 8 GB | 01:10:57 | 1700 | dll1 | |