Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
grid [2017/09/26 13:49] popel |
grid [2017/09/26 18:30] popel delete outdated/old tricks |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== ÚFAL Grid Engine (LRC) ====== | ====== ÚFAL Grid Engine (LRC) ====== | ||
- | LRC (Linguistic Research Cluster) is a name of ÚFAL' | + | LRC (Linguistic Research Cluster) is a name of ÚFAL' |
If you need GPU processing, see a special page about our [[:gpu|GPU cluster called DLL]] (which is actually a subsystem of LRC with independent queue '' | If you need GPU processing, see a special page about our [[:gpu|GPU cluster called DLL]] (which is actually a subsystem of LRC with independent queue '' | ||
Line 7: | Line 7: | ||
===== List of Machines ===== | ===== List of Machines ===== | ||
The list has been updated 2017/09. All machines have Ubuntu 14.04. | The list has been updated 2017/09. All machines have Ubuntu 14.04. | ||
+ | Some machines are at Malá Strana (ground floor, new server room built from Lindat budget), some are at Troja (5 km north-east). | ||
+ | If you need to quickly distinguish which machine is located where, you can use your knowledge of [[https:// | ||
- | ==== Trója | + | ==== Troja (troja-all.q) ==== |
+ | ^ Name ^ CPU type ^ GHz ^cores ^RAM(GB)^ note ^ | ||
+ | | achilles[1-8] | ||
+ | | hector[1-8] | ||
+ | | helena[1-8] | ||
+ | | paris[1-8] | ||
==== MS = Malá Strana (ms-all.q) ==== | ==== MS = Malá Strana (ms-all.q) ==== | ||
^ Name ^ CPU type ^ GHz ^cores ^RAM(GB)^ note ^ | ^ Name ^ CPU type ^ GHz ^cores ^RAM(GB)^ note ^ | ||
- | | andromeda[1-13] | + | | andromeda[1-13] |
- | | fireball[1-10] | + | | hydra[1-4] |
- | | hyperion[1-10] | 2xCore2 | + | | fireball[1-10] |
- | | lucifer[1-10] | + | | hyperion[1-9] |
- | | orion[1-10] | + | | lucifer[1-10] |
- | | tauri[1-10] | + | | orion[1-6] | Intel 2xCore4 Xeon | 2.3 | 8 | 32 | | |
- | | cosmos | + | | orion[7-10] |
- | | belzebub | + | | tauri[1-10] |
- | | iridium | + | | cosmos |
- | | twister[1, | + | | belzebub |
+ | | iridium | ||
+ | | twister[1, | ||
=== Outside LRC cluster (but located as MS) === | === Outside LRC cluster (but located as MS) === | ||
^ Name ^ CPU type ^ GHz ^cores ^ RAM(GB)^ note ^ | ^ Name ^ CPU type ^ GHz ^cores ^ RAM(GB)^ note ^ | ||
- | | lrc[1, | + | | lrc[1, |
| pandora[1-10] | | pandora[1-10] | ||
- | | sol[1-8,11-13] | 2xCore4 AMD Opteron | + | | sol[1-5] | Intel | 2.6 | 4 | 16 | you can ssh here and compute | |
+ | | sol[6-8] | Intel | 2.0 | | ||
+ | |||
+ | The two **lrc machines** are so called heads of the cluster. **No computation is allowed here**, i.e. no CPU-intensive, | ||
+ | |||
+ | Alternatively, | ||
+ | |||
+ | The **pandora machines** are in a special cluster (not accessible from lrc) and queue **ms-guests.q** available for our colleagues from KSVI and for students of [[http:// | ||
- | **pandora machines** are in a special cluster and queue **ms-guests.q** available for our colleagues from KSVI and for students of [[http:// | + | fronty výpočtů, které se odesílají na výpočetní stroje v clusteru. Na hlavě se nemají pouštět žádné náročné výpočty a naopak na ostatní stroje v clusteru se mají výpočty odesílat výhradně prostřednictvím hlavy. Hlava má 2 procesory Intel Pentium D 3 GHz a 1 GB paměti. Ve skutečnosti existují hlavy dvě - lrc1 a lrc2, které sdílí IP adresu lrc.ufal.hide.ms.mff.cuni.cz. V případě výpadku jedné z hlav, přebírá kontrolu ta druhá. |
- | **sol machines** are for us - we can ssh there, submit jobs and also do computing (e.g. collecting statistics from the submitted jobs). | + | |
- | **lrc machines** are so called heads of the cluster. **No computation is allowed here**. | + | |
- | .ufal.hide.ms.mff.cuni.cz: | + | |
Frontovací systém umožňuje: | Frontovací systém umožňuje: | ||
Line 134: | Line 147: | ||
lépe funguje '' | lépe funguje '' | ||
- | |||
- | ==== ~zeman/ | ||
- | |||
- | Podobná věc pro '' | ||
- | |||
- | <code tcsh> | ||
- | echo $* > $SCRIPTFILE | ||
- | echo $* | ||
- | echo qsub -cwd -V -S /bin/tcsh -m e $SCRIPTFILE | ||
- | qsub -cwd -V -S /bin/tcsh -m e $SCRIPTFILE | ||
- | qstat -u ' | ||
- | rm $SCRIPTFILE</ | ||
- | |||
- | Příklad spuštění: | ||
- | |||
- | < | ||
- | |||
- | (Kdybych místo uvozovek použil apostrofy, nerozbalily by se mi proměnné. První argument (název skriptu) klidně mohl být v uvozovkách spolu s přesměrováním. Dal jsem ho ven jen proto, že potom '' | ||
- | |||
- | |||
- | |||
- | |||
- | |||
- | ==== TectoMT: devel/ | ||
- | |||
- | Jako '' | ||
- | |||
- | | ||
- | |||
- | Skript zadanou hromádku souboru rozdělí do '' | ||
- | |||
- | Soubory možno zadat filelistem, nebo pomocí '' | ||
- | |||
- | Je nutné buď zadat '' | ||
- | |||
- | Parametr '' | ||
- | |||
- | Parametr '' | ||
- | |||
- | Výstup každého jobu jde do vlastního logu, '' | ||
- | |||
- | Parametr '' | ||
- | |||
- | Bez '' | ||
===== Monitorování úloh ===== | ===== Monitorování úloh ===== | ||
Line 279: | Line 248: | ||
</ | </ | ||
- | ===== Synchronizace úloh (v Perlu) ===== | ||
- | |||
- | Pokud chci paralelizovat část úlohy (zde '' | ||
- | |||
- | * Obalím svůj skript pro běh na gridu – vytvořím '' | ||
- | < | ||
- | #!/bin/bash | ||
- | . / | ||
- | |||
- | qrsh -cwd -V -p -50 -l mf=5g -now no ' | ||
- | </ | ||
- | * Ve svém hlavním skriptu ho pak zavolám a posbírám výsledky: | ||
- | < | ||
- | use FileHandle; | ||
- | use IPC::Open2; | ||
- | use threads; | ||
- | use threads:: | ||
- | my @threads; | ||
- | my @results; | ||
- | share(@results); | ||
- | for (@inputs) | ||
- | my $t = async { | ||
- | my $reader; my $writer; | ||
- | my $pid = open2($reader, | ||
- | die " | ||
- | $writer-> | ||
- | print $writer " | ||
- | $writer-> | ||
- | for (< | ||
- | chomp; | ||
- | { | ||
- | lock @results; | ||
- | push @results, $_; | ||
- | } | ||
- | } | ||
- | waitpid $pid, 0; # Pockame s ukoncenim vlakna na ukonceni ulohy v gridu | ||
- | return $? >> 8; # Pokusime se ziskat navratovou hodnotu (netestoval jsem) | ||
- | }; | ||
- | push @threads, $t; | ||
- | } | ||
- | for (@threads) | ||
- | die "Child exited with non-zero exit code" if $_-> | ||
- | } | ||
- | </ | ||
- | |||
- | Poznámky: | ||
- | * Pokud lze všechno předat parametry, nemusí se otevírat obousměrná roura a situace bude jednodušší | ||
- | * Pokud '' | ||
- | * Celý příklad je k vidění v Czengu od V.N. |