====== IT tricks ======
Various tips how to increase your productivity (esp. in Linux).

===== Editors =====
Who uses which editor and is willing to provide a mini-training for new users (show config, favorite macros, highlighters etc.).
Feel free to add your name and editor.

  * **vim**: Ondřej Bojar, Ruda Rosa, Milan Straka (C++ completion, Python completion, asynchronous make), ...
    * to start learning vim, run the ''vimtutor'' command
  * **emacs**: ?
  * **nano**:
  * **IntelliJ IDEA**: Jonáš Vidra
  * **Kate**: Jonáš Vidra
  * **PyCharm**:
  * **VS Code** (remote access, IDE - Python, C++, etc. completion, terminals): Peter Polák

===== Bash =====
  * Use Bash auto-completion (''source /etc/bash_completion''). Some tools have plugins, e.g. many [[https://metacpan.org/pod/Bash::Completion|Perl tools]].
  * ''export HISTCONTROL=ignoreboth'' results in bash history ignoring duplicate entries and entries starting with a space.

==== Directory-local Bash history ====
  * Ondřej Bojar prefers to store the history in each directory in ''.history-bojar''. See ''/home/bojar/diplomka/granty/cracker-2015/mtm-2016-organization/accounts-mtm16/bash-profile-for-ufal-accounts''
    * Note that this creates the ".history-your-name" wherever you have write access and some ÚFALers are not happy with this. In addition everyone sees what you did (this may be useful, but be careful).
  * Ondřej Dušek (''/home/odusek/.bashrc'') uses a variant more friendly to others, which saves the histories in his home (but does not handle renamed directories):<code>
if [ -z "$USER" ]; then
    export USER=`whoami`
fi

# Store all history with times and directories
function store_history () {  
    history 1 | awk '($2 !~ "^[mr]?cd[0-9a-z]?$") {$1="_T="strftime("%Y%m%d_%H:%M:%S_") PROCINFO["ppid"] "_PWD="  ENVIRON["PWD"] "\t"; $2=gensub("^_T=[-_0-9:]*[ \t]* *", "", 1, $2); $2=gensub("^_P=[^ \t]* *", "", 1, $2); print;}' >> ~/.history-all-$USER
}
export PROMPT_COMMAND="store_history"

# Grep history
function dhist (){
    DIR=`pwd`
    command grep "_PWD=$DIR"$'\t'".*$@" ~/.history-all-$USER | tail -n 30
}

function hist (){    
    if [ "$#" -eq 0 ]; then
        tail -n 30 ~/.history-all-$USER
    else
        command grep "$@" ~/.history-all-$USER | tail -n 30
    fi
}
</code>

==== Colorful manpages ====
Add this to your .bashrc (tested on .zshrc). Colors can be easily customized.

<code>
man () {
        env LESS_TERMCAP_mb=$'\E[01;31m' \
        LESS_TERMCAP_md=$'\E[01;38;5;74m' \
        LESS_TERMCAP_me=$'\E[0m' \
        LESS_TERMCAP_se=$'\E[0m' \
        LESS_TERMCAP_so=$'\E[38;5;246m' \
        LESS_TERMCAP_ue=$'\E[0m' \
        LESS_TERMCAP_us=$'\E[04;38;5;146m' \
        man "$@"
}
</code>


===== Git =====
  * [[https://github.com/magicmonty/bash-git-prompt|bash-git-prompt]] shows e.g. the current branch and status. Martin Popel prefers to configure it (in bashrc) with

    export GIT_PS1_SHOWDIRTYSTATE=1
    export GIT_PS1_SHOWSTASHSTATE=1
    export PS1='[\t]\[\033[2;31m\]\h:\W\[\033[01;95m\]$(__git_ps1 "(%s)")\[\033[2;31m\]>\[\033[0m\] '

  * [[https://github.com/jonas/tig|tig]] text-mode gitk but better
  * read https://guides.github.com/, understand [[https://guides.github.com/introduction/flow/|GitHub flow]], check [[https://services.github.com/on-demand/resources/learning-path/|GitHub training]]
  * learn [[https://help.github.com/articles/using-keyboard-shortcuts/|GitHub keyboard shortcuts]], esp. "y" for permalink to a given line (line range) in source codes


===== Plots, vector graphic =====
  * [[http://www.gnuplot.info/|gnuplot]]
  * [[https://seaborn.pydata.org/|seaborn]] based on matplotlib (Python)
  * [[https://inkscape.org/en/|inkscape]] manually draw vector graphic (or poster)
  * [[https://en.wikibooks.org/wiki/LaTeX/PGF/TikZ|TikZ]] great for LaTeX/Beamer, offers many "plugins"
    * [[http://pgfplots.sourceforge.net/|pgfplots]]
    * [[http://mirrors.ctan.org/graphics/pgf/contrib/tikz-dependency/tikz-dependency-doc.pdf|tikz-dependency]] dependency trees (in one line)
    * [[http://mirrors.ctan.org/graphics/pgf/contrib/tikz-qtree/tikz-qtree-manual.pdf|tikz-qtree]] classical trees (constituency, but with some hacks even dependency)

===== LaTeX, pdf =====
  * see also [[tex::triky]], [[internal:publikace#collaborative-writing]] and [[internal:thesis]]
  * [[http://mg.readthedocs.io/latexmk.html|latexmk]] can be used with ''-pdf -pvc'', so whenever you save a ''tex'' file, the corresponding ''pdf'' will be regenerated (and your pdf viewer will refresh). Other useful options are ''-interaction=nonstopmode -synctex=1'', see e.g. [[https://tex.stackexchange.com/questions/118489/what-exactly-is-synctex|synctex]]
  * [[https://www.pdflabs.com/docs/pdftk-cli-examples/|pdftk]] for merging and splitting pdf files (and much more) from the command line
  * use [[https://tex.stackexchange.com/questions/23104/using-and-interpreting-pdffonts|pdffonts]] to check if all fonts in your pdf are //embedded// (only the used subset).
  * <code>gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=CompressedWithEmbeddedFonts.pdf Original.pdf</code>
  * http://ufal.mff.cuni.cz/~popel/latex.pdf other hints (mostly for newbies) in Czech
  * [[https://en.wikibooks.org/wiki/LaTeX/Presentations#The_Beamer_package|Beamer]] is a LaTeX package for creating presentations.
    * You can use it with [[https://pdfpc.github.io/|pdfpc]] to show slides on the overhead projector while reading your notes on your laptop's screen. Create the dual-screen PDF with ''\usepackage{pgfpages} \setbeameroption{show notes} \setbeameroption{show notes on second screen=left}'' and add notes using ''\note{Hello!}''. Then display your presentation with ''pdfpc --notes=left presentation.pdf''.
  * If you need a pdf-a validator (e.g. for your thesis), you can use  `/lnet/ms/projects/verapdf/checkpdf` in the ufal internal network, which is a wrapper for the verapdf tool. New versions of the specification of exact requirements for submitting theses at the Charles university (called a "profile") appear regularly at https://cuni.cz/UK-7987.html; by default, the tool uses version UK-7987-version1-custom8.xml .

==== Faster LaTeX compilation ====
   * You should delete the packages you don't need from the preambule (not only for speed reasons).
   * Include just the chapter (or section) you are currently working on, i.e. comment out the inclusion of other chapters or use ''\includeonly''.
   * When separating e.g. each chapter into a separate file and each chapter starts on a new page, you can use ''\include{chapter.tex}'' instead of ''\input{chapter.tex}'', so the auxiliary files of the unchanged chapters from the previous compilation can speedup the compilation (see [[https://tex.stackexchange.com/questions/246/when-should-i-use-input-vs-include|details]]).
   * You can use ''\includegraphics[draft]{...}'', which turns off rendering of images (which is usually a too high price for the speed).
   * If you are using complex tikz pictures you can convert them first to pdf manually and include the pdfs, so they are not compiled again and again (or you can try the externalization library to do this automatically).
   * And when compiling locally, you can use latexmk -pvc to automatically compile a given file whenever it is saved. This does not really speedup the compilation, but it allows you to have a screen split into the editor and pdf viewer (so you can simulate Overleaf GUI, possibly including the forward/inverse search using [[https://tex.stackexchange.com/questions/118489/what-exactly-is-synctex|synctex]]), so that you can continue working, while the compilation runs on background.
===== TAR archive mounting =====
If you work with data consisting of many small files, you **should** store them in tar archives to save inodes and use LUSTRE filesystem efficiently. There is a way to mount a tar archive to a specific directory in a similar way you would mount an ISO image. You can use this command to //mount// (read-only) ARCHIVE (possibly compressed) to DIRECTORY:

  /opt/bin/tarmount ARCHIVE DIRECTORY

and //umount// when you finish your work:

  /opt/bin/tarmount -u DIRECTORY

If you mount your data in this way, there is only one //open// operation instead of thousands or millions of such operations when opening each file separately. This will positively affect runtime when your data is stored on the LUSTRE filesystem.

Run the command with //--help// option to see instructions for more advanced usage. The original tool can be obtained [[https://github.com/mxmlnkn/ratarmount | here]].
It can be installed as [[https://appimage.org | AppImage]].

==== Practical usage notes ====
  - ''tarmount'' is based on FUSE. You need to do call the ''tarmount'' command on every node where you need to access the tarfile contents. If two jobs are running on the same node both of them will see the mounted content.
  - if you finish your work you should always call ''tarmount -u DIRECTORY''. If you don't the directory will not become blocked for any subsequent attempts to ''tarmount'' anything on them and may be in a weird state indicated by many '?'s in output of the ''ls -al'' command. This can be solved by calling simple ''umount DIRECTORY''
  - The ''tarmount'' command creates a database in the directory where the ''ARCHIVE'' is stored. The database is not automatically deleted by ''tarmount -u''. It is however reused by any subsequent ''tarmount'' command called on the same ''ARCHIVE''. This may speed up access for all jobs using the same ''tarmount''ed archive across the cluster.
 




===== Other =====
  * When using **Perl** at ÚFAL, we recommend using [[Perlbrew]] with shared perl interpreters (different versions) and shared Perl modules (which otherwise take several hours to install and 6GB).
  * See [[Python]] for using **Python** at ÚFAL.
  * See [[internal:remote access]], esp. for byobu and mosh.
  * You **should not turn off** Linux workstations at ÚFAL, unless really needed (or agreed with it@ufal). In that case, try the standard ways, and if they do not work, try Ctrl+Alt+SysRq+[R,E,I,S,U,B] rather than plug out power cable. Mnemonics: "Reboot Even If System Utterly Broken", see http://en.wikipedia.org/wiki/Magic_SysRq_key#Uses
    * SysRq has also other interesting uses: For example, if you manage to launch a program that exhausts all RAM, the system starts swapping and becomes unresponsive, you can recover by invoking the OOM-killer by pressing Ctrl+Alt+SysRq+F.
  * Ondřej Bojar has implemented many [[http://www1.cuni.cz/~obo/textutils/|Text utils]], which he likes to use in his scripts and advises his students to do so as well.
    * The web versions of the text utils are often older than the ones in ''/home/bojar/tools/vimtext'' and ''/home/bojar/tools/shell''
  * [[https://en.wikipedia.org/wiki/Shuf|shuf]] shuffle lines randomly (or in a given order with ''--random-source'')
  * Ondřej Plátek keeps some of the suggestions above in configs, and scripts stored in the git repository which you can just clone to your home dir. Feel free to check https://github.com/ufal/oplatek-clustershgit. Pull Requests are welcome!