IT tricks

Various tips how to increase your productivity (esp. in Linux).

Who uses which editor and is willing to provide a mini-training for new users (show config, favorite macros, highlighters etc.).
Feel free to add your name and editor.

• vim: Ondřej Bojar, Ruda Rosa, Milan Straka (C++ completion, Python completion, asynchronous make), …
  • to start learning vim, run the vimtutor command
• emacs: Jan Štěpánek
• nano:
• IntelliJ IDEA: Jonáš Vidra
• Kate: Jonáš Vidra
• PyCharm:
• VS Code (remote access, IDE - Python, C++, etc. completion, terminals): Peter Polák

• Use Bash auto-completion (source /etc/bash_completion). Some tools have plugins, e.g. many Perl tools.
• export HISTCONTROL=ignoreboth results in bash history ignoring duplicate entries and entries starting with a space.

• Ondřej Bojar prefers to store the history in each directory in .history-bojar. See /home/bojar/diplomka/granty/cracker-2015/mtm-2016-organization/accounts-mtm16/bash-profile-for-ufal-accounts
  • Note that this creates the “.history-your-name” wherever you have write access and some ÚFALers are not happy with this. In addition everyone sees what you did (this may be useful, but be careful).
• Ondřej Dušek (/home/odusek/.bashrc) uses a variant more friendly to others, which saves the histories in his home (but does not handle renamed directories):

  if [ -z "$USER" ]; then
      export USER=`whoami`
  fi
  
  # Store all history with times and directories
  function store_history () {  
      history 1 | awk '($2 !~ "^[mr]?cd[0-9a-z]?$") {$1="_T="strftime("%Y%m%d_%H:%M:%S_") PROCINFO["ppid"] "_PWD="  ENVIRON["PWD"] "\t"; $2=gensub("^_T=[-_0-9:]*[ \t]* *", "", 1, $2); $2=gensub("^_P=[^ \t]* *", "", 1, $2); print;}' >> ~/.history-all-$USER
  }
  export PROMPT_COMMAND="store_history"
  
  # Grep history
  function dhist (){
      DIR=`pwd`
      command grep "_PWD=$DIR"$'\t'".*$@" ~/.history-all-$USER | tail -n 30
  }
  
  function hist (){    
      if [ "$#" -eq 0 ]; then
          tail -n 30 ~/.history-all-$USER
      else
          command grep "$@" ~/.history-all-$USER | tail -n 30
      fi
  }

Add this to your .bashrc (tested on .zshrc). Colors can be easily customized.

man () {
        env LESS_TERMCAP_mb=$'\E[01;31m' \
        LESS_TERMCAP_md=$'\E[01;38;5;74m' \
        LESS_TERMCAP_me=$'\E[0m' \
        LESS_TERMCAP_se=$'\E[0m' \
        LESS_TERMCAP_so=$'\E[38;5;246m' \
        LESS_TERMCAP_ue=$'\E[0m' \
        LESS_TERMCAP_us=$'\E[04;38;5;146m' \
        man "$@"
}

• bash-git-prompt shows e.g. the current branch and status. Martin Popel prefers to configure it (in bashrc) with

  export GIT_PS1_SHOWDIRTYSTATE=1
  export GIT_PS1_SHOWSTASHSTATE=1
  export PS1='[\t]\[\033[2;31m\]\h:\W\[\033[01;95m\]$(__git_ps1 "(%s)")\[\033[2;31m\]>\[\033[0m\] '

• tig text-mode gitk but better
• read https://guides.github.com/, understand GitHub flow, check GitHub training
• learn GitHub keyboard shortcuts, esp. “y” for permalink to a given line (line range) in source codes

• gnuplot
• seaborn based on matplotlib (Python)
• inkscape manually draw vector graphic (or poster)
• TikZ great for LaTeX/Beamer, offers many “plugins”
  • pgfplots
  • tikz-dependency dependency trees (in one line)
  • tikz-qtree classical trees (constituency, but with some hacks even dependency)

• see also triky, collaborative-writing and thesis
• latexmk can be used with -pdf -pvc, so whenever you save a tex file, the corresponding pdf will be regenerated (and your pdf viewer will refresh). Other useful options are -interaction=nonstopmode -synctex=1, see e.g. synctex
• pdftk for merging and splitting pdf files (and much more) from the command line
• use pdffonts to check if all fonts in your pdf are embedded (only the used subset).

• gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=CompressedWithEmbeddedFonts.pdf Original.pdf

• http://ufal.mff.cuni.cz/~popel/latex.pdf other hints (mostly for newbies) in Czech
• Beamer is a LaTeX package for creating presentations.
  • You can use it with pdfpc to show slides on the overhead projector while reading your notes on your laptop's screen. Create the dual-screen PDF with \usepackage{pgfpages} \setbeameroption{show notes} \setbeameroption{show notes on second screen=left} and add notes using \note{Hello!}. Then display your presentation with pdfpc –notes=left presentation.pdf.
• If you need a pdf-a validator (e.g. for your thesis), you can use `/lnet/ms/projects/verapdf/checkpdf` in the ufal internal network, which is a wrapper for the verapdf tool. New versions of the specification of exact requirements for submitting theses at the Charles university (called a “profile”) appear regularly at https://cuni.cz/UK-7987.html; by default, the tool uses version UK-7987-version1-custom8.xml .

• You should delete the packages you don't need from the preambule (not only for speed reasons).
• Include just the chapter (or section) you are currently working on, i.e. comment out the inclusion of other chapters or use \includeonly.
• When separating e.g. each chapter into a separate file and each chapter starts on a new page, you can use \include{chapter.tex} instead of \input{chapter.tex}, so the auxiliary files of the unchanged chapters from the previous compilation can speedup the compilation (see details).
• You can use \includegraphics[draft]{…}, which turns off rendering of images (which is usually a too high price for the speed).
• If you are using complex tikz pictures you can convert them first to pdf manually and include the pdfs, so they are not compiled again and again (or you can try the externalization library to do this automatically).
• And when compiling locally, you can use latexmk -pvc to automatically compile a given file whenever it is saved. This does not really speedup the compilation, but it allows you to have a screen split into the editor and pdf viewer (so you can simulate Overleaf GUI, possibly including the forward/inverse search using synctex), so that you can continue working, while the compilation runs on background.

If you work with data consisting of many small files, you should store them in tar archives to save inodes and use LUSTRE filesystem efficiently. There is a way to mount a tar archive to a specific directory in a similar way you would mount an ISO image. You can use this command to mount (read-only) ARCHIVE (possibly compressed) to DIRECTORY:

/opt/bin/tarmount ARCHIVE DIRECTORY

and umount when you finish your work:

/opt/bin/tarmount -u DIRECTORY

If you mount your data in this way, there is only one open operation instead of thousands or millions of such operations when opening each file separately. This will positively affect runtime when your data is stored on the LUSTRE filesystem.

Run the command with –help option to see instructions for more advanced usage. The original tool can be obtained here.
It can be installed as AppImage.

1. tarmount is based on FUSE. You need to do call the tarmount command on every node where you need to access the tarfile contents. If two jobs are running on the same node both of them will see the mounted content.
2. if you finish your work you should always call tarmount -u DIRECTORY. If you don't the directory will not become blocked for any subsequent attempts to tarmount anything on them and may be in a weird state indicated by many '?'s in output of the ls -al command. This can be solved by calling simple umount DIRECTORY
3. The tarmount command creates a database in the directory where the ARCHIVE is stored. The database is not automatically deleted by tarmount -u. It is however reused by any subsequent tarmount command called on the same ARCHIVE. This may speed up access for all jobs using the same tarmounted archive across the cluster.

• When using Perl at ÚFAL, we recommend using Perlbrew with shared perl interpreters (different versions) and shared Perl modules (which otherwise take several hours to install and 6GB).
• See Python for using Python at ÚFAL.
• See remote access, esp. for byobu and mosh.
• You should not turn off Linux workstations at ÚFAL, unless really needed (or agreed with it@ufal). In that case, try the standard ways, and if they do not work, try Ctrl+Alt+SysRq+[R,E,I,S,U,B] rather than plug out power cable. Mnemonics: “Reboot Even If System Utterly Broken”, see http://en.wikipedia.org/wiki/Magic_SysRq_key#Uses
  • SysRq has also other interesting uses: For example, if you manage to launch a program that exhausts all RAM, the system starts swapping and becomes unresponsive, you can recover by invoking the OOM-killer by pressing Ctrl+Alt+SysRq+F.
• Ondřej Bojar has implemented many Text utils, which he likes to use in his scripts and advises his students to do so as well.
  • The web versions of the text utils are often older than the ones in /home/bojar/tools/vimtext and /home/bojar/tools/shell
• shuf shuffle lines randomly (or in a given order with –random-source)
• Ondřej Plátek keeps some of the suggestions above in configs, and scripts stored in the git repository which you can just clone to your home dir. Feel free to check https://github.com/ufal/oplatek-clustershgit. Pull Requests are welcome!