Software Setup#

Anaconda#

Run this command to gain access to conda on the HPC

source /mnt/software/unstowable/miniconda3-4.6.14/etc/profile.d/conda.sh

R and R Studio#

Create a new conda environment specifically for R.

conda create --prefix <Path to store conda environment>

Activate environment

conda activate <Path to store conda environment>

Install R and R Studio

conda install R
conda install rstudio

You will probably see this error, assuming you enabled export QT_DEBUG_PLUGINS=1

Got keys from plugin meta data ("xcb")
QFactoryLoader::QFactoryLoader() checking directory path "/home/teojyj/conda_env/jt_R/bin/platforms" ...
loaded library "/home/teojyj/conda_env/jt_R/plugins/platforms/libqxcb.so"
QLibraryPrivate::loadPlugin failed on "/home/teojyj/conda_env/jt_R/plugins/platforms/libqxcb.so" : "Cannot load library /home/teojyj/conda_env/jt_R/plugins/platforms/libqxcb.so: (libEGL.so.1: cannot open shared object file: No such file or directory)"
This application failed to start because it could not find or load the Qt platform plugin "xcb"
in "".

Available platform plugins are: eglfs, minimal, minimalegl, offscreen, xcb.

Reinstalling the application may fix this problem.
Aborted (core dumped)

This error is caused by problems in qt, specifically a missing path to libEGL.so.1. See

ldd /home/teojyj/conda_env/jt_R/plugins/platforms/libqxcb.so

To remedy it, download libEGL, and add library path containing libEGL.so.1 to LD_LIBRARY_PATH

conda install -c anaconda mesa-libegl-cos6-x86_64
export LD_LIBRARY_PATH=/home/teojyj/conda_env/jt_R/x86_64-conda_cos6-linux-gnu/sysroot/usr/lib64/:$LD_LIBRARY_PATH #to get rstudio to work

Jupyter Notebook#

To run jupyter notebook off an interactive node on the HPC, follow the instructions presented here.

To summarise,

On the server * Start an interactive session. Take note of the name of the session. For example, n069. * (Optional) Enable conda environment. Make sure the right packages are installed

qrsh -l h_rt=23:59:59 -l mem_free=8G -pe OpenMP 4 -q interactive.q
conda activate <env>
  • Start jupyter notebook without browser. Set port to something other than 8888.
jupyter notebook --no-browser --ip='0.0.0.0' --port=8828
...
To access the notebook, open this file in a browser:
        file:///home/teojyj/.local/share/jupyter/runtime/nbserver-46966-open.html
    Or copy and paste one of these URLs:
        http://n069:8828/?token=346sdhgdfgzdgw3563452341234567890
     or http://127.0.0.1:8828/?token=346sdhgdfgzdgw3563452341234567890

Note that this token can be used to connect Atom's Hydrogen to a remote kernel on the cluster! Add the following to Hydrogen's remote kernel setting, and add the token shown above when prompted.

[{
  "name": "Remote server",
  "options": {
    "baseUrl": "http://127.0.0.1:8828"
  }
}]

On the client * To access jupyter after opening port, run jupssh <node name>.

jptnbssh(){
        # Check if the arugment is passed.
        if [[ $# -eq 0 ]];
        then
                echo 'Usage: jptnbssh <node name>'
                return
        fi
        #open -a "Google Chrome" http://127.0.0.1:8828/
        ssh <username>@aquila.gis.a-star.edu.sg -N -L localhost:8828:$1:8828
}

Rendering code on jupyter (markdown cell)#

Using default setting, jupyter renders the background in white. Apply a change to ~/.jupyter/custom/custom.css to change the background. Check this example to change the code snippet's background to gray.

/* change markdown code snipet style */
.rendered_html pre code {
  background-color: #f7f7f7;
  font-family: monaco;
}
/* change markdown code snipet wrapper style */
.rendered_html pre {
  margin: 1em 2em;
  padding: 0px;
  background-color: #f7f7f7;
}

Mkdocs#

I use markdown, mkdocs, combined with github and readthedocs to create online documentation that is organized beautifully and comes with a built-in search tool.

To enable mkdocs locally on 127.0.0.1:8000, activate the conda environment for mkdocs and run mkdocs serve.

conda activate markdown_env
mkdocs serve

If creating an mkdoc server on a remote server, run mkdocs with --dev-addr to create a port, and set up port forwarding locally. The steps are similar to that of creating a jupyter notebook remotely.

# On a remote server
mkdocs serve --dev-addr '0.0.0.0:8829'
# On the client
ssh <username>@<server> -N -L localhost:8829:<node>:8829
# Access site at http://127.0.0.1:8828

TMUX#

To install tmux, download the following files * tmux * Installation * libevent * ncurses

# Install libevent
tar -zxf libevent-*.tar.gz
cd libevent-*/
./configure --prefix=$HOME/libevent --enable-shared
make && make install

# Install ncurses
tar -zxf ncurses-*.tar.gz
cd ncurses-*/
./configure --prefix=$HOME/ncurses --with-shared --enable-pc-files --with-pkg-config-libdir=$HOME/ncurses/lib/pkgconfig
make && make install

There will be errors for ncurses, but it should still work.

# Install TMUX
tar -zxf tmux-*.tar.gz
cd tmux-*/
#PKG_CONFIG_PATH=$HOME/local/lib/pkgconfig ./configure --prefix=$HOME/local
PKG_CONFIG_PATH=/opt/libevent/lib/pkgconfig:/opt/ncurses/lib/pkgconfig ./configure --prefix=$HOME/tmux
make && make install

# Add libevent library to path
# Add to ~/.bashrc
export LD_LIBRARY_PATH=$HOME/libevent/lib:$LD_LIBRARY_PATH

libtool output

Libraries have been installed in:
   /home/teojyj/code/libevent/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.