Software Setup#
Anaconda#
Run this command to gain access to conda on the HPC
source /mnt/software/unstowable/miniconda3-4.6.14/etc/profile.d/conda.sh
R and R Studio#
Create a new conda environment specifically for R.
conda create --prefix <Path to store conda environment>
Activate environment
conda activate <Path to store conda environment>
Install R and R Studio
conda install R
conda install rstudio
You will probably see this error, assuming you enabled export QT_DEBUG_PLUGINS=1
Got keys from plugin meta data ("xcb")
QFactoryLoader::QFactoryLoader() checking directory path "/home/teojyj/conda_env/jt_R/bin/platforms" ...
loaded library "/home/teojyj/conda_env/jt_R/plugins/platforms/libqxcb.so"
QLibraryPrivate::loadPlugin failed on "/home/teojyj/conda_env/jt_R/plugins/platforms/libqxcb.so" : "Cannot load library /home/teojyj/conda_env/jt_R/plugins/platforms/libqxcb.so: (libEGL.so.1: cannot open shared object file: No such file or directory)"
This application failed to start because it could not find or load the Qt platform plugin "xcb"
in "".
Available platform plugins are: eglfs, minimal, minimalegl, offscreen, xcb.
Reinstalling the application may fix this problem.
Aborted (core dumped)
This error is caused by problems in qt, specifically a missing path to libEGL.so.1. See
ldd /home/teojyj/conda_env/jt_R/plugins/platforms/libqxcb.so
To remedy it, download libEGL, and add library path containing libEGL.so.1 to LD_LIBRARY_PATH
conda install -c anaconda mesa-libegl-cos6-x86_64
export LD_LIBRARY_PATH=/home/teojyj/conda_env/jt_R/x86_64-conda_cos6-linux-gnu/sysroot/usr/lib64/:$LD_LIBRARY_PATH #to get rstudio to work
Jupyter Notebook#
To run jupyter notebook off an interactive node on the HPC, follow the instructions presented here.
To summarise,
On the server
* Start an interactive session. Take note of the name of the session. For example, n069.
* (Optional) Enable conda environment. Make sure the right packages are installed
qrsh -l h_rt=23:59:59 -l mem_free=8G -pe OpenMP 4 -q interactive.q
conda activate <env>
- Start jupyter notebook without browser. Set port to something other than 8888.
jupyter notebook --no-browser --ip='0.0.0.0' --port=8828
...
To access the notebook, open this file in a browser:
file:///home/teojyj/.local/share/jupyter/runtime/nbserver-46966-open.html
Or copy and paste one of these URLs:
http://n069:8828/?token=346sdhgdfgzdgw3563452341234567890
or http://127.0.0.1:8828/?token=346sdhgdfgzdgw3563452341234567890
Note that this token can be used to connect Atom's Hydrogen to a remote kernel on the cluster! Add the following to Hydrogen's remote kernel setting, and add the token shown above when prompted.
[{
"name": "Remote server",
"options": {
"baseUrl": "http://127.0.0.1:8828"
}
}]
On the client
* To access jupyter after opening port, run jupssh <node name>.
jptnbssh(){
# Check if the arugment is passed.
if [[ $# -eq 0 ]];
then
echo 'Usage: jptnbssh <node name>'
return
fi
#open -a "Google Chrome" http://127.0.0.1:8828/
ssh <username>@aquila.gis.a-star.edu.sg -N -L localhost:8828:$1:8828
}
Rendering code on jupyter (markdown cell)#
Using default setting, jupyter renders the background in white. Apply a change to ~/.jupyter/custom/custom.css to change the background. Check this example to change the code snippet's background to gray.
/* change markdown code snipet style */
.rendered_html pre code {
background-color: #f7f7f7;
font-family: monaco;
}
/* change markdown code snipet wrapper style */
.rendered_html pre {
margin: 1em 2em;
padding: 0px;
background-color: #f7f7f7;
}
Mkdocs#
I use markdown, mkdocs, combined with github and readthedocs to create online documentation that is organized beautifully and comes with a built-in search tool.
To enable mkdocs locally on 127.0.0.1:8000, activate the conda environment for mkdocs and run mkdocs serve.
conda activate markdown_env
mkdocs serve
If creating an mkdoc server on a remote server, run mkdocs with --dev-addr to create a port, and set up port forwarding locally. The steps are similar to that of creating a jupyter notebook remotely.
# On a remote server
mkdocs serve --dev-addr '0.0.0.0:8829'
# On the client
ssh <username>@<server> -N -L localhost:8829:<node>:8829
# Access site at http://127.0.0.1:8828
TMUX#
To install tmux, download the following files * tmux * Installation * libevent * ncurses
# Install libevent
tar -zxf libevent-*.tar.gz
cd libevent-*/
./configure --prefix=$HOME/libevent --enable-shared
make && make install
# Install ncurses
tar -zxf ncurses-*.tar.gz
cd ncurses-*/
./configure --prefix=$HOME/ncurses --with-shared --enable-pc-files --with-pkg-config-libdir=$HOME/ncurses/lib/pkgconfig
make && make install
There will be errors for ncurses, but it should still work.
# Install TMUX
tar -zxf tmux-*.tar.gz
cd tmux-*/
#PKG_CONFIG_PATH=$HOME/local/lib/pkgconfig ./configure --prefix=$HOME/local
PKG_CONFIG_PATH=/opt/libevent/lib/pkgconfig:/opt/ncurses/lib/pkgconfig ./configure --prefix=$HOME/tmux
make && make install
# Add libevent library to path
# Add to ~/.bashrc
export LD_LIBRARY_PATH=$HOME/libevent/lib:$LD_LIBRARY_PATH
libtool output
Libraries have been installed in:
/home/teojyj/code/libevent/lib
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the `LD_RUN_PATH' environment variable
during linking
- use the `-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to `/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.