How to set up jupyter on personal server#

gpu-jupyter#

From the Quickstart section of this Guide and Guide

"First of all, it is necessary to generate the Dockerfile based on the latest toolstack of hub.docker.com/u/jupyter. As soon as you have access to your GPU locally (it can be tested via a Tensorflow or PyTorch directly on the host node), you can run these commands to start the jupyter notebook via docker-compose (internally):"

./generate_Dockerfile.sh
docker build -t gpu-jupyter .build/
docker run -d -p [port]:8888 gpu-jupyter

"Alternatively, you can configure the environment in docker-compose.yml and run this to deploy the GPU-Jupyter via docker-compose (under-the-hood):"

./generate_Dockerfile.sh
./start-local.sh -p 8888  # where -p stands for the port of the service

Customization#

There are several customizations that I made for my specific use case. After pulling files from github and running ./generate_Dockerfile.sh, we have the following directory structure (I have listed the useful configuration files)

.
├── add-to-swarm.sh
├── .build
│   ├── Dockerfile
│   ├── jupyter_notebook_config.json
│   ├── jupyter_notebook_config.py
│   └── ...
├── data
│   ├── environment
│   ├── Getting_Started
│   └── ...
├── docker-compose-swarm.yml
├── docker-compose.yml
├── generate_Dockerfile.sh
├── src
│   ├── jupyter_notebook_config.json
│   └── ...
└── ...

Config files#

The generated .build/Dockerfile copies jupyter_notebook_config.json and jupyter_notebook_config.py into /etc/jupyter in the container. Instead of building an image everytime we make a change to jupyter_notebook_config.py, instead we mount /etc/jupyter to a new config folder which contains the .json and .py config files.

To mount /etc/jupyter in the image, we add to the Dockerfile

# Mount config folder
VOLUME ["/etc/jupyter"]

and bind mount it to ./config in docker-compose.yml.

volumes:
    - ./config:/etc/jupyter

Note that the pre-generated docker-compose.yml file is primarily a build file instead of a run file. To configure a run file, I created a new docker-compose.yml and backed up the previous one to docker-compose-build.yml.

Installing additional pip requirements#

To run the docker container with a new cookiecutter templated project, we may need to install new packages. I have overwritten the CMD option of the Dockerfile in docker-compose.yml to look for requirements.txt in the mounted data/environment folder and install it.

volumes:
      - ./data:/home/jovyan/work
environment:
      REQFILE: "/home/jovyan/work/environment/requirements.txt"
command: /bin/bash -c "[[ -f "$$REQFILE" ]] && pip install -r $$REQFILE; start-notebook.sh"

Add Traefik and network support#

Refer to Traefik's guide for more information.

services:
  gpu-jupyter:
  ...
    networks:
      - traefik-web
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.gpu-jupyter.rule=Host(`jupyter.jonai.teojy.com`)"
      - "traefik.http.routers.gpu-jupyter.entrypoints=web"

networks:
  traefik-web:
    external: true

Switch from password to token#

I want token based access so that atom/hydrogen can connect to the kernel. To do so, I added and changed the following options in ./config/jupyter_notebook_config.py

c = get_config()
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.port = 8888
c.NotebookApp.open_browser = False
c.NotebookApp.token = '[SecretToken]'
c.NotebookApp.allow_password_change = False
c.NotebookApp.allow_origin = '*'

I also removed the password in ./config/jupyter_notebook_config.json

{
  "NotebookApp": {}
}

If successful, you will be prompted for a token instead of a password.

Install Nvidia GPU Docker#

GPU is not enabled by default. Run the following code to enable nvidia in Docker.

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
echo $distribution   # this shows your version
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

sudo systemctl restart docker
#sudo systemctl status docker

To test if nvidia docker works,

# Method 1
docker run --gpus all nvidia/cuda:10.0-base nvidia-smi
# Method 2 (Download runtime and enable daemon first, see below)
docker run --runtime nvidia nvidia/cuda:10.1-base-ubuntu18.04 nvidia-smi

Note that --runtime nvidia will not work by default. To enable it, * Download nvidia-container-runtime

sudo apt-get install nvidia-container-runtime
sudo systemctl daemon-reload
  • Create the file /etc/docker/daemon.json. Restart daemon with
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

Additonally, both --gpu and --runtimes will not work with a docker-compose file as of April 2020. See 1 and 2. Instead, use docker run. Below is a conversion from docker-compose.yml to docker run.

docker run \
  --restart=always \
  --network="traefik-web" \
  --expose=8888 \
  --gpus all \
  -v "/home/jon/Docker/gpu-jupyter/data:/home/jovyan/work" \
  -v "/home/jon/Docker/gpu-jupyter/config:/etc/jupyter" \
  -e "REQFILE=/home/jovyan/work/environment/requirements.txt" \
  -l "traefik.enable=true" \
  -l 'traefik.http.routers.gpu-jupyter.rule=Host(`jupyter.jonai.teojy.com`)' \
  -l "traefik.http.routers.gpu-jupyter.entrypoints=web" \
  -itd \
  --name=gpu-jupyter \
  gpu-jupyter /bin/bash -c '[[ -f $REQFILE ]] && pip install -r $REQFILE; start-notebook.sh'

Troubleshoot#

  • On jupyter public server, kernel not connecting.
    • Inspect console.
    • Check Jupyter Output ```bash

    Bash output when running container in undetached mode.#

    Note the 400 Get Request#

    gpu-jupyter | [I 11:18:32.754 NotebookApp] Creating new notebook in gpu-jupyter | [I 11:18:34.165 NotebookApp] Kernel started: 721b052c-65a3-42b8-ac36-777dba981ffc gpu-jupyter | [W 11:19:04.785 NotebookApp] 400 GET /api/kernels/721b052c-65a3-42b8-ac36-777dba981ffc/channels?session_id=d004549b7ee749b88f8ff24f98593311 (172.25.0.3) 7.43ms referer=None gpu-jupyter | [W 11:19:06.061 NotebookApp] Replacing stale connection: 721b052c-65a3-42b8-ac36-777dba981ffc:d004549b7ee749b88f8ff24f98593311 ``` * This issue is documented here. To resolve, enable websocket on reverse proxy. Since Traefik enables it by default, in my case, I only need to enable it on Synology.