How to set up jupyter on personal server#
gpu-jupyter#
From the Quickstart section of this Guide and Guide
"First of all, it is necessary to generate the Dockerfile based on the latest toolstack of hub.docker.com/u/jupyter. As soon as you have access to your GPU locally (it can be tested via a Tensorflow or PyTorch directly on the host node), you can run these commands to start the jupyter notebook via docker-compose (internally):"
./generate_Dockerfile.sh
docker build -t gpu-jupyter .build/
docker run -d -p [port]:8888 gpu-jupyter
"Alternatively, you can configure the environment in docker-compose.yml and run this to deploy the GPU-Jupyter via docker-compose (under-the-hood):"
./generate_Dockerfile.sh
./start-local.sh -p 8888 # where -p stands for the port of the service
Customization#
There are several customizations that I made for my specific use case. After pulling files from github and running ./generate_Dockerfile.sh, we have the following directory structure (I have listed the useful configuration files)
.
├── add-to-swarm.sh
├── .build
│ ├── Dockerfile
│ ├── jupyter_notebook_config.json
│ ├── jupyter_notebook_config.py
│ └── ...
├── data
│ ├── environment
│ ├── Getting_Started
│ └── ...
├── docker-compose-swarm.yml
├── docker-compose.yml
├── generate_Dockerfile.sh
├── src
│ ├── jupyter_notebook_config.json
│ └── ...
└── ...
Config files#
The generated .build/Dockerfile copies jupyter_notebook_config.json and jupyter_notebook_config.py into /etc/jupyter in the container. Instead of building an image everytime we make a change to jupyter_notebook_config.py, instead we mount /etc/jupyter to a new config folder which contains the .json and .py config files.
To mount /etc/jupyter in the image, we add to the Dockerfile
# Mount config folder
VOLUME ["/etc/jupyter"]
and bind mount it to ./config in docker-compose.yml.
volumes:
- ./config:/etc/jupyter
Note that the pre-generated docker-compose.yml file is primarily a build file instead of a run file. To configure a run file, I created a new docker-compose.yml and backed up the previous one to docker-compose-build.yml.
Installing additional pip requirements#
To run the docker container with a new cookiecutter templated project, we may need to install new packages. I have overwritten the CMD option of the Dockerfile in docker-compose.yml to look for requirements.txt in the mounted data/environment folder and install it.
volumes:
- ./data:/home/jovyan/work
environment:
REQFILE: "/home/jovyan/work/environment/requirements.txt"
command: /bin/bash -c "[[ -f "$$REQFILE" ]] && pip install -r $$REQFILE; start-notebook.sh"
Add Traefik and network support#
Refer to Traefik's guide for more information.
services:
gpu-jupyter:
...
networks:
- traefik-web
labels:
- "traefik.enable=true"
- "traefik.http.routers.gpu-jupyter.rule=Host(`jupyter.jonai.teojy.com`)"
- "traefik.http.routers.gpu-jupyter.entrypoints=web"
networks:
traefik-web:
external: true
Switch from password to token#
I want token based access so that atom/hydrogen can connect to the kernel. To do so, I added and changed the following options in ./config/jupyter_notebook_config.py
c = get_config()
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.port = 8888
c.NotebookApp.open_browser = False
c.NotebookApp.token = '[SecretToken]'
c.NotebookApp.allow_password_change = False
c.NotebookApp.allow_origin = '*'
I also removed the password in ./config/jupyter_notebook_config.json
{
"NotebookApp": {}
}
If successful, you will be prompted for a token instead of a password.
Install Nvidia GPU Docker#
GPU is not enabled by default. Run the following code to enable nvidia in Docker.
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
echo $distribution # this shows your version
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker
#sudo systemctl status docker
To test if nvidia docker works,
# Method 1
docker run --gpus all nvidia/cuda:10.0-base nvidia-smi
# Method 2 (Download runtime and enable daemon first, see below)
docker run --runtime nvidia nvidia/cuda:10.1-base-ubuntu18.04 nvidia-smi
Note that --runtime nvidia will not work by default. To enable it,
* Download nvidia-container-runtime
sudo apt-get install nvidia-container-runtime
sudo systemctl daemon-reload
- Create the file
/etc/docker/daemon.json. Restart daemon with
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "/usr/bin/nvidia-container-runtime",
"runtimeArgs": []
}
}
}
Additonally, both --gpu and --runtimes will not work with a docker-compose file as of April 2020. See 1 and 2. Instead, use docker run. Below is a conversion from docker-compose.yml to docker run.
docker run \
--restart=always \
--network="traefik-web" \
--expose=8888 \
--gpus all \
-v "/home/jon/Docker/gpu-jupyter/data:/home/jovyan/work" \
-v "/home/jon/Docker/gpu-jupyter/config:/etc/jupyter" \
-e "REQFILE=/home/jovyan/work/environment/requirements.txt" \
-l "traefik.enable=true" \
-l 'traefik.http.routers.gpu-jupyter.rule=Host(`jupyter.jonai.teojy.com`)' \
-l "traefik.http.routers.gpu-jupyter.entrypoints=web" \
-itd \
--name=gpu-jupyter \
gpu-jupyter /bin/bash -c '[[ -f $REQFILE ]] && pip install -r $REQFILE; start-notebook.sh'
Troubleshoot#
- On jupyter public server, kernel not connecting.
- Inspect console.
- Check Jupyter Output ```bash
Bash output when running container in undetached mode.#
Note the 400 Get Request#
gpu-jupyter | [I 11:18:32.754 NotebookApp] Creating new notebook in gpu-jupyter | [I 11:18:34.165 NotebookApp] Kernel started: 721b052c-65a3-42b8-ac36-777dba981ffc gpu-jupyter | [W 11:19:04.785 NotebookApp] 400 GET /api/kernels/721b052c-65a3-42b8-ac36-777dba981ffc/channels?session_id=d004549b7ee749b88f8ff24f98593311 (172.25.0.3) 7.43ms referer=None gpu-jupyter | [W 11:19:06.061 NotebookApp] Replacing stale connection: 721b052c-65a3-42b8-ac36-777dba981ffc:d004549b7ee749b88f8ff24f98593311 ``` * This issue is documented here. To resolve, enable websocket on reverse proxy. Since Traefik enables it by default, in my case, I only need to enable it on Synology.
- Inspect console.