Welcome to HPC Helper’s documentation!#

This documentation mainly summarises my experience with Sunbird and mainly Accelerate AI, a HPC node which is a part of Supercomputing Wales. One can follow this basic tutorial to understand the standard operations on any HPC cluster.

The tutorials are classified as basic, Intermediate and advanced. An official but super complicated guide is available here. So, if you consider yourself capable enough to compile LFS, then go for it.

Personal Tweaks#

By default, the .bashrc is read-only for the $USER and regenerates itself after the major updates. We are provided with ~/.myenv to put all the personal tweaks such as the login directory, print a message, export environment variables etc.

Care should be taken that directly printing a message on STDOUT will create problems with SFTP connections, instead we can use STDERR to print the message. Here is an example.

[s.1915438@sl1(sunbird) s.1915438]$ cat $HOME/.myenv
echo "MESSAGE : env variables loaded">&2
export VL_DIR=/scratch/s.1915438/vltesting/VirtualLab
export PATH=$PATH:/scratch/s.1915438/vltesting/VirtualLab/bin

cd /scratch/s.1915438
pwd>&2

This will echo the MESSAGE, the pwd, create the environment variables and cd to the /scratch/s.1915438 partition each time a new TTY session is opened in a login or a compute node.

EDIT : 15th Feb 2023 : I found a disastrous thing with Python venv. So if you are using Python venv, read the life saving tutorial below.:

Life Saving Tutorials#

Python venv Simlink issue

Slow SSH and GVFS on Ubuntu 22.04

Basic Tutorials#

SSH basics

Remote VS Code

Submit Interactive Jobs

Submit Batch Jobs

MIG partition Issues

Intermediate Tutorials#

Jupyter Lab port forward on local network

Jupyter Lab port forward from the Login node of Sunbird

Jupyter Lab port forward from a compute node of Sunbird

Advanced Tutorials#

Remote VNC session

Swansea University VPN Tunnel

Proxy Jump and dynamic port forwarding

Docker Tutorials#

Basic commands

Advanced commands

Port Mapping of Jupyter server

Singularity Tutorials#

Singularity Basics

Volume Binding

Basic Builds

NVIDIA Modulus Singularity container (Failed)

NVIDIA Modulus Setup#

v22.03#

Interactive GPU session

Multi-GPU

With PySDF

v22.03 Summary

v22.07 - Apptainer Image#

Constructing the Apptainer script

Overiding Defaults of Apptainer

Multi-GPU trainign using mpirun

v22.09 - Apptainer Image#

The Apptainer script

Jupyter-lab port forwarding

Sunbird Module

Foolproof installation of tensorflow on HPC#

Tensorflow installation