[Archived] Welcome to HPC Helper’s documentation!#

Warning

Sunbird is being replaced somewhere around the start of 2025. However, the system will remain mostly the same and these tutorials will still be useful.

Note

I finished my PhD in Nov 2024 and stopped using Sunbird. I am archiving this documentation and will not be updating it anymore. I will start a new repo for ARCHER2, CSD3 and Cumulus.

This documentation summaries my experience with Sunbird and mainly Accelerate AI, a HPC node which is a part of Supercomputing Wales. One can follow this basic tutorial to understand the standard operations on any HPC cluster.

The tutorials are classified as basic, Intermediate and advanced. An official but super complicated guide is available here. So, if you consider yourself capable enough to compile LFS, then go for it.

Personal Tweaks#

By default, the .bashrc is read-only for the $USER and regenerates itself after the major updates. We are provided with ~/.myenv to put all the personal tweaks such as the login directory, print a message, export environment variables etc. In fact, ~/.myenv is not something fancy but the autogenerated ~/.bashrc on sunbird sources the ~/.myenv file.

[s.1915438@sl2(sunbird) s.1915438]$ cat ~/.bashrc
# Personal settings file
if [ -f $HOME/.myenv ]
then
   source $HOME/.myenv
fi

Care should be taken that directly printing a message on STDOUT will create problems with SFTP connections, instead we can use STDERR to print the message. Here is an example.

[s.1915438@sl1(sunbird) s.1915438]$ cat $HOME/.myenv
echo "MESSAGE : env variables loaded">&2
export VL_DIR=/scratch/s.1915438/vltesting/VirtualLab
export PATH=$PATH:/scratch/s.1915438/vltesting/VirtualLab/bin

cd /scratch/s.1915438
pwd>&2

This will echo the MESSAGE, the pwd, create the environment variables and cd to the /scratch/s.1915438 partition each time a new TTY session is opened in a login or a compute node.

EDIT : 15th Feb 2023 : I found a disastrous thing with Python venv. So if you are using Python venv, read the life saving tutorial below.:

Life Saving Tutorials#

Python venv Simlink issue

Slow SSH and GVFS on Ubuntu 22.04

Basic Tutorials#

SSH basics

Remote VS Code

Submit Interactive Jobs

Submit Batch Jobs

MIG partition Issues

X forwarding

Intermediate Tutorials#

Jupyter Lab port forward on local network

Jupyter Lab port forward from the Login node of Sunbird

Jupyter Lab port forward from a compute node of Sunbird

Advanced Tutorials#

Remote VNC session

Swansea University VPN Tunnel

Proxy Jump and dynamic port forwarding

Docker Tutorials#

Basic commands

Advanced commands

Port Mapping of Jupyter server

Singularity Tutorials#

Singularity Basics

Volume Binding

Basic Builds

NVIDIA Modulus Singularity container (Failed)

NVIDIA Modulus Setup#

v22.03#

Interactive GPU session

Multi-GPU

With PySDF

v22.03 Summary

v22.07 - Apptainer Image#

Constructing the Apptainer script

Overiding Defaults of Apptainer

Multi-GPU trainign using mpirun

v22.09 - Apptainer Image#

The Apptainer script

Jupyter-lab port forwarding

Sunbird Module

Foolproof installation of tensorflow on HPC#

Tensorflow installation