Python venv symlink issue

So, in 2021 I created multiple python venv on Sunbird using a conda venv. The other way is to compile the Python source code of the desired version which is painful. In Feb 2023, I deleted all the conda venv as they occupy a lot of space and I don’t use them.

It turns out that, all my python venv stopped working. After a long chat with chatgpt i realised some crazy stuff.

When you create a python venv using python3 -m venv /path/to/new/virtual/environment from a conda env with a desired python version, by default on UNIX systems it will create a symbolic link of the python executable.

For example, here I have created a python venv for NVIDIA Modulus from a conda env at /scratch/s.1915438/modulus.

(modulus) [s.1915438@sl2(sunbird) bin]$ ls -la | grep python
-rwxrwxr-x. 1 s.1915438 s.1915438     3498 Nov 22 11:46 activate-global-python-argcomplete
-rwxrwxr-x. 1 s.1915438 s.1915438      250 Oct 31 12:02 ipython
-rwxrwxr-x. 1 s.1915438 s.1915438      250 Oct 31 12:02 ipython3
lrwxrwxrwx. 1 s.1915438 s.1915438        7 Apr 17  2022 python -> python3
lrwxrwxrwx. 1 s.1915438 s.1915438       38 Apr 17  2022 python3 -> /scratch/s.1915438/modulus/bin/python3
lrwxrwxrwx. 1 s.1915438 s.1915438        7 Apr 17  2022 python3.9 -> python3
-rwxrwxr-x. 1 s.1915438 s.1915438     2581 Nov 22 11:46 python-argcomplete-check-easy-install-script
-rwxrwxr-x. 1 s.1915438 s.1915438      383 Nov 22 11:46 python-argcomplete-tcsh
-rwxrwxr-x. 1 s.1915438 s.1915438     1943 Nov 22 11:46 register-python-argcomplete

If you see the long details of the python executable, it points to the Conda venv’s executable at /scratch/s.1915438/modulus/bin/python3. This is typical Linux behaviour, where they don’t make multiple copies of the same executable to prevent bloats. Thats why I support Windows for bloating dll files. At least when you delete the old useless conda venv it won’t break the symlink. Linux Torvalds, remember you said something about NVIDIA? Well, same to you. Allow the user to bloat the dependencies.

How to copy the executable?

So, the Python developers knew about this and documented this deep within the docs. Here you will see all the usage of venv. We are interested in --copies flag. As per the doc, this flag serves: “Try to use copies rather than symlinks, even when symlinks are the default for the platform.”

Now the modified command to create a venv from a Conda venv without symlinks becomes:

python3 -m venv --copies /path/to/new/virtualenv

I tested this on a new python venv:

(modulus_pysdf) [s.1915438@sl1(sunbird) s.1915438]$ python3 -m venv --copies /scratch/s.1915438/temp1/
(modulus_pysdf) [s.1915438@sl1(sunbird) s.1915438]$ deactivate
[s.1915438@sl2(sunbird) s.1915438]$ cd temp1
[s.1915438@sl2(sunbird) temp1]$ ls -l bin/ | grep python3
-rwxr-xr-x. 1 s.1915438 s.1915438 15035704 Feb 15 12:53 python3
[s.1915438@sl2(sunbird) temp1]$ ls -l bin/ | grep python
-rwxr-xr-x. 1 s.1915438 s.1915438 15035704 Feb 15 12:53 python
-rwxr-xr-x. 1 s.1915438 s.1915438 15035704 Feb 15 12:53 python3

See no simlinks. This endeavours took me 3-4 hours and thanks to ChatGPT to save my time.