Nvidia Modulus with tesselated geometry module on Sunbird#
Date: 28 Jun 2022
I struggled to install the tesselated geometry module (PySDF) for months until today. To start with, let us use the old school trick to install specific version of Python. After some pregress, I found that PySDF only works with Python 3.8 (Nvidia will never tell you this). See the following error:
>>> import pysdf.sdf as pysdf
<frozen importlib._bootstrap>:228: RuntimeWarning: compiletime version 3.8 of module 'pysdf.sdf' does not match runtime version 3.9
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/scratch/s.1915438/modulus/lib/python3.9/site-packages/pysdf-0.1-py3.8-linux-x86_64.egg/pysdf/sdf.py", line 9, in <module>
__bootstrap__()
File "/scratch/s.1915438/modulus/lib/python3.9/site-packages/pysdf-0.1-py3.8-linux-x86_64.egg/pysdf/sdf.py", line 8, in __bootstrap__
spec.loader.exec_module(mod)
File "pysdf/sdf.pyx", line 1, in init pysdf.sdf
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
>>> exit()
This error tells us that the PySDF is compiled on Python 3.8 and won’t work with Python 3.9. So we will create a new Python 3.8 venv. As discussed in earlier ipynb (IronPython Notebook) files, the conda environment is not able to set an environment variable for the Hydra configuration manager to work. So, Conda/ Anaconda is not going to work with Nvidia Modulus. However, I looked at the Nvidia Modulus Docker image, they are using a Conda environment (the following output). Of course, they are smarter than me and can work out problems with their experience.
root@3d6e5895c649:/opt/conda/bin# cat activate
#!/bin/sh
_CONDA_ROOT="/opt/conda"
# Copyright (C) 2012 Anaconda, Inc
# SPDX-License-Identifier: BSD-3-Clause
\. "$_CONDA_ROOT/etc/profile.d/conda.sh" || return $?
conda activate "$@"
Installing Python 3.8#
[s.1915438@sl2 ~]$ module load anaconda/2021.05
[s.1915438@sl2 ~]$ conda create -n modulus_surrogate python=3.8
After long output message the conda environment will be installed.
[s.1915438@sl2 ~]$ conda env list
# conda environments:
#
base * /apps/local/languages/anaconda/2021.05
ml /home/s.1915438/.conda/envs/ml
modulus /home/s.1915438/.conda/envs/modulus
modulus_surrogate /home/s.1915438/.conda/envs/modulus_surrogate
/scratch/s.1915438/modulus
[s.1915438@sl2 ~]$ source activate modulus_surrogate
(modulus_surrogate) [s.1915438@sl2 ~]$ which python
~/.conda/envs/modulus_surrogate/bin/python
(modulus_surrogate) [s.1915438@sl2 ~]$ python --version
Python 3.8.13
Now we can create a new Python venv using the Conda env.
(modulus_surrogate) [s.1915438@sl2 ~]$ cd /scratch/s.1915438
(modulus_surrogate) [s.1915438@sl2 s.1915438]$ mkdir modulus_pysdf
(modulus_surrogate) [s.1915438@sl2 s.1915438]$ cd modulus_pysdf/
(modulus_surrogate) [s.1915438@sl2 modulus_pysdf]$ python3 -m venv modulus_pysdf
(modulus_surrogate) [s.1915438@sl2 modulus_pysdf]$ ls
modulus_pysdf
Now to close the Conda env, either use Module purge or simply logout by pressing ctrl + D, adn then re-login.
(modulus_surrogate) [s.1915438@sl2 modulus_pysdf]$ logout
Connection to sunbird.swansea.ac.uk closed.
(base) hell@Dell-Precision-T1600:~$ ssh sunbird
Installing Pytorch#
[s.1915438@sl2 ~]$ cd /scratch/s.1915438
[s.1915438@sl2 s.1915438]$ cd modulus_pysdf/
[s.1915438@sl2 modulus_pysdf]$ source modulus_pysdf/bin/activate
(modulus_pysdf) [s.1915438@sl2 modulus_pysdf]$ python --version
Python 3.8.13
(modulus_pysdf) [s.1915438@sl2 modulus_pysdf]$ which python
/scratch/s.1915438/modulus_pysdf/modulus_pysdf/bin/python
Remember, I Sunbird’s Accelerated AI node has Nvidia A100, which does not work with CUDA 10.2 so we will use CUDA 11.3.
(modulus_pysdf) [s.1915438@sl2 modulus_pysdf]$ pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
[IN THE END]
Successfully installed certifi-2022.6.15 charset-normalizer-2.0.12 idna-3.3 numpy-1.23.0 pillow-9.1.1 requests-2.28.0 torch-1.12.0+cu113 torchaudio-0.12.0+cu113 torchvision-0.13.0+cu113 typing-extensions-4.2.0 urllib3-1.26.9
Installing Modulus dependencies#
(modulus_pysdf) [s.1915438@sl2 modulus_pysdf]$ pip3 install matplotlib transforms3d future typing numpy quadpy numpy-stl==2.11.2 h5py sympy==1.5.1 termcolor psutil symengine==0.6.1 numba Cython chaospy torch_optimizer vtk chaospy termcolor omegaconf hydra-core einops timm tensorboard pandas orthopy ndim
Similarly we can install PyEVTK.
(modulus_pysdf) [s.1915438@sl2 modulus_pysdf]$ pip3 install -U https://github.com/paulo-herrera/PyEVTK/archive/v1.1.2.tar.gz
These commands are included in requirements.txt in the same directory.
Installing Modulus#
Navigate to Nvidia Modulus’s source directory and invoke the setup.py script.
(modulus_pysdf) [s.1915438@sl2 modulus_pysdf]$ pwd
/scratch/s.1915438/modulus_pysdf
(modulus_pysdf) [s.1915438@sl2 modulus_pysdf]$ cd ..
(modulus_pysdf) [s.1915438@sl2 s.1915438]$ cd Modulus_source/
(modulus_pysdf) [s.1915438@sl2 Modulus_source]$ ls
Modulus Modulus_examples.tar.gz modulus_image_v22.03.1.tar.gz
(modulus_pysdf) [s.1915438@sl2 Modulus_source]$ cd Modulus
(modulus_pysdf) [s.1915438@sl2 Modulus]$ ls
accompanying_licences changelog_tensorflow.md Dockerfile MANIFEST.in modulus.egg-info README.md setup.cfg
build dist external modulus NVIDIA-OptiX-SDK-7.0.0-linux64.sh requirements.txt setup.py
(modulus_pysdf) [s.1915438@sl2 Modulus]$ python setup.py install
[IN THE END]
Using /scratch/s.1915438/modulus_pysdf/modulus_pysdf/lib/python3.8/site-packages
Finished processing dependencies for modulus==22.3
To install PySDF, we need to use easy_install which was depreciated after setuptools 42.0.0. So we will downgrade setuptools to 42.0.0. Before downgrading, if you wish to restore the version after installing PySDF use pip list to note down the current version of setuptools which must be 56.0.0 for Python 3.8
(modulus_pysdf) [s.1915438@sl2 Modulus]$ pip3 install setuptools==42.0.0
Now we can use the easy_install to install the egg files.
(modulus_pysdf) [s.1915438@sl2 Modulus]$ pwd
/scratch/s.1915438/Modulus_source/Modulus
(modulus_pysdf) [s.1915438@sl2 Modulus]$ cd external/eggs/
(modulus_pysdf) [s.1915438@sl2 eggs]$ ls
pysdf-0.1-py3.8-linux-x86_64.egg
(modulus_pysdf) [s.1915438@sl2 eggs]$ python3 -m easy_install pysdf-0.1-py3.8-linux-x86_64.egg
[IN THE END]
Finished processing dependencies for pysdf==0.1
Also we can add the libsdf.so to LD_LIBRARY_PATH as follows:
(modulus_pysdf) [s.1915438@sl2 eggs]$ pwd
/scratch/s.1915438/Modulus_source/Modulus/external/eggs
(modulus_pysdf) [s.1915438@sl2 eggs]$ cd ..
(modulus_pysdf) [s.1915438@sl2 external]$ ls
eggs lib pysdf
(modulus_pysdf) [s.1915438@sl2 external]$ export LD_LIBRARY_PATH=$(pwd)/lib/:${LD_LIBRARY_PATH}
This time PySDF should run fine as follows
(modulus_pysdf) [s.1915438@sl2 external]$ python
Python 3.8.13 (default, Mar 28 2022, 11:38:47)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysdf.sdf
>>> import pysdf.sdf as pysdf
Trying an example#
Fire an A100 and run the Aneurysm example which involves tessellation.
(modulus_pysdf) [s.1915438@sl2 aneurysm]$ salloc --nodes=1 --account=scw1901 --partition=accel_ai --gres=gpu:1
salloc: Pending job allocation 7189600
salloc: job 7189600 queued and waiting for resources
salloc: job 7189600 has been allocated resources
salloc: Granted job allocation 7189600
salloc: Waiting for resource configuration
salloc: Nodes scs2041 are ready for job
(modulus_pysdf) [s.1915438@sl2 aneurysm]$ srun python aneurysm.py >>training.txt
If it runs without any error then its fine otherwise we have to go little wild. In my case, I got this error.
<frozen importlib._bootstrap>:219: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
Traceback (most recent call last):
File "aneurysm.py", line 225, in <module>
run()
File "/scratch/s.1915438/modulus_pysdf/modulus_pysdf/lib/python3.8/site-packages/modulus-22.3-py3.8.egg/modulus/hydra/utils.py", line 66, in func_decorated
_run_hydra(
TypeError: _run_hydra() missing 1 required positional argument: 'args'
This is probably because some packages are missing which Nvidia forgot to mention in the installation instructions.
Running Nvidia Modulus Docker image#
Now we have to launch Nvidia Modulus Docker image on local PC as follows:
(base) hell@Dell-Precision-T1600:~$ docker container ls
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied
(base) hell@Dell-Precision-T1600:~$ sudo chmod 666 /var/run/docker.sock
[sudo] password for hell:
(base) hell@Dell-Precision-T1600:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
modulus 22.03 6459de576d3d 2 months ago 15.8GB
jupyter/all-spark-notebook latest 94d9cd301780 3 months ago 4.58GB
jupyter/minimal-notebook latest 1f969b467a59 3 months ago 1.46GB
alpine latest c059bfaa849c 7 months ago 5.59MB
prakhars962/nvidia_modulus test 90a595b1279a 7 months ago 14.9GB
modulus 21.06 90a595b1279a 7 months ago 14.9GB
hello-world latest feb5d9fea6a5 9 months ago 13.3kB
ibsim/virtuallab latest da04b3610585 19 months ago 11.2GB
nvidia/cuda 11.0-base 2ec708416bb8 22 months ago 122MB
We will load the modulus:22.03 image. You can get it in this list using docker load IMAGE_NAME. Then I will run an interactive session.
(base) hell@Dell-Precision-T1600:~$ docker run -it modulus:22.03
root@3d6e5895c649:/examples# pip freeze
Copy the output in a txt file on local PC.
Alternatively, One can bind any volume and then simply use > to save the output.
(base) hell@Dell-Precision-T1600:~$ docker run -it -v $HOME:/shared/ modulus:22.03
root@3d6e5895c649:/examples# cd /shared
root@3d6e5895c649:/examples# pip freeze >>docker_requirements.txt
Transfer the txt file to SUNBIRD using sftp. You will notice that in the txt file some requirements contains a path which SUNBIRD won’t understand. To skip those specific lines, I took stole the following line of code from this article:
(modulus_pysdf) [s.1915438@sl2 aneurysm]$ cat requirements.txt | grep -Eo '(^[^#]+)' | xargs -n 1 pip install
Now wait for 15-20 minutes. Now you will see that the PyTorch’s CUDA version is 10.2 which is not compatible with Nvidia A100.
(modulus_pysdf) [s.1915438@scs2041 aneurysm]$ python aneurysm.py >> requirements.txt
/scratch/s.1915438/modulus_pysdf/modulus_pysdf/lib/python3.8/site-packages/torch/cuda/__init__.py:145: UserWarning:
NVIDIA A100-PCIE-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA A100-PCIE-40GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
We need to upgrade PyTorch’s CUDA to 11.3 as follows:
(modulus_pysdf) [s.1915438@sl2 aneurysm]$ pip3 install --upgrade torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
Final testing#
Now the full Nvidia Modulus should work.
(modulus_pysdf) [s.1915438@sl2 aneurysm]$ salloc --nodes=1 --account=scw1901 --partition=accel_ai --gres=gpu:1
salloc: Pending job allocation 7189689
salloc: job 7189689 queued and waiting for resources
salloc: job 7189689 has been allocated resources
salloc: Granted job allocation 7189689
salloc: Waiting for resource configuration
salloc: Nodes scs2041 are ready for job
(modulus_pysdf) [s.1915438@sl2 aneurysm]$ srun --pty bash
(modulus_pysdf) [s.1915438@scs2041 aneurysm]$ ls
aneurysm.py conf openfoam outputs requirements.txt stl_files training.txt
(modulus_pysdf) [s.1915438@scs2041 aneurysm]$ python aneurysm.py >>training.txt
Output of the training:
[s.1915438@sl2 aneurysm]$ cat training.txt
training:
max_steps: 1500000
grad_agg_freq: 1
rec_results_freq: 10000
rec_validation_freq: ${training.rec_results_freq}
rec_inference_freq: ${training.rec_results_freq}
rec_monitor_freq: ${training.rec_results_freq}
rec_constraint_freq: 50000
save_network_freq: 1000
print_stats_freq: 100
summary_freq: 1000
amp: false
amp_dtype: float16
ntk:
use_ntk: false
save_name: null
run_freq: 1000
profiler:
profile: false
start_step: 0
end_step: 100
name: nvtx
network_dir: .
initialization_network_dir: ''
save_filetypes: vtk
summary_histograms: false
jit: false
device: ''
debug: false
run_mode: train
arch:
fully_connected:
_target_: modulus.architecture.fully_connected.FullyConnectedArch
layer_size: 512
nr_layers: 6
skip_connections: false
adaptive_activations: false
weight_norm: true
loss:
_target_: modulus.aggregator.Sum
weights: null
optimizer:
_params_:
compute_gradients: adam_compute_gradients
apply_gradients: adam_apply_gradients
_target_: torch.optim.Adam
lr: 0.001
betas:
- 0.9
- 0.999
eps: 1.0e-08
weight_decay: 0.0
amsgrad: false
scheduler:
_target_: custom
_name_: tf.ExponentialLR
decay_rate: 0.95
decay_steps: 15000
batch_size:
inlet: 1100
outlet: 650
no_slip: 5200
interior: 6000
integral_continuity: 310
custom: ???
[13:30:00] - attempting to restore from: outputs/aneurysm
[13:30:00] - optimizer checkpoint not found
[13:30:00] - model flow_network.pth not found
[13:30:02] - [step: 0] record contraint batch time: 2.485e-01s
[13:30:13] - [step: 0] record validators time: 1.049e+01s
[13:30:13] - [step: 0] record monitor time: 9.495e-03s
[13:30:13] - [step: 0] saved checkpoint to outputs/aneurysm
[13:30:13] - [step: 0] loss: 1.086e+01
[13:30:34] - [step: 100] loss: 2.584e+00, time/iteration: 2.140e+02ms
[13:30:55] - [step: 200] loss: 1.867e+00, time/iteration: 2.136e+02ms
[13:31:17] - [step: 300] loss: 1.509e+00, time/iteration: 2.142e+02ms
[13:31:38] - [step: 400] loss: 2.100e+00, time/iteration: 2.144e+02ms
[13:32:00] - [step: 500] loss: 1.361e+00, time/iteration: 2.145e+02ms
[13:32:21] - [step: 600] loss: 1.270e+00, time/iteration: 2.144e+02ms
[13:32:43] - [step: 700] loss: 1.122e+00, time/iteration: 2.145e+02ms
[13:33:04] - [step: 800] loss: 1.113e+00, time/iteration: 2.145e+02ms
[13:33:26] - [step: 900] loss: 1.008e+00, time/iteration: 2.144e+02ms
[13:33:48] - [step: 1000] saved checkpoint to outputs/aneurysm
[13:33:48] - [step: 1000] loss: 9.468e-01, time/iteration: 2.201e+02ms
[13:34:09] - [step: 1100] loss: 1.039e+00, time/iteration: 2.144e+02ms
[13:34:30] - [step: 1200] loss: 8.988e-01, time/iteration: 2.144e+02ms
[13:34:52] - [step: 1300] loss: 9.038e-01, time/iteration: 2.144e+02ms
[13:35:13] - [step: 1400] loss: 8.228e-01, time/iteration: 2.144e+02ms
[13:35:35] - [step: 1500] loss: 8.784e-01, time/iteration: 2.144e+02ms
[13:35:56] - [step: 1600] loss: 7.991e-01, time/iteration: 2.144e+02ms
[13:36:18] - [step: 1700] loss: 7.431e-01, time/iteration: 2.144e+02ms
[13:36:39] - [step: 1800] loss: 7.510e-01, time/iteration: 2.144e+02ms
[13:37:01] - [step: 1900] loss: 9.696e-01, time/iteration: 2.144e+02ms
[13:37:23] - [step: 2000] saved checkpoint to outputs/aneurysm
[13:37:23] - [step: 2000] loss: 7.899e-01, time/iteration: 2.199e+02ms
[13:37:44] - [step: 2100] loss: 7.179e-01, time/iteration: 2.144e+02ms
[13:38:05] - [step: 2200] loss: 6.527e-01, time/iteration: 2.144e+02ms
[13:38:27] - [step: 2300] loss: 6.867e-01, time/iteration: 2.145e+02ms
[13:38:48] - [step: 2400] loss: 6.844e-01, time/iteration: 2.145e+02ms
[13:39:10] - [step: 2500] loss: 6.986e-01, time/iteration: 2.144e+02ms
[13:39:31] - [step: 2600] loss: 8.633e-01, time/iteration: 2.144e+02ms
[13:39:53] - [step: 2700] loss: 7.654e-01, time/iteration: 2.144e+02ms
[13:40:14] - [step: 2800] loss: 6.785e-01, time/iteration: 2.144e+02ms
[13:40:36] - [step: 2900] loss: 5.918e-01, time/iteration: 2.144e+02ms
[13:40:58] - [step: 3000] saved checkpoint to outputs/aneurysm
[13:40:58] - [step: 3000] loss: 6.678e-01, time/iteration: 2.199e+02ms
[13:41:19] - [step: 3100] loss: 6.721e-01, time/iteration: 2.144e+02ms
[13:41:40] - [step: 3200] loss: 5.690e-01, time/iteration: 2.145e+02ms
[13:42:02] - [step: 3300] loss: 8.973e-01, time/iteration: 2.144e+02ms
[13:42:23] - [step: 3400] loss: 7.599e-01, time/iteration: 2.144e+02ms
[13:42:45] - [step: 3500] loss: 5.763e-01, time/iteration: 2.144e+02ms
[13:43:06] - [step: 3600] loss: 5.833e-01, time/iteration: 2.145e+02ms
[13:43:28] - [step: 3700] loss: 5.496e-01, time/iteration: 2.144e+02ms
[13:43:49] - [step: 3800] loss: 5.267e-01, time/iteration: 2.145e+02ms
[13:44:11] - [step: 3900] loss: 6.246e-01, time/iteration: 2.145e+02ms
[13:44:33] - [step: 4000] saved checkpoint to outputs/aneurysm
[13:44:33] - [step: 4000] loss: 5.239e-01, time/iteration: 2.199e+02ms
[13:44:54] - [step: 4100] loss: 5.447e-01, time/iteration: 2.144e+02ms
[13:45:15] - [step: 4200] loss: 6.787e-01, time/iteration: 2.145e+02ms
[13:45:37] - [step: 4300] loss: 5.597e-01, time/iteration: 2.144e+02ms
[13:45:58] - [step: 4400] loss: 5.680e-01, time/iteration: 2.144e+02ms
[13:46:20] - [step: 4500] loss: 5.241e-01, time/iteration: 2.145e+02ms
[13:46:41] - [step: 4600] loss: 5.708e-01, time/iteration: 2.144e+02ms
[13:47:03] - [step: 4700] loss: 5.433e-01, time/iteration: 2.144e+02ms
[13:47:24] - [step: 4800] loss: 4.926e-01, time/iteration: 2.145e+02ms
[13:47:46] - [step: 4900] loss: 6.314e-01, time/iteration: 2.145e+02ms
[13:48:08] - [step: 5000] saved checkpoint to outputs/aneurysm
[13:48:08] - [step: 5000] loss: 5.391e-01, time/iteration: 2.200e+02ms
[13:48:29] - [step: 5100] loss: 5.055e-01, time/iteration: 2.145e+02ms
[13:48:50] - [step: 5200] loss: 4.975e-01, time/iteration: 2.145e+02ms
[13:49:12] - [step: 5300] loss: 5.335e-01, time/iteration: 2.144e+02ms
[13:49:33] - [step: 5400] loss: 5.286e-01, time/iteration: 2.145e+02ms
[13:49:55] - [step: 5500] loss: 5.053e-01, time/iteration: 2.145e+02ms
[13:50:16] - [step: 5600] loss: 4.829e-01, time/iteration: 2.144e+02ms
[13:50:38] - [step: 5700] loss: 5.052e-01, time/iteration: 2.145e+02ms
[13:50:59] - [step: 5800] loss: 6.234e-01, time/iteration: 2.145e+02ms
[13:51:21] - [step: 5900] loss: 4.868e-01, time/iteration: 2.145e+02ms
[13:51:43] - [step: 6000] saved checkpoint to outputs/aneurysm
[13:51:43] - [step: 6000] loss: 4.938e-01, time/iteration: 2.200e+02ms
[13:52:04] - [step: 6100] loss: 5.278e-01, time/iteration: 2.144e+02ms
[13:52:25] - [step: 6200] loss: 4.609e-01, time/iteration: 2.144e+02ms
[13:52:47] - [step: 6300] loss: 4.749e-01, time/iteration: 2.145e+02ms
[13:53:08] - [step: 6400] loss: 5.095e-01, time/iteration: 2.144e+02ms
[13:53:30] - [step: 6500] loss: 4.549e-01, time/iteration: 2.145e+02ms
[13:53:51] - [step: 6600] loss: 4.506e-01, time/iteration: 2.145e+02ms
[13:54:13] - [step: 6700] loss: 4.265e-01, time/iteration: 2.145e+02ms
[13:54:34] - [step: 6800] loss: 4.916e-01, time/iteration: 2.153e+02ms
[13:54:56] - [step: 6900] loss: 5.403e-01, time/iteration: 2.144e+02ms
[13:55:18] - [step: 7000] saved checkpoint to outputs/aneurysm
[13:55:18] - [step: 7000] loss: 4.499e-01, time/iteration: 2.200e+02ms
[13:55:39] - [step: 7100] loss: 4.541e-01, time/iteration: 2.145e+02ms
[13:56:01] - [step: 7200] loss: 4.483e-01, time/iteration: 2.144e+02ms
[13:56:22] - [step: 7300] loss: 5.120e-01, time/iteration: 2.145e+02ms
[13:56:43] - [step: 7400] loss: 4.109e-01, time/iteration: 2.145e+02ms
[13:57:05] - [step: 7500] loss: 4.358e-01, time/iteration: 2.144e+02ms
[13:57:26] - [step: 7600] loss: 4.199e-01, time/iteration: 2.145e+02ms
[13:57:48] - [step: 7700] loss: 5.563e-01, time/iteration: 2.145e+02ms
[13:58:09] - [step: 7800] loss: 5.958e-01, time/iteration: 2.145e+02ms
[13:58:31] - [step: 7900] loss: 4.637e-01, time/iteration: 2.145e+02ms
[13:58:53] - [step: 8000] saved checkpoint to outputs/aneurysm
[13:58:53] - [step: 8000] loss: 5.767e-01, time/iteration: 2.201e+02ms
[13:59:14] - [step: 8100] loss: 4.284e-01, time/iteration: 2.144e+02ms
[13:59:36] - [step: 8200] loss: 4.210e-01, time/iteration: 2.145e+02ms
[13:59:57] - [step: 8300] loss: 4.028e-01, time/iteration: 2.145e+02ms
[14:00:19] - [step: 8400] loss: 3.728e-01, time/iteration: 2.145e+02ms
[14:00:40] - [step: 8500] loss: 3.860e-01, time/iteration: 2.145e+02ms
[14:01:01] - [step: 8600] loss: 4.349e-01, time/iteration: 2.145e+02ms
[14:01:23] - [step: 8700] loss: 3.742e-01, time/iteration: 2.145e+02ms
[14:01:44] - [step: 8800] loss: 3.859e-01, time/iteration: 2.145e+02ms
[14:02:06] - [step: 8900] loss: 3.723e-01, time/iteration: 2.145e+02ms
[14:02:28] - [step: 9000] saved checkpoint to outputs/aneurysm
[14:02:28] - [step: 9000] loss: 4.263e-01, time/iteration: 2.201e+02ms
[14:02:49] - [step: 9100] loss: 4.030e-01, time/iteration: 2.145e+02ms
[14:03:11] - [step: 9200] loss: 4.424e-01, time/iteration: 2.145e+02ms
[14:03:32] - [step: 9300] loss: 4.453e-01, time/iteration: 2.145e+02ms
[14:03:54] - [step: 9400] loss: 3.371e-01, time/iteration: 2.145e+02ms
[14:04:15] - [step: 9500] loss: 6.863e-01, time/iteration: 2.145e+02ms
[14:04:37] - [step: 9600] loss: 3.810e-01, time/iteration: 2.145e+02ms
[14:04:58] - [step: 9700] loss: 7.009e-01, time/iteration: 2.145e+02ms
[14:05:19] - [step: 9800] loss: 3.619e-01, time/iteration: 2.146e+02ms
[14:05:41] - [step: 9900] loss: 3.864e-01, time/iteration: 2.146e+02ms
[14:06:13] - [step: 10000] record validators time: 1.037e+01s
[14:06:13] - [step: 10000] record monitor time: 5.435e-03s
[14:06:13] - [step: 10000] saved checkpoint to outputs/aneurysm
[14:06:13] - [step: 10000] loss: 5.020e-01, time/iteration: 3.239e+02ms
[14:06:35] - [step: 10100] loss: 3.713e-01, time/iteration: 2.143e+02ms
[14:06:56] - [step: 10200] loss: 4.036e-01, time/iteration: 2.144e+02ms
[14:07:18] - [step: 10300] loss: 3.684e-01, time/iteration: 2.144e+02ms
[14:07:39] - [step: 10400] loss: 3.937e-01, time/iteration: 2.145e+02ms
[14:08:00] - [step: 10500] loss: 3.864e-01, time/iteration: 2.145e+02ms
[14:08:22] - [step: 10600] loss: 3.330e-01, time/iteration: 2.145e+02ms
[14:08:43] - [step: 10700] loss: 4.713e-01, time/iteration: 2.146e+02ms
[14:09:05] - [step: 10800] loss: 3.672e-01, time/iteration: 2.146e+02ms
[14:09:26] - [step: 10900] loss: 3.961e-01, time/iteration: 2.145e+02ms
[14:09:48] - [step: 11000] saved checkpoint to outputs/aneurysm
[14:09:48] - [step: 11000] loss: 4.218e-01, time/iteration: 2.201e+02ms
[14:10:10] - [step: 11100] loss: 3.593e-01, time/iteration: 2.146e+02ms
[14:10:31] - [step: 11200] loss: 3.301e-01, time/iteration: 2.145e+02ms
[14:10:53] - [step: 11300] loss: 3.411e-01, time/iteration: 2.145e+02ms
[14:11:14] - [step: 11400] loss: 3.460e-01, time/iteration: 2.146e+02ms
[14:11:36] - [step: 11500] loss: 3.155e-01, time/iteration: 2.145e+02ms
[14:11:57] - [step: 11600] loss: 3.037e-01, time/iteration: 2.146e+02ms
[14:12:19] - [step: 11700] loss: 3.808e-01, time/iteration: 2.146e+02ms
[14:12:40] - [step: 11800] loss: 3.734e-01, time/iteration: 2.145e+02ms
[14:13:01] - [step: 11900] loss: 3.211e-01, time/iteration: 2.146e+02ms
[14:13:23] - [step: 12000] saved checkpoint to outputs/aneurysm
[14:13:23] - [step: 12000] loss: 3.612e-01, time/iteration: 2.202e+02ms
[14:13:45] - [step: 12100] loss: 3.381e-01, time/iteration: 2.145e+02ms
[14:14:06] - [step: 12200] loss: 3.458e-01, time/iteration: 2.146e+02ms
[14:14:28] - [step: 12300] loss: 3.517e-01, time/iteration: 2.146e+02ms
[14:14:49] - [step: 12400] loss: 3.252e-01, time/iteration: 2.145e+02ms
[14:15:11] - [step: 12500] loss: 3.521e-01, time/iteration: 2.146e+02ms
[14:15:32] - [step: 12600] loss: 3.445e-01, time/iteration: 2.146e+02ms
[14:15:54] - [step: 12700] loss: 3.990e-01, time/iteration: 2.146e+02ms
[14:16:15] - [step: 12800] loss: 3.533e-01, time/iteration: 2.146e+02ms
[14:16:37] - [step: 12900] loss: 4.452e-01, time/iteration: 2.146e+02ms
[14:16:59] - [step: 13000] saved checkpoint to outputs/aneurysm
[14:16:59] - [step: 13000] loss: 3.676e-01, time/iteration: 2.202e+02ms
[14:17:20] - [step: 13100] loss: 3.475e-01, time/iteration: 2.145e+02ms
[14:17:42] - [step: 13200] loss: 4.652e-01, time/iteration: 2.154e+02ms
[14:18:03] - [step: 13300] loss: 3.359e-01, time/iteration: 2.146e+02ms
[14:18:24] - [step: 13400] loss: 2.902e-01, time/iteration: 2.145e+02ms
[14:18:46] - [step: 13500] loss: 2.910e-01, time/iteration: 2.145e+02ms
[14:19:07] - [step: 13600] loss: 3.162e-01, time/iteration: 2.145e+02ms
[14:19:29] - [step: 13700] loss: 3.278e-01, time/iteration: 2.146e+02ms
[14:19:50] - [step: 13800] loss: 2.687e-01, time/iteration: 2.145e+02ms
[14:20:12] - [step: 13900] loss: 4.122e-01, time/iteration: 2.146e+02ms
[14:20:34] - [step: 14000] saved checkpoint to outputs/aneurysm
[14:20:34] - [step: 14000] loss: 3.813e-01, time/iteration: 2.201e+02ms
[14:20:55] - [step: 14100] loss: 3.503e-01, time/iteration: 2.145e+02ms
[14:21:17] - [step: 14200] loss: 2.995e-01, time/iteration: 2.146e+02ms
[14:21:38] - [step: 14300] loss: 3.285e-01, time/iteration: 2.146e+02ms
[14:22:00] - [step: 14400] loss: 3.706e-01, time/iteration: 2.145e+02ms
[14:22:21] - [step: 14500] loss: 3.059e-01, time/iteration: 2.146e+02ms
[14:22:42] - [step: 14600] loss: 2.865e-01, time/iteration: 2.146e+02ms
[14:23:04] - [step: 14700] loss: 2.791e-01, time/iteration: 2.145e+02ms
[14:23:25] - [step: 14800] loss: 3.385e-01, time/iteration: 2.146e+02ms
[14:23:47] - [step: 14900] loss: 3.718e-01, time/iteration: 2.146e+02ms
[14:24:09] - [step: 15000] saved checkpoint to outputs/aneurysm
[14:24:09] - [step: 15000] loss: 6.232e-01, time/iteration: 2.201e+02ms
[14:24:30] - [step: 15100] loss: 4.105e-01, time/iteration: 2.145e+02ms
[14:24:52] - [step: 15200] loss: 3.469e-01, time/iteration: 2.146e+02ms
[14:25:13] - [step: 15300] loss: 3.105e-01, time/iteration: 2.146e+02ms
[14:25:35] - [step: 15400] loss: 3.544e-01, time/iteration: 2.146e+02ms
[14:25:56] - [step: 15500] loss: 2.775e-01, time/iteration: 2.146e+02ms
[14:26:18] - [step: 15600] loss: 2.938e-01, time/iteration: 2.145e+02ms
[14:26:39] - [step: 15700] loss: 2.758e-01, time/iteration: 2.146e+02ms
[14:27:01] - [step: 15800] loss: 3.064e-01, time/iteration: 2.146e+02ms
[14:27:22] - [step: 15900] loss: 2.639e-01, time/iteration: 2.146e+02ms
[14:27:44] - [step: 16000] saved checkpoint to outputs/aneurysm
[14:27:44] - [step: 16000] loss: 2.508e-01, time/iteration: 2.202e+02ms
[14:28:05] - [step: 16100] loss: 2.763e-01, time/iteration: 2.146e+02ms
[14:28:27] - [step: 16200] loss: 2.486e-01, time/iteration: 2.146e+02ms
[14:28:48] - [step: 16300] loss: 2.500e-01, time/iteration: 2.146e+02ms
[14:29:10] - [step: 16400] loss: 4.083e-01, time/iteration: 2.146e+02ms
[14:29:31] - [step: 16500] loss: 3.035e-01, time/iteration: 2.146e+02ms
[14:29:53] - [step: 16600] loss: 2.859e-01, time/iteration: 2.146e+02ms
[14:30:14] - [step: 16700] loss: 2.809e-01, time/iteration: 2.146e+02ms
[14:30:36] - [step: 16800] loss: 3.628e-01, time/iteration: 2.146e+02ms
[14:30:57] - [step: 16900] loss: 2.583e-01, time/iteration: 2.145e+02ms
[14:31:19] - [step: 17000] saved checkpoint to outputs/aneurysm
[14:31:19] - [step: 17000] loss: 3.024e-01, time/iteration: 2.202e+02ms
[14:31:41] - [step: 17100] loss: 2.249e-01, time/iteration: 2.145e+02ms
[14:32:02] - [step: 17200] loss: 2.643e-01, time/iteration: 2.145e+02ms
[14:32:24] - [step: 17300] loss: 3.582e-01, time/iteration: 2.146e+02ms
[14:32:45] - [step: 17400] loss: 2.720e-01, time/iteration: 2.145e+02ms
[14:33:06] - [step: 17500] loss: 3.062e-01, time/iteration: 2.145e+02ms
[14:33:28] - [step: 17600] loss: 2.389e-01, time/iteration: 2.146e+02ms
[14:33:49] - [step: 17700] loss: 2.888e-01, time/iteration: 2.146e+02ms
[14:34:12] - [step: 17800] loss: 2.438e-01, time/iteration: 2.221e+02ms
[14:34:33] - [step: 17900] loss: 3.314e-01, time/iteration: 2.145e+02ms
[14:34:55] - [step: 18000] saved checkpoint to outputs/aneurysm
[14:34:55] - [step: 18000] loss: 2.549e-01, time/iteration: 2.212e+02ms
[14:35:17] - [step: 18100] loss: 3.006e-01, time/iteration: 2.144e+02ms
[14:35:38] - [step: 18200] loss: 2.891e-01, time/iteration: 2.145e+02ms
[14:36:00] - [step: 18300] loss: 2.273e-01, time/iteration: 2.145e+02ms
[14:36:21] - [step: 18400] loss: 2.650e-01, time/iteration: 2.151e+02ms
[14:36:42] - [step: 18500] loss: 2.436e-01, time/iteration: 2.145e+02ms
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0247115s
Timing for cpatResultsToArrays: 0.00019965s
Timing for cpatDistanceField: 0.361814s
Timing for computeDistanceField: 0.361823s
Timing for cpatComputeSignedFieldWithNormals: 0.00129972s
Timing for computeSignedDistanceField: 0.366096s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0298971s
Timing for cpatResultsToArrays: 0.00019364s
Timing for cpatDistanceField: 0.338028s
Timing for computeDistanceField: 0.338036s
Timing for cpatComputeSignedFieldWithNormals: 0.00129649s
Timing for computeSignedDistanceField: 0.34143s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.028599s
Timing for cpatResultsToArrays: 0.00019324s
Timing for cpatDistanceField: 0.350655s
Timing for computeDistanceField: 0.350664s
Timing for cpatComputeSignedFieldWithNormals: 0.00130845s
Timing for computeSignedDistanceField: 0.354075s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0245218s
Timing for cpatResultsToArrays: 0.00019225s
Timing for cpatDistanceField: 0.332682s
Timing for computeDistanceField: 0.332691s
Timing for cpatComputeSignedFieldWithNormals: 0.00129491s
Timing for computeSignedDistanceField: 0.336083s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0246146s
Timing for cpatResultsToArrays: 0.0001943s
Timing for cpatDistanceField: 0.323005s
Timing for computeDistanceField: 0.323014s
Timing for cpatComputeSignedFieldWithNormals: 0.00130315s
Timing for computeSignedDistanceField: 0.326419s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0245882s
Timing for cpatResultsToArrays: 0.00019256s
Timing for cpatDistanceField: 0.320198s
Timing for computeDistanceField: 0.320207s
Timing for cpatComputeSignedFieldWithNormals: 0.00130505s
Timing for computeSignedDistanceField: 0.323606s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0283451s
Timing for cpatResultsToArrays: 0.00019359s
Timing for cpatDistanceField: 0.308867s
Timing for computeDistanceField: 0.308876s
Timing for cpatComputeSignedFieldWithNormals: 0.00047817s
Timing for computeSignedDistanceField: 0.312159s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0244186s
Timing for cpatResultsToArrays: 0.00019375s
Timing for cpatDistanceField: 0.320358s
Timing for computeDistanceField: 0.320367s
Timing for cpatComputeSignedFieldWithNormals: 0.00030738s
Timing for computeSignedDistanceField: 0.323674s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0242566s
Timing for cpatResultsToArrays: 0.00019376s
Timing for cpatDistanceField: 0.352964s
Timing for computeDistanceField: 0.352972s
Timing for cpatComputeSignedFieldWithNormals: 0.00130229s
Timing for computeSignedDistanceField: 0.356353s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0245538s
Timing for cpatResultsToArrays: 0.00019427s
Timing for cpatDistanceField: 0.344792s
Timing for computeDistanceField: 0.344801s
Timing for cpatComputeSignedFieldWithNormals: 0.000304841s
Timing for computeSignedDistanceField: 0.345905s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0246179s
Timing for cpatResultsToArrays: 0.00019321s
Timing for cpatDistanceField: 0.334468s
Timing for computeDistanceField: 0.334477s
Timing for cpatComputeSignedFieldWithNormals: 0.000342201s
Timing for computeSignedDistanceField: 0.338122s
NUM Triangles: 37578
Timing for Build CPAT Model (build bvh): 0.0315542s
Timing for cpatResultsToArrays: 0.000193811s
Timing for cpatDistanceField: 0.332796s
Timing for computeDistanceField: 0.332805s
Timing for cpatComputeSignedFieldWithNormals: 0.000312471s
Timing for computeSignedDistanceField: 0.335961s
[s.1915438@sl2 aneurysm]$
The constraint file:
The u velocity:
—–HAPPY ENDING—–