Installing GPAW with conda

In short, in a clean environment, everything should work with just five lines:

wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh
bash Miniconda3-py39_4.12.0-Linux-x86_64.sh

Initialize conda. If it is in the .bashch, source it. If not, source “PATHTOCONDA/miniconda3/etc/profile.d/conda.sh”.

conda create --name gpaw22 python=3.10
conda activate gpaw22
conda install -c conda-forge gpaw=*=*openmpi* openmpi=4.1.2 gcc=12.1.0 ucx

For details, see the description below.

1. Install conda – software and environment management system.

Here is the official instruction: docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html

On July 2022, run these:

wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh
bash Miniconda3-py39_4.12.0-Linux-x86_64.sh

If you wish to autostart conda, allow it to write to .bashrc.

P.S. Here are good intro to conda:

2. Create a conda virtual environment:

conda create --name gpaw22 python=3.10

If needed, remove the environment as:

conda remove --name gpaw22 --all

You can check the available environments as:

conda env list

3. Activate the virtual environment.

conda activate gpaw22

4. Install gpaw:

Ensure that no interfering modules and environments are loaded.

To purge modules, execute:

module purge

To check whether some code (like mpirun) has an alternative path, try:

which codename

There should be no mpirun, ase, libxc, numpy, scipy, etc. Otherwise, the installation with conda will most probably fail due to conflicting paths.

4.1. It is safer to install using gpaw22.yml file from vliv/conda directory on FEND:

conda env create -f gpaw22.yml

4.2. Pure installation is simple, but might not work:

conda install -c conda-forge gpaw=*=*openmpi*

Recently, there were problems with openmpi. Try downgrading it to version 4.1.2:

conda install -c conda-forge openmpi=4.1.2

To enable the Rose support, for example, on FEND v4 adn v5, install Ucx:

conda install -c conda-forge ucx

If you get an error about GLIBCXX, try upgrading gcc:

conda install -c conda-forge gcc=12.1.0

4.3. To make a quick check of the installation, run “gpaw -P 2 test” or “gpaw info”.

The installation might fail. In case you succeed, save the yml file as:

conda env export | grep -v "^prefix: " > gpaw22.yml

Now you can use it to install gpaw as:

conda env create -f gpaw22.yml

To properly test the installation install pytest and follow wiki.fysik.dtu.dk/gpaw/devel/testing.html. That might take hours.

conda install -c conda-forge pytest pytest-xdist 

5. If needed, install extra packages within your specific conda environment (gpaw22).

To apply D4 dispersion correction:

conda install -c conda-forge dftd4 dftd4-python

To analyze trajectories:

conda install -c conda-forge mdanalysis

To analyze electronic density (some might not work):

pip install git+https://github.com/funkymunkycool/Cube-Toolz.git
pip install git+https://github.com/theochem/grid.git
pip install git+https://github.com/theochem/denspart.git
pip install pybader
pip install cpmd-cube-tools
conda install -c conda-forge chargemol

To use catlearn:

pip install catlearn

6. Run calculations by adding these lines to the submission script:

Note1: Check the path and change the USERNAME

Note2: Use ucx and export variables.

Note3: You may play with the number of openmp threads.

module purge
source "/groups/kemi/USERNAME/miniconda3/etc/profile.d/conda.sh"
conda activate gpaw22
export OMP_NUM_THREADS=1
export OMPI_MCA_pml="^ucx"
export OMPI_MCA_osc="^ucx"
mpirun gpaw python script.py

7. Use scalapack for speeding-up your calculations.

Add the “parallel” keyword to GPAW calculator:

parallel = {'augment_grids':True,'sl_auto':True}

For more options see wiki.fysik.dtu.dk/gpaw/documentation/parallel_runs/parallel_runs.html#manual-parallel. For example, try ELPA  for LCAO mode. See wiki.fysik.dtu.dk/gpaw/documentation/lcao/lcao.html#notes-on-performance.

parallel = {'augment_grids':True,'sl_auto':True,'use_elpa':True}

8. If needed, add fixes.

To do Bayesian error estimation see doublelayer.eu/vilab/2022/03/30/bayesian-error-estimation-for-rpbe/.

To use MLMin/NEB apply corrections from github.com/SUNCAT-Center/CatLearn/pulls

9. Something worth trying:

gpaw-tools:

github.com/lrgresearch/gpaw-tools/

www.sciencedirect.com/science/article/pii/S0927025622000155

ase-notebook:

github.com/chrisjsewell/ase-notebook

ase-notebook.readthedocs.io/en/latest/

gpaw benchmarking:

github.com/OleHolmNielsen/GPAW-benchmark-2021

github.com/mlouhivu/gpaw-benchmarks

members.cecam.org/storage/presentation/Ask_Hjorth_Larsen-1622631504.pdf

Working abroad

Sometimes for some reason some VPN might not work as one expects. Then one can login to a node with a key.

First, check if you have a public key already. On your own computer in a terminal run:

cat ~/.ssh/id_rsa.pub

If this command worked, then skip the next step. If an error appeared, run this:

ssh-keygen

It will ask for a password. This is the password you give to the key. You can make one without password, in which case it will not ask for a password each time you try to use it.

You must have an access to the cluster somehow to run the following command:

ssh-copy-id <your-username>@cluster

This trick does not work for all machines. So, you might use the established connection to connect to other machines. In my case I wanted to work with some notebooks, so I did:

ssh -N -L localhost:PORT1:localhost:PORT2 <your-username>@cluster

and so it worked.

Installing Gromacs and Lammps

FFTW:

./configure –enable-float –enable-shared –enable-sse2
make -j N
make install

===

Gromacs:

$ tar xvfz gromacs-x.y.z.tar.gz
$ ls
gromacs-x.y.z
$ mkdir build
$ cd build
$ cmake ../gromacs-x.y.z -DCMAKE_INSTALL_PREFIX=/home/yourUser/opt/gromacs.x.y.z -DGMX_CPU_ACCELERATION=SSE2 -DGMX_SIMD=SSE2
$ make -j N
$ make install

===

LAMMPS:

$ git clone git://git.lammps.org/lammps-ro.git LAMMPS
$ make yes-molecule
$ make mpi

===

To the .bashrc add:

#Gromacs

source /home/yourUser/opt/gromacs.x.y.z/bin/GMXRC
#or source /your/installation/prefix/here/bin/GMXRC

#LAMMPS
export LD_LIBRARY_PATH=~/LAMMPS/src:$LD_LIBRARY_PATH
export PATH=~/LAMMPS/src:$PATH

gsissh at ubuntu

To access one of the PRACE supercomputers, I was required to use gsissh. The corresponding gsi-openssh-clients package is not in the standard Ubuntu repositories. I have downloaded it from http://toolkit.globus.org/ftppub/gt5/5.2/stable/packages/deb/ubuntu/12.04/pool/contrib/g/gsi-openssh/.Besides that also globus-proxy-utils is needed. This one was easy to install via apt-get install globus-proxy-utils. Finally, everything got working when the bundle of X509 trusted certificates was downloaded from http://software.ligo.org/gridtools/debian/pool/main/o/osg-ca-certs/. Note, for some reason .globus folder did not appear in my home directory. To use gsissh, I had to create it and then move my user certificate to it with proper correct permissions -rw-------.

P.S. Do not forget to create a proxy certificate (grid-proxy-init) before login to your supercomputer.