... | ... | @@ -150,248 +150,5 @@ $ |
|
|
|
|
|
It can take some time. Some script have been developped to speed up things on some clusters, see [Machine Specific Stuff](Building#machine-specific-stuff) section.
|
|
|
|
|
|
## Machine specific stuff
|
|
|
## [Machine specific stuff](User's-Guide/Installation/Machine specific stuff)
|
|
|
|
|
|
Some cluster specific scripts have been developed that allows to run the tests in parallel. These scripts will automatically dispatch the tests on the cluster and track the results. From the build directory:
|
|
|
|
|
|
* Licallo: `./tools/dev/licTest.sh`
|
|
|
* Occigen: `./tools/dev/occTest.sh`
|
|
|
* Aurora: `./tools/dev/lunarcTest.sh`
|
|
|
|
|
|
refer to those script help for more details.
|
|
|
|
|
|
### Occigen (CINES)
|
|
|
|
|
|
* **Environment**
|
|
|
You need load the environment that way in order to workaround an *occigen* specific environment dependency issue:
|
|
|
```
|
|
|
module purge
|
|
|
module load intel/17.2 python/3.6.3
|
|
|
module rm intel
|
|
|
module load intel/18.0 openmpi/intel/2.0.1 boost/1.65.1
|
|
|
module rm intel
|
|
|
module load intel/18.1 cmake/3.9.0
|
|
|
module load hdf5-seq/1.8.17
|
|
|
module load git/2.10.2
|
|
|
```
|
|
|
* **Git LFS**
|
|
|
You need to explicitly load the git module in order to have lfs.
|
|
|
|
|
|
* **CC and CXX**
|
|
|
Note that on *occigen*, loading the compiler module does not set the `CC` and `CXX` environment variables. So you need to:
|
|
|
```
|
|
|
$ export CC=icc
|
|
|
$ export CXX=icpc
|
|
|
```
|
|
|
|
|
|
Once this environment is loaded, you just need to run `cmake`:
|
|
|
```
|
|
|
[elega@login0:~/fargOCA/build]$ cmake [options] ..
|
|
|
```
|
|
|
* **Testing** platform specific scripts allows you to run the test in parallel
|
|
|
* `<builddir>/tools/dev/occTest.sh` dispatch the tests on the cluster and track the result. You can interrupt the monitoring phase and resume it later with...
|
|
|
* `<builddir>/tools/dev/occTrackTest.sh` track the dispatched tests.
|
|
|
|
|
|
refer to those script help (`-h`) option for more details.
|
|
|
|
|
|
### Licallo (O.C.A.)
|
|
|
|
|
|
#### Environment
|
|
|
You need load the environment that way in order to workaround an *occigen* specific environment dependency issue:
|
|
|
```
|
|
|
module purge
|
|
|
module load userspace/OCA
|
|
|
module load python/3.6.3_anaconda3
|
|
|
module load cmake/3.9.4
|
|
|
module load scorep/4.0
|
|
|
module load cube/gui/4.4
|
|
|
module load scalasca/2.4
|
|
|
module load intel/mkl/64/2019.4
|
|
|
module load intel/mpi/64/2019.4
|
|
|
module load intel/compiler/64/2019.4
|
|
|
module load intel/tbb/64/2019.4
|
|
|
module load boost-intel19/1.68.0
|
|
|
module load hdf5-intel2019/1.8.20-seq
|
|
|
module load git
|
|
|
```
|
|
|
#### Configuration
|
|
|
|
|
|
Once this environment is loaded, you just need to run `cmake`:
|
|
|
```
|
|
|
[elega@login0:~/fargOCA/build]$ cmake [options] ..
|
|
|
```
|
|
|
#### Build
|
|
|
As usual
|
|
|
|
|
|
#### Testing
|
|
|
|
|
|
Platform specific scripts allows you to run the test in parallel
|
|
|
* `<builddir>/tools/dev/licTest.sh` dispatch the tests on the cluster and track the result. You can interrupt the monitoring phase and resume it later with...
|
|
|
* `<builddir>/tools/dev/licTrackTest.sh` track the dispatched tests.
|
|
|
|
|
|
refer to those script help (`-h`) option for more details.
|
|
|
#### Build
|
|
|
As usual
|
|
|
|
|
|
### Ada (Idris)
|
|
|
|
|
|
#### Environment
|
|
|
|
|
|
The following environment has been tested
|
|
|
|
|
|
```
|
|
|
[roth005@ada338: bld2]$ more ~/.modules
|
|
|
module purge
|
|
|
module load intel/2018.2
|
|
|
module load boost/1.67.0
|
|
|
module load hdf5/seq/1.8.14
|
|
|
# for some reasons, hdf5 want to link static, we need shared (run path is embedded, so LD_LIBRARY_PATH does not need updating)
|
|
|
export WRAPPER_LDFLAGS="$(h5c++ -show -shlib | sed -e 's/.*-D\w\+//')"
|
|
|
module load python/3.6.1
|
|
|
module load gcc/4.9.4 # only loaded for cmake, which could have been linked static
|
|
|
module load cmake/3.7.2
|
|
|
[roth005@ada338: bld2]$ source ~/.modules
|
|
|
(remove) cmake version 3.7.2
|
|
|
(remove) gcc version 4.9.4
|
|
|
(remove) python version 3.6.1
|
|
|
(remove) hdf5 seq 1.8.14
|
|
|
(remove) boost version 1.67.0
|
|
|
(remove) Intel(R) Parallel Studio XE 2018 Update 2 Cluster Edition for Linux*
|
|
|
(load) Intel(R) Parallel Studio XE 2018 Update 2 Cluster Edition for Linux*
|
|
|
...blah blah blah....
|
|
|
(load) boost version 1.67.0
|
|
|
(load) hdf5 seq 1.8.14
|
|
|
(load) python version 3.6.1
|
|
|
(load) gcc version 4.9.4
|
|
|
(load) cmake version 3.7.2
|
|
|
[roth005@ada338: bld2]$
|
|
|
```
|
|
|
|
|
|
#### Configuration
|
|
|
MPI detect does not work that well, so wee need to provide the MPI wrapper as compilers:
|
|
|
```
|
|
|
[roth005@ada338: bld2]$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=mpiicc -DCMAKE_CXX_COMPILER=mpiicpc ../
|
|
|
-- The CXX compiler identification is Intel 18.0.2.20180210
|
|
|
-- Check for working CXX compiler: /smplocal/pub/Modules/IDRIS/wrappers/mpiicpc
|
|
|
-- Check for working CXX compiler: /smplocal/pub/Modules/IDRIS/wrappers/mpiicpc -- works
|
|
|
...blah blah blah....
|
|
|
-- Boost version: 1.67.0
|
|
|
-- Found the following Boost libraries:
|
|
|
-- mpi
|
|
|
-- program_options
|
|
|
-- chrono
|
|
|
-- filesystem
|
|
|
-- system
|
|
|
-- Configuring done
|
|
|
-- Generating done
|
|
|
-- Build files have been written to: /workgpfs/rech/oth/roth005/fargOCA/bld2
|
|
|
[roth005@ada338: bld2]$
|
|
|
```
|
|
|
#### Build
|
|
|
|
|
|
As usual
|
|
|
|
|
|
#### Test:
|
|
|
We test the unit tests and integration tests (with 4 MPI process) with:
|
|
|
```
|
|
|
<builddir>$./tools/dev/adaTest.sh
|
|
|
```
|
|
|
|
|
|
You can interupt the tracking phase and relaunch it wth:
|
|
|
```
|
|
|
<builddir>$./tools/dev/adaTrackTest.sh
|
|
|
```
|
|
|
### Aurora (Lund)
|
|
|
|
|
|
#### Environment
|
|
|
|
|
|
Currently running with:
|
|
|
```
|
|
|
[alain@aurora1 rel]$ module list
|
|
|
|
|
|
Currently Loaded Modules:
|
|
|
1) GCCcore/8.2.0 7) intel/2019a 13) HDF5/1.10.5 19) GMP/6.1.2 25) expat/2.2.6 31) ipp/2019.4.243
|
|
|
2) binutils/2.31.1 8) bzip2/1.0.6 14) CMake/3.9.1 20) libffi/3.2.1 26) libxml2/2.9.8
|
|
|
3) icc/2019.1.144-GCC-8.2.0-2.31.1 9) zlib/1.2.11 15) ncurses/6.1 21) Python/3.7.2 27) gettext/0.19.8.1
|
|
|
4) ifort/2019.1.144-GCC-8.2.0-2.31.1 10) XZ/5.2.4 16) libreadline/8.0 22) SciPy-bundle/2019.03 28) Perl/5.28.1
|
|
|
5) impi/2018.4.274 11) Boost/1.70.0 17) Tcl/8.6.9 23) h5py/2.9.0 29) git/2.21.0
|
|
|
6) imkl/2019.1.144 12) Szip/2.1.1 18) SQLite/3.27.2 24) cURL/7.63.0 30) git-lfs/2.7.1
|
|
|
|
|
|
|
|
|
[alain@aurora1 rel]$
|
|
|
```
|
|
|
|
|
|
#### Configuration
|
|
|
|
|
|
```
|
|
|
cmake -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icpc ..
|
|
|
```
|
|
|
|
|
|
#### Build
|
|
|
|
|
|
```
|
|
|
make -j 8
|
|
|
```
|
|
|
|
|
|
#### Testing
|
|
|
|
|
|
For ctest, srun needs to know the slurm account, which can be set in the .bashrc:
|
|
|
|
|
|
```
|
|
|
export SLURM_ACCOUNT=snicxxx
|
|
|
export SALLOC_ACCOUNT=$SLURM_ACCOUNT
|
|
|
export SBATCH_ACCOUNT=$SLURM_ACCOUNT
|
|
|
```
|
|
|
|
|
|
```
|
|
|
ctest
|
|
|
```
|
|
|
|
|
|
Tests can also be run in parallel with:
|
|
|
```
|
|
|
$./tools/dev/lunarcTest.sh
|
|
|
```
|
|
|
|
|
|
* **Submitting job**
|
|
|
```
|
|
|
#!/bin/bash
|
|
|
#
|
|
|
#SBATCH --job-name=3D_unperturbed_disk
|
|
|
#SBATCH -A snicxxx
|
|
|
#SBATCH --output=OUT.%J
|
|
|
#
|
|
|
#SBATCH --nodes=2
|
|
|
#SBATCH --ntasks-per-node=2
|
|
|
#SBATCH --time=00:10:00
|
|
|
#SBATCH --mail-type=ALL
|
|
|
#SBATCH --mail-user=michiel@astro.lu.se
|
|
|
|
|
|
# max 10 on aurora
|
|
|
#SBATCH --cpus-per-task=10
|
|
|
#
|
|
|
#
|
|
|
|
|
|
# Set OMP_NUM_THREADS to the same value as -c
|
|
|
# with a fallback in case it isn't set.
|
|
|
# SLURM_CPUS_PER_TASK is set to the value of -c, but only if -c is explicitly set
|
|
|
if [ -n "$SLURM_CPUS_PER_TASK" ]; then
|
|
|
omp_threads=$SLURM_CPUS_PER_TASK
|
|
|
else
|
|
|
omp_threads=1
|
|
|
fi
|
|
|
export OMP_NUM_THREADS=$omp_threads
|
|
|
|
|
|
# Setup environment
|
|
|
|
|
|
date >time_run.dat
|
|
|
echo $SLURM_NTASKS
|
|
|
echo $SLURM_CPUS_PER_TASK
|
|
|
srun --mpi=pmi2 -K1 /home/michiel/fargOCA/build/fargoInit -o out --prepare --config config.info
|
|
|
srun --mpi=pmi2 -K1 /home/michiel/fargOCA/build/fargOCA ./out -p --monitor
|
|
|
date>>time_run.dat
|
|
|
```
|
|
|
|
|
|
* **small stuff**
|
|
|
|
|
|
push only to the current branch
|
|
|
```git config --global push.default simple``` |