A fast and precise DFT wavelet code

Examples of compilations on different architectures

A fast and precise DFT wavelet code
Jump to: navigation, search

Contents

Piz Daint XC50/XC40, CSCS Lugano, Swiss

BigDFT 1.8 - Development Version

Compilation for DeltaTest script

This compilation enables (LG, 18/12/16)

  • Shared library compilation of BigDFT suite modules
  • CUDA Support for Poisson Solver for Exact Exchange computation
  • OpenCL acceleration for BigDFT convolutions
  • Python bindings via gobject-introspection of the BigDFT main routines
  • BigDFT python calculators to be used for Deltatest bechmark

The intel compiler 17.0 has to be unloaded as the optimised version of the convolutions undergoes a compiler optimisation bug. Therefore Intel 16.0 is loaded instead:

> module swap PrgEnv-cray PrgEnv-intel
> module swap intel intel/16.0.3.210
> module load cudatoolkit

The code might be built with the command

<path_to_sources>/Installer.py build

and the rcfile daint-deltatest.rc might be chosen for the compilation.

Compilation with GPU support only

If the (introspected) python bindings are not required then another rcfile might be chosen. The procedure is the same as the "Compilation for DeltaTest script" subsection. Choose the file daint-intel.rc instead.

Summer, CEA Grenoble

BigDFT 1.8 version, Intel Compiler

Compilation was done on 07th January 2016.

From a clean bash or sh shell set the environment variables for MPI and Intel compiler. then export the configure flags variable

module load python/2.7.4 icc mkl
module load ompi
source /opt/intel/composer_xe_2013_sp1.6.214/bin/ifortvars.sh intel64
export BIGDFT_CONFIGURE_FLAGS="FC=mpif90 CC=icc FCFLAGS='-O2 -openmp -I${MKLROOT}/include' '--with-ext-linalg= -Wl,--start-group ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a ${MKLROOT}/lib/intel64/libmkl_intel_thread.a -Wl,--end-group -lpthread -lm'"


To compile the bigdft suite you might now go in a fresh directory and type

python /home/sp2m_l_sim/bigdft/jhbuild.py -f /home/sp2m_l_sim/bigdft-jhbuildrc build bigdft

To run the code, use

mpirun -np <n> <path to bigdft>/install/bin/bigdft

The number of OpenMP threads can be set by

setenv OMP_NUM_THREADS <n>


Compilation of the code with old build system (configure line)

Here we list the compilation of BigDFT that have been performed with the old version of the code. This information might be useful to create a configuration file for the 1.8 series and successive. The configure line might either be passed as arcument to the -C option or written in the associated rcfile.

Cray Titan, Oak Ridge

BigDFT 1.7-dev.26 (LG)

Compilation was performed on 03/04/2013.

Starting from a Clean shell, swap the module related to Portland Group Compilers to the one of Cray (for example). Also remove atp module which conflicts with C linker (of gcc) due to a wrong ordering in ftn script. Also load cudatoolkit module.

> module list

Currently Loaded Modulefiles:
 1) modules/3.2.6.6                         13) csa/3.0.0-1_2.0401.37452.4.50.gem       25) xe-sysroot/4.1.40
 2) craype-network-gemini                   14) dvs/1.8.6_0.9.0-1.0401.1401.1.120       26) atp/1.6.1
 3) xt-asyncpe/5.17                         15) rca/1.0.0-2.0401.38656.2.2.gem          27) PrgEnv-pgi/4.1.40
 4) pgi/12.10.0                             16) audit/1.0.0-1.0401.37969.2.32.gem       28) cray-mpich2/5.5.5
 5) xt-libsci/12.0.00                       17) ccm/2.2.0-1.0401.37254.2.142            29) xtpe-interlagos
 6) udreg/2.3.2-1.0401.5929.3.3.gem         18) configuration/1.0-1.0401.35391.1.2.gem  30) eswrap/1.0.15
 7) ugni/4.0-1.0401.5928.9.5.gem            19) hosts/1.0-1.0401.35364.1.115.gem        31) torque/4.1.4
 8) pmi/4.0.1-1.0000.9421.73.3.gem          20) lbcd/2.1-1.0401.35360.1.2.gem           32) moab/7.1.3
 9) dmapp/3.2.1-1.0401.5983.4.5.gem         21) nodehealth/5.0-1.0401.38460.12.18.gem   33) lustredu/1.2
10) gni-headers/2.1-1.0401.5675.4.4.gem     22) pdsh/2.26-1.0401.37449.1.1.gem          34) DefApps
11) xpmem/0.1-2.0401.36790.4.3.gem          23) shared-root/1.0-1.0401.37253.3.50.gem   35) altd/1.0
12) job/1.5.5-0.1_2.0401.35380.1.10.gem     24) switch/1.0-1.0401.36779.2.72.gem

> module swap PrgEnv-pgi  PrgEnv-cray
> module unload atp
> module load cudatoolkit/5.0.35.102
> module list

Currently Loaded Modulefiles:
 1) modules/3.2.6.6                         13) ccm/2.2.0-1.0401.37254.2.142            25) cce/8.1.4
 2) craype-network-gemini                   14) audit/1.0.0-1.0401.37969.2.32.gem       26) xt-asyncpe/5.17
 3) cray-mpich2/5.5.5                       15) dvs/1.8.6_0.9.0-1.0401.1401.1.120       27) xtpe-interlagos
 4) xe-sysroot/4.1.40                       16) csa/3.0.0-1_2.0401.37452.4.50.gem       28) eswrap/1.0.15
 5) rca/1.0.0-2.0401.38656.2.2.gem          17) job/1.5.5-0.1_2.0401.35380.1.10.gem     29) torque/4.1.4
 6) switch/1.0-1.0401.36779.2.72.gem        18) xpmem/0.1-2.0401.36790.4.3.gem          30) moab/7.1.3
 7) shared-root/1.0-1.0401.37253.3.50.gem   19) gni-headers/2.1-1.0401.5675.4.4.gem     31) lustredu/1.2
 8) pdsh/2.26-1.0401.37449.1.1.gem          20) dmapp/3.2.1-1.0401.5983.4.5.gem         32) DefApps
 9) nodehealth/5.0-1.0401.38460.12.18.gem   21) pmi/4.0.1-1.0000.9421.73.3.gem          33) altd/1.0
10) lbcd/2.1-1.0401.35360.1.2.gem           22) ugni/4.0-1.0401.5928.9.5.gem            34) PrgEnv-cray/4.1.40
11) hosts/1.0-1.0401.35364.1.115.gem        23) udreg/2.3.2-1.0401.5929.3.3.gem         35) cudatoolkit/5.0.35.102
12) configuration/1.0-1.0401.35391.1.2.gem  24) xt-libsci/12.0.00

Then the configuring and compilation can be done:

> <path_to_bigdft>/bigdft-1.7-dev.26/configure 'FC=ftn' 'FCFLAGS=-em -O thread2' \
                --with-blas=no --with-lapack=no --with-ext-linalg=' ' \
                --enable-opencl --enable-cuda-gpu --with-cuda-path=/opt/nvidia/cudatoolkit/5.0.35.102 \
                --with-ocl-path=/opt/nvidia/cudatoolkit/5.0.35.102 \
                LDFLAGS=-L/opt/cray/nvidia/304.47.15-1_1.0401.942.1.1.gem/lib64

...

BigDFT configure summary:
========================

Basics:
 Fortran90 compiler:        ftn
 Fortran90 compiler name:   
 Fortran90 flags:           -em -O thread2
 Fortran77 compiler:        f77
 Fortran77 flags:           -O2
 Linker flags:              -L/opt/cray/nvidia/304.47.15-1_1.0401.942.1.1.gem /lib64 -L/opt/nvidia/cudatoolkit/5.0.35.102/lib64 
                            -L/opt/nvidia/cudatoolkit/5.0.35.102/lib -L$(top_builddir)/libxc-1.1.0/src/.libs -L$(top_builddir)/libABINIT/src 
 Linked libraries:           -lrt 

Build:
 Library ABINIT:            yes
 Library BigDFT:            yes
 Main binaries (bigdft,...):yes
 Minima hopping binary:     no
 atom and pseudo binaries:  no
 Bindings:                  no
 User documentation:        no
 Devel. documentation:      no / no

Options:
 Debug version:             no
 Max. Memory per task (GB): 0
 With MPI:                  yes
  | include dir.:           
  | linker flags:           
  | linked libraries:       
  | MPI_INIT_THREAD avail:  yes
  | MPI2 support:           yes
 With optimised conv.:      yes
 With Cuda GPU conv.:       yes
  | NVidia Cuda Compiler:   /opt/nvidia/cudatoolkit/5.0.35.102/bin/nvcc
  | Cuda flags:             
 With OpenCL support:       yes
 With Intel MIC support:    no
 With dgemmsy support:      no
 With libXC:                yes
  | internal built:         yes
  | include dir.:           -I$(top_builddir)/libxc-1.1.0/src
 With libABINIT:            yes
  | internal built:         yes
  | include dir.:           -I$(top_builddir)/libABINIT/src
 With libS_GPU:             yes
  | internal built:         yes
  | include dir.:           -DCUBLAS_USE_THUNKING -I$(top_srcdir)/S_GPU/src
 With ETSF_IO:              no
  | include dir.:           
 With libarchive:           warn
  | C flags:                
  | linked libraries:       
 With libyaml:              yes
  | internal built:         yes
  | C flags:                -I$(top_srcdir)/yaml-0.1.4/include
  | linked libraries:       $(top_builddir)/yaml-0.1.4/src/.libs/libyaml.a
 With GLib and co.:         no
  | linked libraries:        

Installation paths:
 Source code location:      /ccs/home/luigi/bigdft-1.7-dev.26
 Prefix:                    /usr/local
 Exec prefix:               ${prefix}
 Binaries:                  ${exec_prefix}/bin
 Static libraries:          ${exec_prefix}/lib
 Fortran modules:           ${prefix}/include/
 Documentation:             ${datarootdir}/doc/${PACKAGE_TARNAME}

Warnings:
 libarchive:                disabled by tests

> make -j4
 ...

=======================================================
Type 'make check' in the directory tests      
    Use the environment variable run_parallel
    ex: export run_parallel='mpirun -np 2'  
    Use the environment variable run_ocl
    ex: export run_ocl='on' to use OpenCL acceleration
    use run_ocl='CPU' or 'ACC' to force use of hardware
    different than GPU
    and the environment variable ocl_platform
    ex: export ocl_platform='NVIDIA'
    and the environment variable ocl_devices
    ex: export ocl_devices='K20'
=======================================================

[...] some warnings related to libyaml C library compilation (unimportant)
> 

After this compilation the executables are created. At present, I am verifying whether the new K20 hardware exhibit performace loss on memory transfers with respect to older Fermi GPUs.


WITHOUT GPUs:

> module swap PrgEnv-pgi  PrgEnv-cray
> module unload atp
> <path_to_bigdft>/bigdft-1.7-dev.28/configure 'FC=ftn' 'FCFLAGS=-em \
>                      -O thread2' '--with-blas=no' '--with-lapack=no' \
>                      '--with-ext-linalg= '

Bull's Curie, TGCC, France

BigDFT 1.7.7 (LG and PP)

Compilation was done by the software support of the TGCC, the 16th March 2016.

Cuda and Intel compiler are used. Se as an example what you should have (some module version might change due to upgrades):

$ module load cuda
$ module list
Currently Loaded Modulefiles:
1) ccc                       4) licsrv/intel              7) fortran/intel/14.0.3.174 10) intel/14.0.3.174         
2) datadir/own               5) c/intel/14.0.3.174        8) mkl/14.0.3.174           11) mpi/bullxmpi/1.2.8.4     
3) dfldatadir/own            6) c++/intel/14.0.3.174      9) idb/14.0.3.174           12) cuda/6.5.14              


OpenMP support is activated. OpenCL and Cuda are enabled by configuration. Multi threaded MKL are used with Scalapack. We also enable SSE implementation of a dgemm for symetrical matrices.

The paths for libraries are stored in environment variables, obtained with:

module show mkl

Then one can configure the package:

./configure FC=mpif90 CC=icc CXX=icpc \
            FCFLAGS="-O2 -openmp" LDFLAGS="-openmp" --enable-dgemmsy \
            --enable-cuda-gpu --with-cuda-path="$CUDA_ROOT" \
            --enable-opencl --with-ocl-path="$CUDA_ROOT" \
            --with-ext-linalg="$MKL_LIBS_MT"

There is a bug in the build system of S_GPU. Using "-openmp" perturbates nvcc. After issuing make, it will stop in S_GPU/src. To circumvent:

cd S_GPU
make clean
make LDFLAGS=""
cd -

Then, one can resume make in BigDFT directory.

Turing BG/Q, France

BigDFT 1.7.6.1 (LG)

Compilation was done the 01th December 2014.

$ module load lapack

OpenMP support is activated.

./configure 'FC=mpixlf90_r' 'FCFLAGS=-g -O2 -qxlf90=autodealloc \
 -qessl -qsmp=omp -qtune=450 -qnoipa -qnohot -qstrict' 'CC=gcc'

Juqueen BG/Q, Germany

BigDFT 1.7.0.28 (TD from H. Servat)

Compilation was done the 2nd July 2013.

./configure FCFLAGS='-qsmp=omp -g -O2 -qxlf90=autodealloc -qnoipa -qnohot -qstrict' \
CFLAGS='-qsmp=omp -g -O2 -qnohot' CXXFLAGS='-qsmp=omp -g -O2 -qnohot'
FC=/bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90_r CXX=/usr/bin/bgxlC   CC=/usr/bin/bgxlc \
--with-ext-linalg='-L/bgsys/local/scalapack/2.0.2_simd/lib -L/opt/ibmmath/lib64 -L/bgsys/local/scalapack/2.0.2_simd/lib -L/bgsys/local/lapack/3.4.2_simd/lib -lscalapack -llapack -lesslbg' \
--disable-internal-libsgpu --disable-cuda-gpu --disable-opencl \
--disable-intel-mic --disable-shared

Cray XE6 "Rosa", CSCS

BigDFT 1.7.0.91

Compilation was performed on 17/11/2013.

Install the PrgEnv-Cray module from a clean shell. The problem with the atp library can be solved with static linking. In addition, also python module should be loaded in order to build the PyYaml library for python. This operation is useful for logfile analysis and for make check, and does not have any impact on the bigdft executable. It is also important to load the gcc module as this compiler is used for the C section.

> module list
Currently Loaded Modulefiles:
 1) eswrap/1.0.19-1.010101.279.0   3) xt-mpich2/5.6.0
 2) xtpe-interlagos                4) slurm
> mmodule load PrgEnv-cray python/2.7.2 gcc
> module list

Currently Loaded Modulefiles:

 1) eswrap/1.0.19-1.010101.279.0          9) acml/5.2.0                           17) xe-sysroot/4.0.46
 2) xtpe-interlagos                      10) xt-libsci/12.0.00                    18) rca/1.0.0-2.0400.31553.3.58.gem
 3) xt-mpich2/5.6.0                      11) udreg/2.3.1-1.0400.4264.3.1.gem      19) xt-asyncpe/5.16
 4) slurm                                12) ugni/2.3-1.0400.4374.4.88.gem        20) atp/1.6.0
 5) xtpe-network-gemini                  13) pmi/4.0.0-1.0000.9282.69.4.gem       21) PrgEnv-cray/4.0.46
 6) cce/8.1.2                            14) dmapp/3.2.1-1.0400.4255.2.159.gem    22) python/2.7.2
 7) totalview-support/1.1.5              15) gni-headers/2.1-1.0400.4351.3.1.gem
 8) xt-totalview/8.12.0                  16) xpmem/0.1-2.0400.31280.3.1.gem

Then configure with multithreading enabled. You should obtain a successful configuring.

> <path_to_bigdft>/configure 'FC=ftn' 'FCFLAGS=-em -O thread2' \
                    '--with-blas=-libsci_cray_mp' '--with-lapack=no' '--with-ext-linalg= ' '--enable-minima-hopping' CFLAGS="-O2 -static -fPIC"
...

Compilation can then start, and ends successfully.

> make 
...
========================================================
To run the tests, type 'make check' (might take a while)
    Use the environment variable run_parallel
        ex: export run_parallel='mpirun -np 2'  
========================================================
make[2]: Leaving directory ...
>

Cray XK7 "Todi", CSCS

BigDFT 1.7.1.11

Compilation was performed on 08/02/2014.

This machine is similar to the Titan supercomputer in Oak Ridge. The environment is slightly different. The clean shell starts with the Cray programming environment already loaded, therefore there is only need to load the cudatoolkit module for GPU support:

> module load cudatoolkit


For the configuring a complicated set of options has to be performed as the atp linking libraries included in ftn compile have been entered in the wrong ordering (WARNING: do not insert carriage returns inside the quotes):

> <path_to_bigdft>/configure 'FC=ftn' 'FCFLAGS=-em -O thread2' \
                --with-blas=no --with-lapack=no --with-ext-linalg=' ' \
                --enable-opencl --enable-cuda-gpu \
                --with-cuda-path=/opt/nvidia/cudatoolkit/default \
                --with-ocl-path=/opt/nvidia/cudatoolkit/default \
                LIBS=-L/opt/cray/nvidia/default/lib64 \
                CFLAGS=-O2 FCLIBS=' '

Then compilation will go through smoothly. Pay attention that at the configure summary the OpenCL part should been activated.


Then make can be given. Like Titan machine, when using cutatoolkit of 5.0 version the GPU usage s strongly affected by the limitation that no more than one MPI task per node can be used due to configuration in exclusive mode decided by Cray administrators. This limitation is not present for example in Curie machine. In any case, the executable which is generated is fully functional.

BigDFT 1.7.0.31 - Cpu only

The code can be also compiled without the GPU support on the same machine. However, it is important to load the same modules as before otherwise a error is produced in the libxc configure due to craypat (most likely a problem of LD_LIBRARY_PATH of the default loaded modules). Therefore

> module load cudatoolkit
> module load gcc/4.4.4

together with the configure line

> <path_to_bigdft>/configure 'FC=ftn' 'FCFLAGS=-em -O thread2' '--with-blas=no' '--with-lapack=no' '--with-ext-linalg= ' \
    LIBS='-lhugetlbfs -L/apps/todi/mpip/3.4/cce_819/lib -lmpiP -L/apps/todi/unwind/1.1/gnu_434/lib -lunwind -lbfd -liberty -lz -ldl -L/opt/nvidia/cudatoolkit/5.5.20-1.0402.7700.8.1/lib64 -L/opt/nvidia/cudatoolkit/5.5.20-1.0402.7700.8.1/extras/CUPTI/lib64 -lcupti -lcudart -L/opt/cray/nvidia/default/lib64 -lcuda -L/opt/cray/udreg/2.3.2-1.0402.7546.1.5.gem/lib64 -L/opt/cray/ugni/5.0-1.0402.7551.1.10.gem/lib64 -L/opt/cray/dmapp/4.0.1-1.0402.7784.4.1.gem/lib64 -L/opt/cray/xpmem/0.1-2.0402.45248.1.5.gem/lib64 -L/opt/cray/pmi/4.0.1-1.0000.9753.86.3.gem/lib64 -L/opt/cray/rca/1.0.0-2.0402.47290.7.1.gem/lib64 -lrca -L/opt/cray/atp/1.7.0/lib/ -lAtpSigHandler -lAtpSigHCommData -L/opt/cray/cce/8.2.0/craylibs/x86-64 -L/opt/gcc/4.4.4/snos/lib64 -L/opt/cray/mpt/6.1.0/gni/mpich2-cray/81/lib -L/opt/cray/libsci/12.1.01/cray/81/interlagos/lib -lscicpp_cray -lsci_cray_mp -lmpichf90_cray -lmpich_cray -lmpl -lpmi -lalpslli -lalpsutil -lpthread -L/usr/lib/alps -L/opt/gcc/4.4.4/snos/lib/gcc/x86_64-suse-linux/4.4.4 -L/opt/cray/cce/8.2.0/cray-binutils/x86_64-unknown-linux-gnu/lib -L//usr/lib64 -lpgas-dmapp -lxpmem -lu -ldmapp -lugni -ludreg -lfi -lstdc++ -lf -lmodules -lcraymath -lgfortran -lquadmath -lcraymp -lcsup -lrt -ltcmalloc_minimal -lm'


will make the code compile.

Cray XC30 "Piz Daint", CSCS

BigDFT 1.7.1.20

Compilation was done the 16/06/2014. Since this machine has Intel SandyBridge CPUs, we will show the compilation instructions both for Cray and Intel Programming Environment.

Cray Compiler

Load the Cray programming environment. Cuda toolkit module can also be used to benefit of GPU acceleration.

> module load PrgEnv-cray
> module load cudatoolkit

Type module list to check whether the modules were correctly loaded:

Currently Loaded Modulefiles:
 1) modules/3.2.6.7                       10) udreg/2.3.2-1.0501.7914.1.13.ari      19) alps/5.1.1-2.0501.8507.1.1.ari
 2) eswrap/1.1.0-1.010400.915.0           11) ugni/5.0-1.0501.8253.10.22.ari        20) rca/1.0.0-2.0501.48090.7.46.ari
 3) switch/1.0-1.0501.47124.1.93.ari      12) pmi/5.0.2-1.0000.9906.117.2.ari       21) atp/1.7.1
 4) craype-network-aries                  13) dmapp/7.0.1-1.0501.8315.8.4.ari       22) PrgEnv-cray/5.1.29
 5) craype/2.05                           14) gni-headers/3.0-1.0501.8317.12.1.ari  23) craype-sandybridge
 6) cce/8.2.4                             15) xpmem/0.1-2.0501.48424.3.3.ari        24) slurm
 7) totalview-support/1.1.4               16) job/1.5.5-0.1_2.0501.48066.2.43.ari   25) cray-mpich/6.2.2
 8) totalview/8.11.0                      17) csa/3.0.0-1_2.0501.47112.1.91.ari     26) cudatoolkit/5.5.20-1.0501.7945.8.2
 9) cray-libsci/12.1.3                    18) dvs/2.4_0.9.0-1.0501.1672.2.122.ari


To compile with OpenMP support use the following configure line:

> <path_to_bigdft>/configure 'FC=ftn' 'FCFLAGS=-em -O thread2' '--with-blas=no' '--with-lapack=no' '--with-ext-linalg= ' \
'--enable-opencl' '--enable-cuda-gpu' '--with-cuda-path=/opt/nvidia/cudatoolkit/default' \
'--with-ocl-path=/opt/nvidia/cudatoolkit/default' 'LIBS=-L/opt/cray/nvidia/default/lib64' 'CFLAGS=-O2' 'FCLIBS= '

And then you can compile by typing make.

Intel Compiler

From a clean shell, the Intel programming environment can be loaded instead of the default Cray environment

> module swap PrgEnv-cray PrgEnv-intel
> module load cudatoolkit
> module list
Currently Loaded Modulefiles:
 1) modules/3.2.6.7                       10) cray-libsci/12.1.3                    19) job/1.5.5-0.1_2.0501.48066.2.43.ari
 2) eswrap/1.1.0-1.010400.915.0           11) totalview-support/1.1.4               20) csa/3.0.0-1_2.0501.47112.1.91.ari
 3) switch/1.0-1.0501.47124.1.93.ari      12) totalview/8.11.0                      21) dvs/2.4_0.9.0-1.0501.1672.2.122.ari
 4) craype-network-aries                  13) udreg/2.3.2-1.0501.7914.1.13.ari      22) alps/5.1.1-2.0501.8507.1.1.ari
 5) craype/2.05                           14) ugni/5.0-1.0501.8253.10.22.ari        23) rca/1.0.0-2.0501.48090.7.46.ari
 6) craype-sandybridge                    15) pmi/5.0.2-1.0000.9906.117.2.ari       24) atp/1.7.1
 7) slurm                                 16) dmapp/7.0.1-1.0501.8315.8.4.ari       25) PrgEnv-intel/5.1.29
 8) cray-mpich/6.2.2                      17) gni-headers/3.0-1.0501.8317.12.1.ari  26) cudatoolkit/5.5.20-1.0501.7945.8.2
 9) intel/14.0.1.106                      18) xpmem/0.1-2.0501.48424.3.3.ari

The configure line, to compile with mkl libraries, requres specific instructions given by mkl line advisor:

> <path to branch>/configure 'FC=ftn' 'CC=cc' 'CXX=icpc' 'FCFLAGS=-I/opt/intel/14.0.1.106/composer_xe_2013_sp1.1.106/mkl/include -O2 -openmp' \
'--with-ext-linalg=-L/opt/intel/14.0.1.106/composer_xe_2013_sp1.1.106/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lpthread -lm' \
'--enable-opencl' '--enable-cuda-gpu' '--with-cuda-path=/opt/nvidia/cudatoolkit/default/' \
'--with-ocl-path=/opt/nvidia/cudatoolkit/default' 'LIBS=-L/opt/cray/nvidia/default/lib64 -L/opt/nvidia/cudatoolkit/default/lib64' 'CFLAGS=-O2' 'FCLIBS= '

Then the code can be compiled.

BigDFT 1.7.6.3

Intel Compiler (Dec. 19 2014)

> module swap PrgEnv-cray PrgEnv-intel
> module load cudatoolkit
> module list
  Currently Loaded Modulefiles:
 1) modules/3.2.6.7                        7) slurm                                 13) totalview/8.11.0                      19) xpmem/0.1-2.0501.48424.3.3.ari        25) atp/1.7.5
 2) eswrap/1.1.0-1.010400.915.0            8) cray-mpich/7.0.4                      14) udreg/2.3.2-1.0501.7914.1.13.ari      20) job/1.5.5-0.1_2.0501.48066.2.43.ari   26) PrgEnv-intel/5.1.29
 3) switch/1.0-1.0501.47124.1.93.ari       9) ddt/4.3rc7                            15) ugni/5.0-1.0501.8253.10.22.ari        21) csa/3.0.0-1_2.0501.47112.1.91.ari     27) cudatoolkit/5.5.20-1.0501.7945.8.2
 4) craype-network-aries                  10) intel/14.0.2.144                      16) pmi/5.0.5-1.0000.10300.134.8.ari      22) dvs/2.4_0.9.0-1.0501.1672.2.122.ari
 5) craype/2.2.1                          11) cray-libsci/13.0.1                    17) dmapp/7.0.1-1.0501.8315.8.4.ari       23) alps/5.1.1-2.0501.8713.1.1.ari
 6) craype-sandybridge                    12) totalview-support/1.1.4               18) gni-headers/3.0-1.0501.8317.12.1.ari  24) rca/1.0.0-2.0501.48090.7.46.ari
<path to branch>/configure 'FC=ftn' 'CC=cc' 'CXX=icpc' 'FCFLAGS=-I/opt/intel/14.0.1.106/composer_xe_2013_sp1.1.106/mkl/include -O2 -openmp  -xAVX -mavx -march=corei7-avx -mtune=corei7-avx -simd'\
'--with-ext-linalg=-mkl=parallel' '--enable-opencl' '--enable-cuda-gpu' '--with-cuda-path=/opt/nvidia/cudatoolkit/default/'\
'--with-ocl-path=/opt/nvidia/cudatoolkit/default' 'LIBS=-L/opt/cray/nvidia/default/lib64 -L/opt/nvidia/cudatoolkit/default/lib64' 'CFLAGS=-O2 ' 'FCLIBS= '

BigDFT 1.8a.5

Intel Compiler (compilation performed Apr 5, 2016 (LG,NN)

> module swap PrgEnv-cray PrgEnv-intel
> module load cudatoolkit
 <path to branch>/configure  ~/bigdft-1.8a.5/configure 'FC=ftn' 'CC=cc' \
'CXX=icpc' "FCFLAGS=-I${MKLROOT}/include -O2 -openmp" \
"--with-ext-linalg=-L${MKLROOT}/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lpthread -lm" \
 '--enable-opencl' '--enable-cuda-gpu' "--with-cuda-path=${CUDATOOLKIT_HOME}" \
"--with-ocl-path=${CUDATOOLKIT_HOME}" \
"LIBS=-L${CUDATOOLKIT_HOME}/lib64 -lnvToolsExt" 'CFLAGS=-O2' 'FCLIBS= '


Cray Compiler (compilation performed Apr 5, 201 (LG,NN)

> module load cudatoolkit
<path to branch>/configure 'FC=ftn' 'FCFLAGS=-em -O thread2' '--with-blas=no' \
'--with-lapack=no' '--with-ext-linalg= ' '--enable-opencl' \
'CFLAGS=-O2' 'FCLIBS= ' '--enable-opencl' '--enable-cuda-gpu' \
"--with-cuda-path=${CUDATOOLKIT_HOME}" \
"--with-ocl-path=${CUDATOOLKIT_HOME}" \
"LIBS=-L${CUDATOOLKIT_HOME}/lib64 -lnvToolsExt"

Cray XC40 "Piz Dora", CSCS

BigDFT 1.7.6.3

Intel Compiler (Dec. 22 2014)

> module swap PrgEnv-cray PrgEnv-intel
> <path to bigdft>/configure 'FC=ftn' 'CC=cc' 'CXX=icpc' 'FCFLAGS=-O2 -openmp -march=core-avx2' '--with-ext-linalg=-mkl=parallel' 'CFLAGS=-O2'  'FCLIBS= '

Stampede Supercomputer, TACC, Texas

BigDFT 1.7.0.31 (16/07/2013 - LG)

Load the Intel 13 module

> module load intel/13.1.1.163 cuda
> module unload mvapich2
> module load impi
> module list
Currently Loaded Modules:
 1) TACC    2) TACC-paths    3) Linux    4) cluster    5) cluster-paths    6) intel/13.1.1.163    7) cuda/5.0    8) impi/4.1.0.030


Then the configure line:

> <path to bigdft>/configure FC=mpif90 FCFLAGS='-O2 -openmp' --with-blas=no --with-lapack=no \
             --with-ext-linalg='-lmkl_scalapack_lp64 -lmkl_intel_lp64 \
             -lmkl_intel_thread -lmkl_core -lmkl_blacs_intelmpi_lp64 -liomp5 -lpthread -lm' \
             --enable-opencl CC=icc CFLAGS='-O2 -openmp' \
             --with-ext-linalg-path=-L/opt/apps/intel/13/composer_xe_2013.3.163/mkl/lib/intel64

Then the code can be compiled by typing make.

Vesta BG/Q, Argonne National Laboratory

BigDFT 1.7.7 (LR)

Compilation was done the 25th September 2015.

The default software environment should be modified for example using the following .soft file:

@default
+utility_paths
+mpiwrapper-xl-autoperf
@ibm-compilers-2015-02

The code should then be configured as follows:

./configure '--prefix=/home/ratcliff/install/local' '--build=powerpc64-unknown-linux-gnu' \
  '--with-ext-linalg-path=-L/soft/libraries/essl/current/lib64 \
  -L/soft/libraries/alcf/current/xl/SCALAPACK/lib' \
  '--with-ext-linalg=-L/soft/libraries/alcf/current/xl/LAPACK/lib/ \
  -L/soft/libraries/essl/current/lib64 -L/soft/libraries/alcf/current/xl/SCALAPACK/lib \
  -llapack -lscalapack -lesslsmpbg -Wl,--allow-multiple-definition ' \
  '--with-blas-path=-L/soft/libraries/essl/current/lib64' \
  'CFLAGS= -g -O3 ' 'CC=mpixlc' 'CXX=mpixlcxx' \
  'FC=mpixlf90' 'build_alias=powerpc64-unknown-linux-gnu' \
  'LIBS=-L. -L/soft/compilers/ibmcmp-feb2013/xlf/bg/14.1/bglib64 -lxlf90_r \
  -lxlopt -lxl -L/soft/compilers/ibmcmp-feb2013/xlmass/bg/7.3/bglib64 -lmass  \
  -L/soft/compilers/ibmcmp-feb2013/xlsmp/bg/3.1/bglib64 -lxlsmp \
  -lgfortran -lpthread -ldl ' 'CPPFLAGS=-I.' 'FCFLAGS= -g -O3 \
  -qsmp=omp  -qarch=qp -qtune=qp -qcache=auto -qunroll=auto -qfree=f90 -qxlf90=autodealloc \
   -qessl -qstrict -qmaxmem=-1 -qsimd=noauto -qnoipa -qnohot'

The file libxc-2.0.x/src/libxc.f90 has to be manually deleted as it must not exist (see also this bug report: https://bugs.launchpad.net/bigdft/+bug/1365494)

The code can then be compiled by typing make

Theta Cray/Intel KNL, Argonne National Laboratory

BigDFT 1.8.1 (AVM)

Compilation was done the 23th May 2017.

Compilation using Intel 17 Update 2 with Cray wrappers and libsci.

Currently Loaded Modulefiles:

 1) modules/3.2.10.5                              6) cray-libsci/16.11.1                          11) gni-headers/5.0.11-2.2                       16) rca/2.1.6_g2c60fbf-2.265
 2) eproxy/2.0.14-4.3                             7) udreg/2.3.2-7.54                             12) xpmem/2.1.1_gf9c9084-2.38                    17) atp/2.1.0
 3) intel/17.0.2.174                              8) ugni/6.0.15-2.2                              13) job/2.1.1_gc1ad964-2.175                     18) PrgEnv-intel/6.0.3
 4) craype-network-aries                          9) pmi/5.0.11                                   14) dvs/2.7_2.1.67_geb949b6-1.0000.eb949b6.2.32  19) craype-mic-knl
 5) craype/2.5.9                                 10) dmapp/7.1.1-39.37                            15) alps/6.3.4-2.21                              20) cray-mpich/7.5.3

After the untar of the distribution Tarball, I created a buildtheta directory where I executed:

../bigdft-suite/Installer.py  build bigdft -d -c  'CFLAGS=-O3 -g'  'CC=cc' 'CXX=CC' 'FC=ftn' 'FCFLAGS=-O3 -g -qopenmp  -ip  -ldl -I/usr/include  -I. -I/usr/include -I/home/avazquez/soft.tmp/bigdft/buildtheta/install/include -xMIC-AVX512' '--host=x86_64' '--with-ext-linalg=-g' 


Summer, CEA Grenoble

BigDFT 1.7.0.93, Intel Compiler

Compilation was done on 11th December 2013.

- If csh

From a clean csh shell set the environment variables for MPI and Intel compiler:
setenv LD_LIBRARY_PATH /opt/softs/openmpi-1.4-intel/lib/
source /opt/intel/composer_xe_2013.1.117/bin/ifortvars.csh intel64

- Else if bash

From a clean bash or sh shell set the environment variables for MPI and Intel compiler:
export LD_LIBRARY_PATH=/opt/softs/openmpi-1.4-intel/lib/
source /opt/intel/composer_xe_2013.1.117/bin/ifortvars.sh intel64

The output of echo $LD_LIBRARY_PATH should now look as follows:

/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/opt/intel/composer_xe_2013.1.117/mpirt/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/../compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/ipp/lib/intel64:/opt/intel/composer_xe_2013.1.117/compiler/lib/intel64:/opt/intel/composer_xe_2013.1.117/mkl/lib/intel64:/opt/intel/composer_xe_2013.1.117/tbb/lib/intel64:/opt/softs/openmpi-1.4-intel/lib/

- If binaries kept in the source folder (<path to bigdft> == <path to builddir>/src):

Do configure with the following line:
<path to bigdft>/configure FC=/opt/softs/openmpi-1.4-intel/bin/mpif90 CC=icc FCFLAGS="-O2 -openmp" --with-ext-linalg="-Wl,-rpath,/opt/softs/intel/mkl/10.2.2.025/lib/em64t -L/opt/softs/intel/mkl/10.2.2.025/lib/em64t -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lmkl_intel_lp64 -lmkl_lapack -lmkl_sequential -lmkl_core"

- Else if binaries in an installation folder "<path to bigdft>":

Add the following option to the previous configure command:
--prefix="<path to bigdft>"

Now the code can be compiled by typing make.


BC2, Basel

BigDFT 1.8.0

module load Python/2.7.11-iomkl-6.6.2
<path_to_bigdft>/Installer.py build -f maia.rc

Maia, Basel

BigDFT 1.8.0 (LG)

module load iomkl Autotools/20150215
<path_to_bigdft>/Installer.py build

Then choose the configuration file maia.rc

BigDFT 1.7.5 with Intel compilers (icc,icpc,ifort) linked to MKL (BS)

Compilation was performed on July 22, 2014; April 10, 2015

module list
Currently Loaded Modules:
  1) autotools/1.10.3   2) intel/14.0.0.080   3) openmpi/intel/1.6.5   4) cuda/4.2.9

Configure line:

 <path_to_bigdft>/bigdft-1.7.5/configure FC=mpif90 FCFLAGS="-O2 -openmp" -with-ext-linalg="-mkl=parallel" -with-ext-blas="-mkl=parallel"

Now the code can be compiled by typing make. To run the code, use

 mpirun -np <n> <path to bigdft>/bigdft

BigDFT 1.7.5.13 with gnu compilers (gcc,g++,gfortran) linked to MKL (BS)

Compilation was performed on Nov. 5, 2014.

module list
Currently Loaded Modules:
  1) openmpi/gnu/1.6.5   2) autotools/1.10.3

Configure line:

 <path_to_bigdft>/bigdft-1.7.5/configure FC=mpif90 FCFLAGS="-O2 -fopenmp -m64 -I/opt/intel/composer_xe_2013_sp1.0.080/mkl/include" -with-ext-linalg="" LDFLAGS="-Wl,--start-group /opt/intel/composer_xe_2013_sp1.0.080/mkl/lib/intel64/libmkl_gf_lp64.a /opt/intel/composer_xe_2013_sp1.0.080/mkl/lib/intel64/libmkl_core.a /opt/intel/composer_xe_2013_sp1.0.080/mkl/lib/intel64/libmkl_gnu_thread.a -Wl,--end-group -ldl -lpthread -lm"

Now the code can be compiled by typing make. To run the code, use

 mpirun -np <n> <path to bigdft>/bigdft

BigDFT 1.7.5.13 with Intel compilers (icc,icpc,ifort) linked to MKL and ambertools (BS)

Compilation was performed on Nov. 6, 2014.

module list
Currently Loaded Modules:
  1) autotools/1.10.3   2) intel/14.0.0.080   3) openmpi/intel/1.6.5   4) cuda/4.2.9

Configure line:

 <path_to_bigdft>/bigdft-1.7.5/configure FC=mpif90 FCFLAGS="-O2 -openmp" -with-ext-linalg="-mkl=parallel" -with-ambertools="/kernph/schaba00/amber/build/amber14_intel/lib/"

Now the code can be compiled by typing make. To run the code, use

 mpirun -np <n> <path to bigdft>/bigdft


Rothorn, CSCS

This is a big memory machine (more than 2 Tb of RAM).

BigDFT 1.7.5.12 (SS and LG)

Compilation was performed on November 4, 2014. The compilation is done without MPI support as it seems there is no way to compile with MPI (updates would be welcome).

From a clean shell, load inter 14.0 compiler suite and the python headers:

module load intel/14.0.1.106 python

the intel module should be enough:

> module list
Currently Loaded Modulefiles:
  1) slurm              2) intel/14.0.1.106 3) python/2.7.8

Configure line:

 <path_to_bigdft>/configure 'FC=ifort' 'CC=icc' 'CXX=icpc' 'FCFLAGS=-I/include -C -g -traceback -openmp' \
 '--with-ext-linalg=-mkl=parallel' '--with-blas=no' '--with-lapack=no' '--with-blacs=no' '--with-scalapack=no' \
'CFLAGS=-O2 -openmp -dynamic' --disable-mpi

Now the code can be compiled by typing make. To run the code do *not* use mpirun as MPI support is disabled with this compilation.

OCCIGEN, Cines

Recent Machine installed at CINES in Monpellier and available for GENCI projects.

BigDFT 1.7.6.6 (SM and LG)

> module load intel intelmpi mkl python
> module list
Currently Loaded Modulefiles:
 1) /opt/modules/modulefiles/oscar-modules/1.0.3   4) fortran/intel/.15.0.0.090                      7) intel/15.0.0.090
 2) c/intel/.15.0.0.090                            5) mkl/15.0.0.090                                 8) intelmpi/5.0.1.035
 3) c++/intel/.15.0.0.090                          6) idb/15.0.0.090                                 9) python/2.7.8

The configuration can be done with usual multithreaded instructions:

> <path_to_sources>/configure FC=mpif90 FCFLAGS="-O2 -openmp" -with-ext-linalg="-mkl=parallel"

and the compilation can be done by typing make.

Personal tools