4 * [Using deal.II on Mac OS X and Linux via Spack](#using-dealii-on-mac-os-x-and-linux-via-spack)
5 * [Quick installation on the desktop](#quick-installation-on-the-desktop)
6 * [Installation example on a Centos7 cluster (Emmy cluster of RRZE, Erlangen, Germany)](#installation-example-on-a-centos7-cluster-emmy-cluster-of-rrze-erlangen-germany)
7 * [Enabling CUDA](#enabling-cuda)
8 * [Environment Modules](#environment-modules)
9 * [System provided packages](#system-provided-packages)
10 * [Installing GCC](#installing-gcc)
11 * [Using macOS](#using-macos)
12 * [Best practices using Spack:](#best-practices-using-spack)
14 * [Extra options:](#extra-options)
15 * [Compiler flags](#compiler-flags)
16 * [Different versions coexisting:](#different-versions-coexisting)
17 * [Filesystem Views:](#filesystem-views)
18 * [Check before build:](#check-before-build)
19 * [Develop using Spack](#develop-using-spack)
20 * [Keep the stage to run unit tests](#keep-the-stage-to-run-unit-tests)
21 * [Licensed software](#licensed-software)
22 * [Freeze package versions](#freeze-package-versions)
23 * [Known issues using Spack:](#known-issues-using-spack)
25 # Using deal.II on Mac OS X and Linux via Spack
27 The deal.II suite is also available on Spack (https://github.com/spack/spack) -- a flexible package manager developed with High-Performance-Computing in mind. It is intended to let you build for many combinations of compiler, architectures, dependency libraries, and build configurations, all with a friendly, intuitive user interface.
29 For a quick overview of Spack's features, we recommend this short presentation https://tgamblin.github.io/files/Gamblin-Spack-SC15-Talk.pdf
30 or the following videos from lead developers:
31 [Massimiliano Culpo - SPACK: A Package Manager for Supercomputers, Linux and MacOS](https://www.youtube.com/watch?v=Qok-nXfIWfg) and
32 [Todd Gamblin (LLNL) - Managing HPC Software Complexity with Spack](https://www.youtube.com/watch?v=Ie1cZTR09kk).
34 [Spack tutorial 101](http://spack.readthedocs.io/en/latest/tutorial.html) is good place to start as well.
36 Note: Spack is in active development and is in alpha state, thereby below we recommend using the specific snapshot of the code (hash), that was tested on macOS / Ubuntu / CentOS7 with deal.II and is being used to run the complete testsuite on `Unbutu18.04` with `openmpi` and `openblas`. See [Continuous section of CDash](https://cdash.43-1.org/index.php?project=deal.II).
38 ## Quick installation on the desktop
40 Add the following to `~/.bashrc` (or equivalent)
42 export SPACK_ROOT=/path/to/spack
43 export PATH="$SPACK_ROOT/bin:$PATH"
45 `SPACK_ROOT` is the destination where you want Spack to be installed (i.e. `$HOME/spack`).
47 Make sure that `python`, `curl`, `git`, `make`, `g++` and `gfortran` are installed, i.e. by running:
49 sudo apt install python git curl make g++ gfortran
55 git clone https://github.com/spack/spack.git .
57 git reset --hard b4501c458623ed12cfb9a93cb6ebdfc4604b3fe2
59 The last command will reset Spack to the version dated 12.01.2019.
61 **Make sure C/C++/Fortran compilers are in path** (on Ubuntu you need to `sudo apt-get install gfortran`, on macOS you can compile `gcc` with spack, see [below](#installing-gcc), and you have **curl** (`sudo apt-get install curl`) to download packages. Then install the complete deal.II suite
63 spack install --test=root dealii ^suite-sparse@5.2.0
65 Additional option `--test=root` instructs Spack to run quick tests for `dealii`.
67 **DONE**! No extra (preliminary) configuration steps are needed on most Linux distributions. For configuration files, have a look at [xsdk project](https://github.com/xsdk-project/installxSDK/tree/master/platformFiles), [ceed project](https://ceed.exascaleproject.org/ceed-2.0/) or [spack-configs repository](https://github.com/spack/spack-configs).
69 **IMPORTANT:** If you compile deal.II on a cluster, see the next section on how to use externally provided MPI implementation instead.
71 Before configuring your project you need to set `DEAL_II_DIR` by
73 export DEAL_II_DIR=$(spack location -i dealii)
75 You may jump ahead and read [best practices using spack](#best-practices-using-spack). Also a good starting point is [Getting Started Guide](http://spack.readthedocs.io/en/latest/getting_started.html).
78 ## Installation example on a Centos7 cluster (Emmy cluster of RRZE, Erlangen, Germany)
79 In order to use Spack on a cluster, there are two options: (1) you are a sysadmin and you know hardware details and can use Spack to properly configure and build MPI providers (e.g. `openmpi`); (2) you are a user and you need to make Spack work with MPI provided on your cluster. In what follows we consider the latter case.
81 Below is a brief step-by-step instruction to install deal.II on [Emmy cluster](https://www.rrze.fau.de/dienste/arbeiten-rechnen/hpc/systeme/emmy-cluster.shtml#access) of RRZE, Erlangen, Germany. For configuration examples at other HPC clusters see [spack-configs](https://github.com/spack/spack-configs) repository.
85 mkdir $WOODYHOME/spack
89 (1a) If you plan on using `GCC/Clang` as a compiler:
91 # Clone standard version of Spack
92 git clone https://github.com/spack/spack.git $WOODYHOME/spack
93 export PATH=$WOODYHOME/spack/bin:$PATH
95 (1b) If you plan on using `Intel` compilers (this point is expanded upon in step (5)):
97 # Clone modified version of Spack to fit RRZE setup
98 git clone -b pkg/intel_mpi_fix https://github.com/davydden/spack.git .
99 export PATH=$WOODYHOME/spack/bin:$PATH
101 (2) Load the `openmpi` module and let Spack find `GCC` compiler which is also loaded as a dependency. You might get a warning that spack has not detected any new compilers - this only means that the `compilers.yaml` file already has the correct version of `GCC` listed in it.
103 module load openmpi/2.0.2-gcc
106 (3) Add `openmpi` as an external package, along with `python` and a few other self explanatory setting for `deal.ii`. That is done by adding the following to `~/.spack/linux/packages.yaml`:
118 openmpi@2.0.2%gcc@4.8.5: /apps/OpenMPI/2.0.2-gcc/
123 variants: +optflags~python
125 Those paths are the location where external packages can be found (i.e. `<prefix>` instead of `<prefix>/bin` or `<prefix>/lib`). `providers` section essentially tells Spack which packages to use to satisfy virtual dependencies such as `MPI`, `BLAS`, `LAPACK`, `ScaLAPACK`, etc. Here we also limit version of `suite-sparse` to `5.1.0` as we build with `gcc@4.8.5` whereas `5.2.0` requires at least `gcc@4.9`.
127 (4) Now install deal.II: `spack install dealii`.
129 (5) Here is an alternative setup with Intel compilers. Run `module load intel64/18.0up02` followed by `spack compiler find`. You should get the following new entry in `~/.spack/linux/compilers.yaml`:
136 operating_system: centos7
138 cc: /apps/intel/ComposerXE2018/compilers_and_libraries_2018.2.199/linux/bin/intel64/icc
139 cxx: /apps/intel/ComposerXE2018/compilers_and_libraries_2018.2.199/linux/bin/intel64/icpc
140 f77: /apps/intel/ComposerXE2018/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort
141 fc: /apps/intel/ComposerXE2018/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort
145 Edit it to set environment variable (with license) and extra rpaths:
149 INTEL_LICENSE_FILE: 1713@license4
150 extra_rpaths: # take from echo $LIBRARY_PATH
151 - /apps/intel/ComposerXE2018/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
152 - /apps/intel/ComposerXE2018/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
154 Then add to `~/.spack/linux/packages.yaml` paths to `intel-mpi`, `intel-mkl` as well as `cmake` which currently does not build with `Intel`:
163 scalapack: [intel-mkl]
165 version: [2018.2.199]
167 intel-mpi@2018.2.199%intel@18.0.2: /apps/intel/ComposerXE2018/
170 version: [2018.2.199]
172 intel-mkl@2018.2.199%intel@18.0.2: /apps/intel/ComposerXE2018/
177 cmake@3.6.0%intel@18.0.2: /apps/cmake/3.6.0/
182 variants: +optflags~python
184 Finally install `deal.II`
186 spack install -j 20 dealii%intel~assimp~petsc~slepc+mpi^intel-mpi^intel-mkl
188 Note that `%intel` specified the compiler whereas `^intel-mpi` and `^intel-mkl` specified which implementation of MPI and BLAS/LAPACK we want to use. Here we also disabled few other packages that have issues building with `Intel` compilers.
190 See [this discussion](https://groups.google.com/d/msg/spack/NxyNTAZyMQg/Klu2CHR8GQAJ) on more info about using Intel compilers in Spack.
192 Note that in order to install multiple flavors of `intel-mkl` via Spack (i.e. with and without threading), you may have to delete `$HOME/intel/.pset`, see [this comment](https://github.com/spack/spack/issues/9713#issuecomment-436081356).
196 You can build the current development version of `dealii` with CUDA. A possible configuration of `packages.yaml` is
201 variants: +cuda cuda_arch=52
205 where we specified explicitly CUDA architecture as well as version of `cuda` to be used.
207 ## Environment Modules
208 Spack provides some integration with Environment Modules and Dotkit to make it easier to use the packages it installs. For a full description, read http://spack.readthedocs.io/en/latest/getting_started.html#environment-modules
210 To add the support for Environment Modules run
212 spack install environment-modules
214 and then add to `~/.bashrc` (or equivalent)
216 MODULES_HOME=$(spack location --i environment-modules)
217 source ${MODULES_HOME}/init/bash
218 . $SPACK_ROOT/share/spack/setup-env.sh
221 If you install `deal.II` before setting up environment modules,
222 the module files have to be regenerated
224 spack module tcl refresh
232 Now `DEAL_II_DIR` environment variable should be set appropriately and `cmake` executable will be available in path. Keep in mind that `spack load dealii` will also set `LD_LIBRARY_PATH` accordingly, this may or may not be what you need. An alternative is to use `export DEAL_II_DIR=$(spack location -i dealii)`.
234 ## System provided packages
235 Spack is flexible to use both self-compiled and system provided packages.
236 In most cases this is desirable for `MPI`, which is already installed on computational clusters. To configure external packages you need to edit `~/.spack/linux/packages.yaml`. For `openmpi` this could be
242 openmpi@1.8.8%gcc@6.2.0: /opt/openmpi-1.8.8
245 In order to make sure that we use to build packages `1.8.8` version of `openmpi` and not the most recent one (i.e. `2.0.2`), we specified conretization preferences with `version: [1.8.8]`.
247 One can also specify which packages should be used to provide `mpi`, `blas/lapack` , preferred compilers and preferred variants:
251 compiler: [gcc,clang]
260 For more elaborated discussion, see [Configuration Files in Spack](http://spack.readthedocs.io/en/latest/configuration.html).
263 If your system does not provide any Fortran compiler or you want to have the most recent `gcc`,
264 you can install it by
269 Assuming that you configured [Environment Modules](#environment-modules), load `gcc` and let Spack find the newly installed compilers:
274 Now you can install deal.II with `gcc`
276 spack install dealii%gcc
281 If you are on the mac, read the following instructions on [Mixed Toolchains](http://spack.readthedocs.io/en/latest/getting_started.html#mixed-toolchains).
283 Note that XCode 10 does **NOT** install command line tools (`$xcode-select --install`) in `/usr/include` (see [this discussion](https://forums.developer.apple.com/thread/104296)). Therefore prior to installing `gcc` package via Spack, please install `/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg`.
285 Once you have `gcc` installed and edit `~/.spack/compilers.yaml` to use it with Apple's clang, you will be able to install deal.II
287 spack install dealii%clang
290 ## Best practices using Spack:
292 Spack is complicated and flexible package manager primarily aimed at High-Performance-Computing.
293 Below are some examples of using Spack to build and develop deal.II:
296 If you are not sure which options a given package has, simply run
300 The output will contain available versions, variants, dependencies and description:
306 C++ software library providing well-documented tools to build finite
307 element codes for a broad variety of PDEs.
309 Homepage: https://www.dealii.org
312 8.5.1 https://github.com/dealii/dealii/releases/download/v8.5.1/dealii-8.5.1.tar.gz
315 develop [git] https://github.com/dealii/dealii.git
316 8.5.1 https://github.com/dealii/dealii/releases/download/v8.5.1/dealii-8.5.1.tar.gz
317 8.5.0 https://github.com/dealii/dealii/releases/download/v8.5.0/dealii-8.5.0.tar.gz
318 8.4.2 https://github.com/dealii/dealii/releases/download/v8.4.2/dealii-8.4.2.tar.gz
319 8.4.1 https://github.com/dealii/dealii/releases/download/v8.4.1/dealii-8.4.1.tar.gz
320 8.4.0 https://github.com/dealii/dealii/releases/download/v8.4.0/dealii-8.4.0.tar.gz
321 8.3.0 https://github.com/dealii/dealii/releases/download/v8.3.0/dealii-8.3.0.tar.gz
322 8.2.1 https://github.com/dealii/dealii/releases/download/v8.2.1/dealii-8.2.1.tar.gz
323 8.1.0 https://github.com/dealii/dealii/releases/download/v8.1.0/dealii-8.1.0.tar.gz
326 Name [Default] Allowed values Description
329 adol-c [off] True, False Compile with Adol-c
330 arpack [on] True, False Compile with Arpack and
331 PArpack (only with MPI)
332 build_type [DebugRelease] Debug, Release, The build type to build
334 doc [off] True, False Compile with documentation
335 gsl [on] True, False Compile with GSL
336 hdf5 [on] True, False Compile with HDF5 (only with
338 int64 [off] True, False Compile with 64 bit indices
340 metis [on] True, False Compile with Metis
341 mpi [on] True, False Compile with MPI
342 nanoflann [off] True, False Compile with Nanoflann
343 netcdf [on] True, False Compile with Netcdf (only with
345 oce [on] True, False Compile with OCE
346 optflags [off] True, False Compile using additional
348 p4est [on] True, False Compile with P4est (only with
350 petsc [on] True, False Compile with Petsc (only with
352 python [on] True, False Compile with Python bindings
353 slepc [on] True, False Compile with Slepc (only with
355 sundials [off] True, False Compile with Sundials
356 trilinos [on] True, False Compile with Trilinos (only
363 adol-c cmake hdf5 muparser oce slepc trilinos
364 arpack-ng doxygen lapack nanoflann p4set suite-sparse zlib
365 blas graphivz metis netcdf petsc sundials
366 boost gsl mpi netcdf-cxx python tbb
369 adol-c doxygen lapack nanoflann p4est suite-sparse zlib
370 arpack-ng graphviz metis netcdf petsc sundials
371 blas graphviz mpi netcdf-cxx python tbb
372 boost hdf5 muparser oce slepc trilinos
381 A lot of `spack` commands have help, for example
384 usage: spack install [-h] [--only {package,dependencies}] [-j JOBS]
385 [--keep-prefix] [--keep-stage] [--restage] [-n] [-v]
386 [--fake] [-f] [--clean | --dirty] [--run-tests]
387 [--log-format {junit}] [--log-file LOG_FILE]
390 build and install packages
392 positional arguments:
393 package spec of the package to install
396 -h, --help show this help message and exit
397 --only {package,dependencies}
398 select the mode of installation. the default is to
399 install the package along with all its dependencies.
400 alternatively one can decide to install only the
401 package or only the dependencies
402 -j JOBS, --jobs JOBS explicitly set number of make jobs. default is #cpus
403 --keep-prefix don't remove the install prefix if installation fails
404 --keep-stage don't remove the build stage if installation succeeds
405 --restage if a partial install is detected, delete prior state
406 -n, --no-checksum do not check packages against checksum
407 -v, --verbose display verbose build output while installing
408 --fake fake install. just remove prefix and create a fake
410 -f, --file install from file. Read specs to install from .yaml
412 --clean clean environment before installing package
413 --dirty do NOT clean environment before installing
414 --run-tests run package level tests during installation
415 --log-format {junit} format to be used for log files
416 --log-file LOG_FILE filename for the log file. if not passed a default
421 One can specify extra options for packages in the deal.II suite. For example if you want to have boost with `python` module, this can be done by
423 spack install dealii@develop+mpi~python ^boost+python
426 If you want to specify blas/lapack/mpi implementations, this can be done similarly
428 spack install dealii@develop+mpi ^mpich
430 To check which packages implement `mpi` run
434 One can also specify which Blas/Lapack implementation to use. For example to build deal.II suite with `atlas` run
436 spack install dealii ^atlas
440 You can specify compiler flags on the command line as
442 spack install dealii cppflags="-march=native -O3"
444 Note that these flags will be inherited by dependencies such as `petsc`, `trilinos`, etc. Same can be done by declaring these flags in `~/.spack/compilers.yaml`:
449 operating_system: centos6
453 f77: /usr/bin/gfortran
454 fc: /usr/bin/gfortran
462 See this [google forum topic](https://groups.google.com/forum/?fromgroups#!topic/dealii/3Yjy8CBIrgU) for discussion on which flags to use. You can also use `spack install dealii+optflags` to enable extra optimization flags in release build.
464 ### Different versions coexisting:
465 One can easily have slightly different versions of deal.II side-by-side, e.g. to compile development version of deal.II with complex-valued PETSc and `gcc` compiler run
467 spack install dealii@develop+mpi+petsc~int64%gcc ^petsc+complex~hypre
469 The good thing is that if you already have deal.ii built with real-valued petsc, then only `petsc`, `slepc` and `deal.ii` itself will be rebuild. Everything else (`trilinos`, `mumps`, `metis`, etc) will be reused.
471 One can use `environment-modules` (see above) to automatically set `DEAL_II_DIR` to the complex version:
473 spack load dealii@develop+mpi+petsc~int64%gcc ^petsc+complex~hypre
476 ### Filesystem Views:
477 If you prefer to have the whole dealii suite (and possible something else) symlinked into a single path (like `/usr/local`), one can use [Filesystem Views](http://spack.readthedocs.io/en/latest/workflows.html#filesystem-views):
479 spack view -v symlink dealii_suite dealii@develop
481 You can also add to this view other packages, i.e.
483 spack view -v symlink dealii_suite the-silver-searcher
487 ### Check before build:
488 It is often convenient to check which version of packages, compilers, variants etc will be used before actually starting installation. That can be done by examining the concretized spec via `spack spec` command, e.g.
490 spack spec dealii@develop+mpi+petsc~int64%gcc ^petsc+complex~hypre
493 The `spec` command has a useful flag `-I` which will show install status of dependencies in the graph.
495 ### Develop using Spack
496 Probably the easiest way to use Spack while contributing patches to the deal.II is the following.
498 1. Install `deal.II` via Spack as usual `spack install <dealii-spec>` (`<dealii-spec>` can be `dealii@develop+mpi^openmpi^openblas` or alike)
499 2. Clone the deal.II sources and `cd` to a build folder (we assume it's in the root of source location)
500 3. Get `cmake` command used by Spack to build `deal.II` from `<dealii-prefix>/.spack/build.out` and save it in `build_command.sh` via
503 cat $(spack location -i <dealii-spec>)/.spack/build.out | grep "==> 'cmake'" | sed -e "s/[^ ]*[^ ]/'..\/'/3" | cut -d " " -f2- > build_command.sh
506 You might want to remove the `DCMAKE_INSTALL_PREFIX` as you probably don't want to accidentally override the version installed by Spack.
508 4. Setup build environment to be exactly the same as used by Spack
511 spack build-env dealii@develop+mpi^openmpi^openblas bash
514 5. Configure and build
516 source build_command.sh
520 #### Installation of dependencies alone
521 It is possible to get Spack to install only the dependencies, and leave the configuration and installation of deal.II entirely up to you.
522 The manual configuration process includes having to point CMake to the Spack installed libraries, which could possibly be assisted using [Filesystem view](#filesystem-views)).
523 To install only the dependencies of deal.II (specifically, the developer version), issue the following command:
525 spack install --only dependencies dealii@develop ^cmake@3.9.4
529 An alternative is to create a [Filesystem view](#filesystem-views) for an already installed library and then compile patched version of deal.II manually by providing path to the view for each dependency.
531 ### Keep the stage to run unit tests
532 By default, the build folder (aka `stage`) for each package is created in a temporary system dependent location.
533 This gets purge by system on next restart.
534 If you want to keep the stage after the installation, you need to do two things:
535 First, make spack use a custom location for stage by adding the following to your `~/.spack/config.yaml`:
539 - $spack/var/spack/stage
541 Second, prescribe an extra argument to `install` command to keep the stage after successful installation:
543 spack install --keep-stage dealii@develop
546 ### Licensed software
547 Spack supports installation of [licensed software](http://spack.readthedocs.io/en/latest/packaging_guide.html#license) as well as usage of [Licensed compilers](http://spack.readthedocs.io/en/latest/getting_started.html#licensed-compilers). In order to configure Intel compilers see [this page](http://spack.readthedocs.io/en/latest/getting_started.html#vendor-specific-compiler-configuration).
550 ### Freeze package versions
551 Currently Spack does not try to re-use already installed packages. On another hand, by default
552 the most recent version of a package will be installed. When updating deal.II build (for example to use the new version of `trilinos`), the combination of the two factors may lead to recompilation of many other packages used in the deal.II suite when one of the main build dependency like `cmake` has a new version.
554 To circumvent the problem, the user can specify preferred versions of packages in `~/.spack/packages.yaml`:
566 This settings will be taken into account during conretization process and thus will help to avoid rebuilding most of the deal.II suite when, for example, `openssl` is updated to the new version.
568 ## Known issues using Spack:
570 ### Incompatible CMake versions
572 At the moment there is an incompatibility between deal.II and `cmake` with version 3.10 and greater.
573 To install deal.II using an earlier version of `cmake`, either use the `packages.yaml` file to freeze `cmake` to an earlier version, or specify the version at install time, e.g.
575 spack install dealii^cmake@3.9.4