ParFlow setup on MacOS Catalina 10.14.x
Given that some changes have been made to MacOS Catalina it’s probably time for an updated ParFlow installation post. A lot of what you see here is echoed in the Mojave install but there are some important differences. As before, we’ll walk through a “typical” ParFlow installation, that is one that uses the most common options and configurations on macOS using the default shell (which is one of the changes). We’ll primarily use the command line for this so go ahead and open a terminal window (if you don’t know where terminal is use spotlight search, command+space, then type in terminal).
* And a special thanks to our beta testers of these instructions. Every Mac is a little different and they help us find the little nuances across different setups.
Preliminaries
For commands entered into the terminal window I’ll use the following convention where the command is preceded with “>>” for example:
>> make
One of the major changes in Catalina is that bash (bsh) is no longer the default shell and Z-shell (zsh) has replaced it. Fortunately, they’re very similar so the main difference is in where we’ll define our environment variables. If you recently updated from an older version of MacOS you may need to manually switch shells. This can be done by typing:
>> chsh -s /bin/zsh
You’ll need to enter your admin password and the switch will be complete. You can check the current shell by typing “echo $SHELL” and that should display “/bin/zsh” if you are running in Z-shell
Defining environment variables for our ParFlow installation in zsh is similar to how we did this in previous posts. These variable definitions may seem odd if you’re new to compiling but we’ll explain everything as it gets used in the process. Navigate to your home folder and create (or edit) your .zshrc file:
>> cd ~
>> vim .zshrc
In vim, hit “i” to enter insert mode and add the following lines, preferably at the bottom:
export MPI_DIR=~/PFTree/OMPI-4.0.2
export MPI_RUN_DIR=~/PFTree/OMPI-4.0.2/bin
export PARFLOW_DIR=~/PFTree/parflow
export HYPRE_DIR=~/PFTree/hypre
export SILO_DIR=~/PFTree/silo
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++
export FC=gfortran
export F77=gfortran
After that, this next line needs to be at the very bottom of this file:
export PATH=$MPI_RUN_DIR:$PATH
That last one is important because it overrides the search order for MPI, which usually looks in “/usr/local/” first but we want it to see our version first instead of any others that may be floating around on your system. If you have other customizations on your system those can affect the PATH variable so be sure to check your “.zshrc” from time to time.
After entering those into your “.zshrc” press “escape” followed by “:” then “x” to save the changes and exit. To apply the changes:
>> source .zshrc
So what did that do? We’ve just defined some “environment variables” and shortcuts we’ll take advantage of later on that will be loaded automatically when you start your computer or open a new terminal window. These let us refer to $PARFLOW_DIR instead of having to write the full path every time. Keep in mind that you may need to update some of these paths depending on how you decided to configure your installation. I’ll refer to these environment variables as we come to them and point out where changes might be needed.
Next, we’ll create a directory to hold all the various bits that comprise a typical installation. From your home directory, make a folder called “PFTree” and enter it:
>> cd ~
>> mkdir PFTree
>> cd PFTree
There are 6 “parts” to these instructions and we’re going to handle each separately: 1) install gcc/gfortran, 2) install cmake, 3) install OpenMPI, 4) Install Silo, 5) Install Hypre, 6) install ParFlow. All except step 6 are supporting libraries or programs that ParFlow needs for our typical installation.
Step 1) Installing gfortran and gcc
The most stable installations on MacOS have used Apple’s native “clang” compiler that comes with your Mac, but we still need to install gfortran; it’s usually bundled with gcc so we’ll install both. Those can be downloaded here: http://hpc.sourceforge.net
The file you want is usually the first entry right below the word “Binaries” after the “big” paragraph at the top. Today, for me, that is “gcc-9.2-bin.tar.gz” and clicking on that link put it in my “Downloads” folder. You don’t need to bother with the “gfortran” link because it’s already included in this one. Before we forget, in your terminal window you’ll need to enable compiling programs on your mac. This is necessary after every major MacOS upgrade before you can compile new programs (previously compiled programs are usually fine though).
>> sudo xcode-select --install
A confirmation window will open and after you click OK through all of that, that step is done. You might also see a message saying command line tools are already installed, which is not a problem.
Next, navigate to wherever you put that download and extract it to /usr/local with:
>> cd ~/Downloads
>> sudo tar -xvf gcc-9.2-bin.tar -C /
A lot of text will fly by then you’ll get a command prompt. To make sure this has copied correctly, at the command prompt type:
>> gfortran -v
And you should get something like this:
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/x86_64-apple-darwin19/9.2.0/lto-wrapper
Target: x86_64-apple-darwin19
Configured with: ../gcc-9.2.0/configure --build=x86_64-apple-darwin19 --disable-nls --enable-checking=release --with-system-zlib --disable-multilib --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk --enable-languages=c,c++,fortran --with-gmp=/usr/local --with-mpc=/usr/local --with-mpfr=/usr/local
Thread model: posix
gcc version 9.2.0 (GCC)
The key thing to watch is that the version shown is the version you installed.
It’s now time to take a quick break while you restart your system. Seriously, go ahead and do it right meow; these instructions aren’t going anywhere, and not restarting may lead to non-trivial frustration later on.
Step 2) Installing cmake
Past installation instructions have used the GNU autoconfig system. We’ll still use that for some of the supporting components, but ParFlow itself is now built using the CMake interface and this makes life a lot easier on the end user. CMake isn’t installed on your mac by default, so our next task is adding it. Note: If you upgraded to Catalina and had a previous installation of CMake you do not to reinstall CMake and can skip to step 3.
We’re going to download the latest version of the CMake source. There is also a GUI for CMake but we don’t need it; I find it more trouble than it’s worth. Before proceeding, check here: https://cmake.org/download/ for the latest version of CMake. On the download page, go ahead and click on the link for the archive in the Unix/Linux source box and the download should complete shortly. Assuming the archive is in your ~/Downloads/ folder, we’ll copy it into PFTree, the expand the archive: As of this writing that file is “cmake-3.16.0-rc4.tar.gz”
To move that download directly into the PFTree folder, in your terminal window, be sure you’re in the PFTree directory, then type:
>> cd ~/PFTree/
>> cp ~/Downloads/cmake-3.16.0-rc4.tar .
>> tar -xvf cmake-3.16.0-rc4.tar
Note the period at the end of the second line. You will need to update the file name as newer versions of CMake are released, or if you used a different download or PFTree path.
Now we’ll enter the new directory and start building using cmake’s helper:
>> cd cmake-3.16.0-rc4
>> ./bootstrap
>> make
>> sudo make install
Those middle two steps will take a long time. At the end of the make process you’ll see percentage signs on the left edge of your screen and if it makes it to 100 without errors, you’re all set (though you may need to scroll up to see them).
Important: Remember that sudo is the “nuclear option” on linux variant systems and should only be used when absolutely necessary. It is here so that cmake will be available across your entire system, but this and installing gcc are the only places we need to go nuclear. Sudo should not be used for any other steps!
Step 3) Installing OpenMPI
OpenMPI is the message passing interface we use for ParFlow (note that this is not the same as OpenMP), and this is what makes it parallel. This section will build the latest version of OpenMPI in its own folder within your ParFlow tree. Even if you have an older version of OpenMPI, I recommend following these instructions to add a newer version to your system because there have been some changes; the newer versions tend to be more reliable and seem to have addressed some bugs we’ve discovered over the years. I’ll reiterate that these instructions will not overwrite or replace any existing MPI installations; to return to compiling/running with your existing OpenMPI version (if you have one), simply comment out the “export PATH” line we added to your .zshrc, or change the MPI_RUN_DIR variable, then source the file and you’re good to go. However, our goal is to install a new version, which is 4.0.2 as of this writing: see https://www.open-mpi.org/software/ompi/. Note that you can also get more recent versions from the OpenMPI GitHub page or download a nightly build (pre-release) and that's exactly what we're going to do.
Those details aside, navigate back to your PFTree directory and create a run folder for OpenMPI to run from (this folder should be the same as the environment variable we defined called MPI_DIR):
>> cd ~/PFTree
>> mkdir OMPI-4.0.2
Now, this name is largely for our own bookkeeping and you should update this yourself (here and in .zshrc) as more recent versions are released. We’re going to obtain the openmpi source, build the package, then install it to our MPI_RUN directory. The simplest way to get the source that is to clone the GitHub repository, which lives at: https://github.com/open-mpi/ompi
The clone command is:
>> git clone https://github.com/open-mpi/ompi.git --branch master --single-branch
which will create a folder called "ompi" in PFTree. Now we’ll configure the installation package for our system, but first I'm going to rename the folder to reflect my version number, then navigate into the newly created/renamed directory:
>> mv ompi ompi-4.0.2
>> cd ompi-4.0.2
>> ./configure --prefix=$MPI_DIR
(Note: your version may not actually be 4.0.2 but we just want to give it a unique name so it doesn't accidentally get overwritten. Also note that this is NOT the same path as our MPI_RUN_DIR! The source folder is lower case and the run folder is upper case; this distinction is intentional)
If you get any weird behavior you can add “CC=/usr/bin/clang CXX=/usr/bin/clang++” to the end of that configure line. Those CC and CXX arguments force the computer to use a specific compiler, which is clang. On most systems OpenMPI should install fine without those, besides we set them in our .zshrc, but sometimes computers seem to have a mind of their own and need a gentle push. After you press return, lots of stuff will fly by your screen for what will seem like forever and a decade. Once that’s done, type:
If you get any weird behavior you can add “CC=/usr/bin/clang CXX=/usr/bin/clang++” to the end of that configure line. Those CC and CXX arguments force the computer to use a specific compiler, which is clang. On most systems OpenMPI should install fine without those, besides we set them in our .zshrc, but sometimes computers seem to have a mind of their own and need a gentle push. After you press return, lots of stuff will fly by your screen for what will seem like forever and a decade. Once that’s done, type:
>> make
>> make install
The first command builds OpenMPI within the tree of the archive we downloaded (which normally takes just shy of eternity before it completes), and the second copies it into the place we’ll be running it from. With that, OpenMPI is now installed. Go ahead and run:
>> ompi_info
and look for your version. This command displays a lot of stuff that usually doesn’t fit on one screen so scroll up to the top and just after your “ompi_info” command there should be “Open MPI: 4.0.2” (or your version number). If you want to update OpenMPI in the future you’ll follow these same steps and create a new MPI_DIR (and MPI_RUN_DIR variable) directory for your version, whatever that might be. This prevents overwriting of OpenMPI versions and also saves you from having to uninstall it. If you see an old version of OpenMPI listed by ompi_info you probably need to source your .zshrc file and/or restart (or your installation was not successful).
Note that your compiler and OpenMPI version numbers are not related so having one installed correctly doesn't mean that both are. OpenMPI "wraps" itself around other compilers when you install it to provide its functionality, so if you update your system, or install a new compiler, it's not a bad idea to update and/or re-build OpenMPI afterwards. Finally, you can also clone the OpenMPI source from their GitHub page, the advantage being you'll always get the most recent version.
Step 4) Installing Silo
Silo is a file format that is used as the default for viewing ParFlow outputs, but we also now have other options available so it’s not as crucial as it once was. Some of the test cases use it, so we always recommend installing it, but technically you could omit it. It's pretty small and simple to install though so if nothing else it's good practice building programs for you. The latest version of silo can be found at: https://wci.llnl.gov/simulation/computer-codes/silo/downloads
You could download the archive in a web browser and move it but instead I’m going to use curl again. Navigate to the PFTree folder and the commands are:
>> cd ~/PFTree
>> curl "https://wci.llnl.gov/content/assets/docs/simulation/computer-codes/silo/silo-4.10.2/silo-4.10.2.tar.gz" -o "silo-4.10.2.tar.gz"
(Those quotation marks might need to be deleted and manually added if you're copying and pasting; html makes them look nicer but doesn't always encode them the same way)
Mimicking the process we used for OpenMPI, to expand and extract the archive type:
Mimicking the process we used for OpenMPI, to expand and extract the archive type:
>> tar -xzvf silo-4.10.2.tar.gz
If you get an error during this process the download link may have been corrupted or you may have inadvertently downloaded a version that uses a different compression (there are two different kinds of compression used for the SILO archives...ugh). A simple fix is to follow the same steps we used for CMake to download from a web browser into your "~/Downloads" folder then move it over to the target location.
Now we’ll rename the newly expanded directory to match our environment variable SILO_DIR:
>> mv silo-4.10.2 silo
Next we’ll navigate into the new directory, configure, and install:
>> cd silo
>> ./configure --disable-silex
>> make install
>> cd ..
And that completes the installation.
Note that if your version of Silo differs, you’ll need to update the appropriate file/directory names environment variable we set to reflect your installation path.
Step 5) Installing the HYPRE library
HYPRE is a library of numerical solvers for high-performance computing applications and ParFlow hooks into it for some of its functionality.
As before the simplest way to get this is curl. There were some issues with hypre version 10 and ParFlow but the current release (2.11.2) has been working just fine. You can check for newer versions here if you'd like: https://computing.llnl.gov/projects/hypre-scalable-linear-solvers-multigrid-methods/software
If you download a different you'll need to modify filenames/paths accordingly.
Obtaining and installing hypre is nothing more than a minor variation of what we’ve already done so navigate back to your PFTree:
>> cd ~/PFTree
Like OpenMPI, we're going to clone the GitHub repository to get the very latest version. To do so, use the following command from your PFTree directory:
Like OpenMPI, we're going to clone the GitHub repository to get the very latest version. To do so, use the following command from your PFTree directory:
And that will create a directory called "hypre" in your PFTree with all the source files in it.
Regardless of which option you picked, we'll move on to the setup.
>> cd ~/PFTree/hypre/src
>> ./configure --prefix=$HYPRE_DIR CC=$MPI_RUN_DIR/mpicc
>> make install
Of course, if you didn't define "$MPI_RUN_DIR" that won't work for you and you can substitute "CC=mpicc" in a pinch, but doing that won't guarantee that the version of OpenMPI we just installed is used. This installation also takes a while to install, but it’s usually not as long as OpenMPI. You’re likely to see a lot of warnings about “could not find any symbols” but as long as there are no errors you should be able to move on to ParFlow. One quick note is that the reason for the manual specification of the CC flag here is that some users have reported issues with an “mpi.h not found” error. We’ll gloss over the details of why that happens (that’d be an express ticket to nap town), but this syntax makes sure the compiler we just associated with our new OpenMPI build is used when we build hypre.
Step 6) Installing ParFlow
Last, we’re going to build ParFlow and PFTools using the unified CMake approach. There are a lot of different setup options for ParFlow but for our “typical” setup we’re assuming you want to run in parallel, you might want to run CLM, you have Silo and Hypre, and a few other little things. All of this can be changed but I’d suggest building with these options first, then playing around after you have a ParFlow build that passes the verification tests.
First, we need to download the code. ParFlow is hosted on GitHub (link here), so to obtain the code:
>> cd ~/PFTree
>> git clone https://github.com/parflow/parflow.git --branch master --single-branch
which will create a folder called parflow in PFTree and now you’ve got the source code. You may want to rename the folder (I like to use the date I downloaded it so /parflow.20191124 for example) but you’ll need to update your PARFLOW_DIR too if you change that. I’m going to proceed assuming you have not renamed it.
Next, we’ll create a directory for the incremental files of the build process called “build”
>> mkdir build
>> cd build
Note: You can also update an existing ParFlow build by navigating to the folder where the parflow source tree lives and using “git pull.” This command will synchronize you to the master repository. If you had a previous build the best thing to do is make sure to remove your old “build” directory and then do exactly what is listed here. If the same environment variables are used, the old binaries will be overwritten.
The next step will tell CMake how to configure ParFlow using the typical options (copy and paste will make you much happier than typing this all):
>> cmake ../parflow -DCMAKE_INSTALL_PREFIX=$PARFLOW_DIR -DHYPRE_ROOT=$HYPRE_DIR -DPARFLOW_AMPS_LAYER=mpi1 -DPARFLOW_AMPS_SEQUENTIAL_IO=true -DPARFLOW_ENABLE_TIMING=true -DSILO_ROOT=$SILO_DIR -DPARFLOW_HAVE_CLM=ON
Note that if your parflow source tree is NOT in a folder called parflow, you’ll need to change that right after “cmake” in the above command or this will go badly for you. The manual provides a few extra details on how you can customize an installation using the “ccmake” semi-GUI interface, but we don’t need that now. Next type:
>> make
and lots of stuff will happen. If the last line resemble has a [100%] resembling this:
[100%] Built target gmstinvertices
(the text might differ as changes are made in the ParFlow source tree). With that, you’re ready for the last command:
>> make install
The difference is that “make” builds everything in our temporary structure (the reason we created that build directory in the first), and “make install” copies the final build into our installation location. Think of it this way: cmake fully compiles the package in an isolated compartment, then, after that works, we copy it to the final destination.
And that’s it for the installation. The last step is to test the installation. Our old testing interface has been removed in favor of one that runs directly from the build folder. After your “make install” command, don’t navigate anywhere, instead just type:
>> make test
A lot of things will fly by as ParFlow verifies that it is working correctly with “Passed” or “Failed” for each. Some of these are designed to test the parallel performance and may use a higher number of processor than your system has. For example, default_richards_3_3_3.tcl will fail on any system that doesn’t have 27 processors so that’s not really an issue. For example, a 6-core Mac will fail 8 of the 155 tests and an 18-core Mac will fail 2. As long as all tests pass that are within the realm of the number of cores on your Mac, you’re in good shape.
Step 7) Some troubleshooting tips
If you are connected to an wifi network with enterprise level authentication you may experience errors when you try to run ParFlow. These will be cryptic and look something like:
*** An error occurred in MPI_Comm_split
*** reported by process [1866268673,0]
*** on communicator MPI_COMM_WORLD
*** MPI_ERR_ARG: invalid argument of some other kind
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
Most users are able to overcome this by simply turning off wifi when they need to run ParFlow but a few folks have had other issues. This results from a bug in OpenMPI when running on MacOS and the OpenMPI developers are aware of the issue but we don’t have an estimate of when that fix will arrive. If you are having these kinds of problems we recommend using the Docker build of ParFlow, which can be found from the ParFlow GitHub page.
The single most common reasons that steps of the installation don’t go well is 1) not being in the correct path, 2) not defining the environment variables, and 3) failing to source .zshrc. You can always verify your location in the directory tree using the “pwd” command and each step tells you where you should be before you begin entering commands.
Your mac is also case-sensitive so pay attention to capitalization. That can really mess with you and be sure NOT to use spaces in any path names; drop in an underscore _ if you need to. If you’re getting a bunch of strange behaviors, it’s a good idea to stop and restart your system before trying too many things, the last of which may include tossing your computer out a window and getting a new one.
If you prefer to use the c-shell, or bash-shell, or some other shell, most of the steps are the same but your environment variables may be defined differently. The setup example in the ParFlow manual outlines the c-shell variations of all the steps we covered here; bash is basically identical to z-shell and is described in the Mojave blog post. If you really want to use another shell you’re probably just being stubborn, but if you're that stubborn you're aso probably experienced enough in said shell to figure out any differences.
An issue was identified in a previous build where a system with more than one version of TCL/TK can result in memory allocation errors at runtime. If you have any issues with this, look at the following comment from the MacOS Mojave installation and apply a similar solution for your PATH variable.
An issue was identified in a previous build where a system with more than one version of TCL/TK can result in memory allocation errors at runtime. If you have any issues with this, look at the following comment from the MacOS Mojave installation and apply a similar solution for your PATH variable.
Lastly, if you have MacPorts or any other “helper packages” installed, your system paths may have been altered from the defaults. Defining all the environment variables we need for ParFlow at the bottom of your .zshrc should take care of this, making sure that our PATH override comes last.
If you find an issue that isn't listed here that doesn't necessarily mean you've found a bug. There can be a lot of variability across machines because of the impacts of other programs/libraries. The majority of the time bugs in that install process aren't a problem in the source code but are instead machine-specific configuration details, so we recommend that you contact the ParFlow user email list (search for the text "Parflow-Users" on the GitHub page) for help. That should always be your first step when troubleshooting.
