Thursday, June 4, 2020

Installing ParFlow on Ubuntu LTS

Since we have been working with ParFlow on Ubuntu, we thought that it might be helpful to provide a guide for others to install ParFlow on Ubuntu systems. The following directions are to install the current version of ParFlow on the two most recent Ubuntu LTS releases, 18.04 and 20.04.

1) Setup development packages on Linux

First, you need to download the new package information and install some new packages. Copy and paste the following in your command line:
   sudo apt-get update 
   sudo apt-get install \ 
     build-essential \ 
     curl \ 
     git \ 
     gfortran \ 
     libopenblas-dev \ 
     liblapack-dev \ 
     openssh-client \ 
     openssh-server \ 
     openmpi-bin \ 
     libopenmpi-dev \ 
     tcl-dev \ 
     tk-dev 

This may take a little while. Note: you may not need the sudo command if you are running with root privileges.


2) Create the ParFlow directory and its components

Next, you need to create a directory for ParFlow with a few subfolders. This example installs ParFlow and all its dependencies in the home (~) folder, although you can change this to whatever you’d like.

   mkdir -p ~/parflow/build \
     ~/parflow/dependencies/cmake \
     ~/parflow/dependencies/hypre-src 

3) Fetch the dependencies (cmake and hypre)

Let’s start with downloading cmake (3.17.3 being the current version):

   cd ~/parflow/dependencies/cmake 
   curl -L https://cmake.org/files/v3.17/cmake-3.17.3-Linux-x86_64.tar.gz \
     | tar --strip-components=1 -xzv 

Once that is completed, you can install hypre (2.19.0 being the current version):

   cd ~/parflow/dependencies/hypre-src 
   curl -L https://github.com/hypre-space/hypre/archive/v2.19.0.tar.gz \
     | tar --strip-components=1 -xzv 
   cd src 
   ./configure --prefix=~/parflow/dependencies/hypre \
     --with-MPI 
   make install 

4) Download and build ParFlow

You can then download ParFlow from GitHub using the commands below. The current release as of this post is v3.6.0. Note: Ubuntu 20.04 has a newer version of tcl that is not compatible with the current release. If you are installing ParFlow on Ubuntu 20.04, replace the “v3.6.0” with “master” to clone the master branch instead. The current master branch of ParFlow is compatible with the newer version of tcl on Ubuntu 20.04.

   git clone --single-branch --branch v3.6.0 \
     --recursive https://github.com/parflow/parflow.git \
     ~/parflow/src 

Enter the following in your command line to build ParFlow.

   ~/parflow/dependencies/cmake/bin/cmake \
-S ~/parflow/src \
-B ~/parflow/build \
-D PARFLOW_AMPS_LAYER=mpi1 \
-D PARFLOW_AMPS_SEQUENTIAL_IO=TRUE \
-D HYPRE_ROOT=
~/parflow/dependencies/hypre \
-D PARFLOW_ENABLE_TIMING=TRUE \
-D PARFLOW_HAVE_CLM=TRUE

~/parflow/dependencies/cmake/bin/cmake --build \
~/parflow/build

~/parflow/dependencies/cmake/bin/cmake --install \
~/parflow/build --prefix ~/parflow/install

5) Create an environment file

Here, you will create an environment file “env.sh” to store the ParFlow directory, making it easier to reference ParFlow when you want to run it later. Type the following:

   echo export PARFLOW_DIR=~/parflow/install \
     >> ~/parflow/env.sh 

With that environment file, you can use ParFlow from anywhere by sourcing it, as shown:

   source ~/parflow/env.sh 

6) Test ParFlow

Here, you will create an environment file “env.sh” which points to the necessary directories for keeping the components necessary for running ParFlow. Enter the following:

Hopefully you haven’t gotten any errors at this point and you have installed ParFlow successfully. To test ParFlow, source the environment file above and then type:
   cd ~/parflow/build/ 
   ~/parflow/dependencies/cmake/bin/ctest -VV 
This will run the series of tests to make sure your install of ParFlow is working correctly. As of this writing, nine tests fail that are attempting to reference .silo files (note that this installation guide does not install Silo). We are currently addressing this issue. The tests that will fail due to the bug are: water_balance_x.hardflow.nojac.tcl (all topologies), water_balance_x.hardflow.jac.tcl (all topologies), and impes.plinear.tcl.

Conclusion

Congratulations, you now have ParFlow on your system! The ParFlow executable is located in ~/parflow/src/pfsimulator/, and the pftools are located in ~/parflow/src/pftools/. Before running a ParFlow script, you may have to re-source your environment file by typing in “source ~/parflow/env.sh” as shown above.

These install instructions are combined into shell scripts within the scripts folder on this GitHub repo. The install directions above are divided into two scripts: “01_packages.sh” executes step 1, and “02_parflow.sh” executes steps 2 through 5. You will have to execute the testing on your own. The Readme file provides directions for how to use these scripts, as well as instructions and files for building ParFlow within an Ubuntu Docker image.

For more information, please refer to the ParFlow manual PDF located in ~/parflow/src/ and ParFlow’s GitHub page (https://github.com/parflow/parflow). Happy modeling!


Tuesday, December 3, 2019

The MacOS Catalina installation guide for ParFlow

ParFlow setup on MacOS Catalina 10.14.x

Given that some changes have been made to MacOS Catalina it’s probably time for an updated ParFlow installation post. A lot of what you see here is echoed in the Mojave install but there are some important differences. As before, we’ll walk through a “typical” ParFlow installation, that is one that uses the most common options and configurations on macOS using the default shell (which is one of the changes). We’ll primarily use the command line for this so go ahead and open a terminal window (if you don’t know where terminal is use spotlight search, command+space, then type in terminal).

* And a special thanks to our beta testers of these instructions. Every Mac is a little different and they help us find the little nuances across different setups. 

Preliminaries

For commands entered into the terminal window I’ll use the following convention where the command is preceded with “>>” for example:

>> make                       

One of the major changes in Catalina is that bash (bsh) is no longer the default shell and Z-shell (zsh) has replaced it. Fortunately, they’re very similar so the main difference is in where we’ll define our environment variables. If you recently updated from an older version of MacOS you may need to manually switch shells. This can be done by typing:

>> chsh -s /bin/zsh

You’ll need to enter your admin password and the switch will be complete. You can check the current shell by typing “echo $SHELL” and that should display “/bin/zsh” if you are running in Z-shell

Defining environment variables for our ParFlow installation in zsh is similar to how we did this in previous posts. These variable definitions may seem odd if you’re new to compiling but we’ll explain everything as it gets used in the process. Navigate to your home folder and create (or edit) your .zshrc file:

>> cd ~
>> vim .zshrc

In vim, hit “i” to enter insert mode and add the following lines, preferably at the bottom:

export MPI_DIR=~/PFTree/OMPI-4.0.2
export MPI_RUN_DIR=~/PFTree/OMPI-4.0.2/bin
export PARFLOW_DIR=~/PFTree/parflow
export HYPRE_DIR=~/PFTree/hypre
export SILO_DIR=~/PFTree/silo
export CC=/usr/bin/clang
export CXX=/usr/bin/clang++
export FC=gfortran
export F77=gfortran

After that, this next line needs to be at the very bottom of this file:

export PATH=$MPI_RUN_DIR:$PATH

That last one is important because it overrides the search order for MPI, which usually looks in “/usr/local/” first but we want it to see our version first instead of any others that may be floating around on your system. If you have other customizations on your system those can affect the PATH variable so be sure to check your “.zshrc” from time to time. 

After entering those into your “.zshrc” press “escape” followed by “:” then “x” to save the changes and exit. To apply the changes:

>> source .zshrc

So what did that do? We’ve just defined some “environment variables” and shortcuts we’ll take advantage of later on that will be loaded automatically when you start your computer or open a new terminal window. These let us refer to $PARFLOW_DIR instead of having to write the full path every time. Keep in mind that you may need to update some of these paths depending on how you decided to configure your installation. I’ll refer to these environment variables as we come to them and point out where changes might be needed.

Next, we’ll create a directory to hold all the various bits that comprise a typical installation. From your home directory, make a folder called “PFTree” and enter it:

>> cd ~                                    
>> mkdir PFTree
>> cd PFTree

There are 6 “parts” to these instructions and we’re going to handle each separately: 1) install gcc/gfortran, 2) install cmake, 3) install OpenMPI, 4) Install Silo, 5) Install Hypre, 6) install ParFlow. All except step 6 are supporting libraries or programs that ParFlow needs for our typical installation. 

Step 1) Installing gfortran and gcc

The most stable installations on MacOS have used Apple’s native “clang” compiler that comes with your Mac, but we still need to install gfortran; it’s usually bundled with gcc so we’ll install both. Those can be downloaded here: http://hpc.sourceforge.net

The file you want is usually the first entry right below the word “Binaries” after the “big” paragraph at the top. Today, for me, that is “gcc-9.2-bin.tar.gz” and clicking on that link put it in my “Downloads” folder. You don’t need to bother with the “gfortran” link because it’s already included in this one. Before we forget, in your terminal window you’ll need to enable compiling programs on your mac. This is necessary after every major MacOS upgrade before you can compile new programs (previously compiled programs are usually fine though).

>> sudo xcode-select --install

A confirmation window will open and after you click OK through all of that, that step is done. You might also see a message saying command line tools are already installed, which is not a problem.

Next, navigate to wherever you put that download and extract it to /usr/local with:

>> cd ~/Downloads                  
>> sudo tar -xvf gcc-9.2-bin.tar -C /

A lot of text will fly by then you’ll get a command prompt. To make sure this has copied correctly, at the command prompt type:

>> gfortran -v

And you should get something like this:
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/x86_64-apple-darwin19/9.2.0/lto-wrapper
Target: x86_64-apple-darwin19
Configured with: ../gcc-9.2.0/configure --build=x86_64-apple-darwin19 --disable-nls --enable-checking=release --with-system-zlib --disable-multilib --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk --enable-languages=c,c++,fortran --with-gmp=/usr/local --with-mpc=/usr/local --with-mpfr=/usr/local
Thread model: posix
gcc version 9.2.0 (GCC)

The key thing to watch is that the version shown is the version you installed. 

It’s now time to take a quick break while you restart your system. Seriously, go ahead and do it right meow; these instructions aren’t going anywhere, and not restarting may lead to non-trivial frustration later on. 

Step 2) Installing cmake

Past installation instructions have used the GNU autoconfig system. We’ll still use that for some of the supporting components, but ParFlow itself is now built using the CMake interface and this makes life a lot easier on the end user. CMake isn’t installed on your mac by default, so our next task is adding it. Note: If you upgraded to Catalina and had a previous installation of CMake you do not to reinstall CMake and can skip to step 3.

We’re going to download the latest version of the CMake source. There is also a GUI for CMake but we don’t need it; I find it more trouble than it’s worth. Before proceeding, check here: https://cmake.org/download/ for the latest version of CMake. On the download page, go ahead and click on the link for the archive in the Unix/Linux source box and the download should complete shortly. Assuming the archive is in your ~/Downloads/ folder, we’ll copy it into PFTree, the expand the archive:  As of this writing that file is “cmake-3.16.0-rc4.tar.gz”

To move that download directly into the PFTree folder, in your terminal window, be sure you’re in the PFTree directory, then type:

>> cd ~/PFTree/
>> cp ~/Downloads/cmake-3.16.0-rc4.tar .
>> tar -xvf cmake-3.16.0-rc4.tar

Note the period at the end of the second line. You will need to update the file name as newer versions of CMake are released, or if you used a different download or PFTree path. 

Now we’ll enter the new directory and start building using cmake’s helper: 

>> cd cmake-3.16.0-rc4
>> ./bootstrap                          
>> make                                   
>> sudo make install                 

Those middle two steps will take a long time. At the end of the make process you’ll see percentage signs on the left edge of your screen and if it makes it to 100 without errors, you’re all set (though you may need to scroll up to see them).

Important: Remember that sudo is the “nuclear option” on linux variant systems and should only be used when absolutely necessary. It is here so that cmake will be available across your entire system, but this and installing gcc are the only places we need to go nuclear. Sudo should not be used for any other steps!


Step 3) Installing OpenMPI

OpenMPI is the message passing interface we use for ParFlow (note that this is not the same as OpenMP), and this is what makes it parallel. This section will build the latest version of OpenMPI in its own folder within your ParFlow tree. Even if you have an older version of OpenMPI, I recommend following these instructions to add a newer version to your system because there have been some changes; the newer versions tend to be more reliable and seem to have addressed some bugs we’ve discovered over the years. I’ll reiterate that these instructions will not overwrite or replace any existing MPI installations; to return to compiling/running with your existing OpenMPI version (if you have one), simply comment out the “export PATH” line we added to your .zshrc, or change the MPI_RUN_DIR variable, then source the file and you’re good to go. However, our goal is to install a new version, which is 4.0.2 as of this writing: see https://www.open-mpi.org/software/ompi/. Note that you can also get more recent versions from the OpenMPI GitHub page or download a nightly build (pre-release) and that's exactly what we're going to do. 

Those details aside, navigate back to your PFTree directory and create a run folder for OpenMPI to run from (this folder should be the same as the environment variable we defined called MPI_DIR):

>> cd ~/PFTree
>> mkdir OMPI-4.0.2

Now, this name is largely for our own bookkeeping and you should update this yourself (here and in .zshrc) as more recent versions are released. We’re going to obtain the openmpi source, build the package, then install it to our MPI_RUN directory. The simplest way to get the source that is to clone the GitHub repository, which lives at: https://github.com/open-mpi/ompi 
The clone command is: 

>> git clone https://github.com/open-mpi/ompi.git --branch master --single-branch

which will create a folder called "ompi" in PFTree. Now we’ll configure the installation package for our system, but first I'm going to rename the folder to reflect my version number, then navigate into the newly created/renamed directory:

>> mv ompi ompi-4.0.2
>> cd ompi-4.0.2
>> ./configure --prefix=$MPI_DIR

(Note: your version may not actually be 4.0.2 but we just want to give it a unique name so it doesn't accidentally get overwritten. Also note that this is NOT the same path as our MPI_RUN_DIR! The source folder is lower case and the run folder is upper case; this distinction is intentional) 
If you get any weird behavior you can add “CC=/usr/bin/clang CXX=/usr/bin/clang++” to the end of that configure line. Those CC and CXX arguments force the computer to use a specific compiler, which is clang. On most systems OpenMPI should install fine without those, besides we set them in our .zshrc, but sometimes computers seem to have a mind of their own and need a gentle push. After you press return, lots of stuff will fly by your screen for what will seem like forever and a decade. Once that’s done, type:

>> make
>> make install

The first command builds OpenMPI within the tree of the archive we downloaded (which normally takes just shy of eternity before it completes), and the second copies it into the place we’ll be running it from. With that, OpenMPI is now installed. Go ahead and run:

>> ompi_info

and look for your version. This command displays a lot of stuff that usually doesn’t fit on one screen so scroll up to the top and just after your “ompi_info” command there should be “Open MPI: 4.0.2” (or your version number). If you want to update OpenMPI in the future you’ll follow these same steps and create a new MPI_DIR (and MPI_RUN_DIR variable) directory for your version, whatever that might be. This prevents overwriting of OpenMPI versions and also saves you from having to uninstall it. If you see an old version of OpenMPI listed by ompi_info you probably need to source your .zshrc file and/or restart (or your installation was not successful). 
Note that your compiler and OpenMPI version numbers are not related so having one installed correctly doesn't mean that both are. OpenMPI "wraps" itself around other compilers when you install it to provide its functionality, so if you update your system, or install a new compiler, it's not a bad idea to update and/or re-build OpenMPI afterwards. Finally, you can also clone the OpenMPI source from their GitHub page, the advantage being you'll always get the most recent version.  


Step 4) Installing Silo

Silo is a file format that is used as the default for viewing ParFlow outputs, but we also now have other options available so it’s not as crucial as it once was. Some of the test cases use it, so we always recommend installing it, but technically you could omit it. It's pretty small and simple to install though so if nothing else it's good practice building programs for you. The latest version of silo can be found at: https://wci.llnl.gov/simulation/computer-codes/silo/downloads

You could download the archive in a web browser and move it but instead I’m going to use curl again. Navigate to the PFTree folder and the commands are:

>> cd ~/PFTree
>> curl "https://wci.llnl.gov/content/assets/docs/simulation/computer-codes/silo/silo-4.10.2/silo-4.10.2.tar.gz" -o "silo-4.10.2.tar.gz"

(Those quotation marks might need to be deleted and manually added if you're copying and pasting; html makes them look nicer but doesn't always encode them the same way
Mimicking the process we used for OpenMPI, to expand and extract the archive type:

>> tar -xzvf silo-4.10.2.tar.gz 

If you get an error during this process the download link may have been corrupted or you may have inadvertently downloaded a version that uses a different compression (there are two different kinds of compression used for the SILO archives...ugh). A simple fix is to follow the same steps we used for CMake to download from a web browser into your "~/Downloads" folder then move it over to the target location.

Now we’ll rename the newly expanded directory to match our environment variable SILO_DIR:

>> mv silo-4.10.2 silo

Next we’ll navigate into the new directory, configure, and install:

>> cd silo
>> ./configure --disable-silex
>> make install
>> cd ..

And that completes the installation.

Note that if your version of Silo differs, you’ll need to update the appropriate file/directory names environment variable we set to reflect your installation path. 

Step 5) Installing the HYPRE library

HYPRE is a library of numerical solvers for high-performance computing applications and ParFlow hooks into it for some of its functionality.  

As before the simplest way to get this is curl. There were some issues with hypre version 10 and ParFlow but the current release (2.11.2) has been working just fine. You can check for newer versions here if you'd like: https://computing.llnl.gov/projects/hypre-scalable-linear-solvers-multigrid-methods/software 
If you download a different you'll need to modify filenames/paths accordingly. 

Obtaining and installing hypre is nothing more than a minor variation of what we’ve already done so navigate back to your PFTree:

>> cd ~/PFTree

Like OpenMPI, we're going to clone the GitHub repository to get the very latest version. To do so, use the following command from your PFTree directory:

>> git clone https://github.com/LLNL/hypre.git --branch master --single-branch

And that will create a directory called "hypre" in your PFTree with all the source files in it. 

Regardless of which option you picked, we'll move on to the setup. 

>> cd ~/PFTree/hypre/src
>> ./configure --prefix=$HYPRE_DIR CC=$MPI_RUN_DIR/mpicc
>> make install

Of course, if you didn't define "$MPI_RUN_DIR" that won't work for you and you can substitute "CC=mpicc" in a pinch, but doing that won't guarantee that the version of OpenMPI we just installed is used. This installation also takes a while to install, but it’s usually not as long as OpenMPI. You’re likely to see a lot of warnings about “could not find any symbols” but as long as there are no errors you should be able to move on to ParFlow. One quick note is that the reason for the manual specification of the CC flag here is that some users have reported issues with an “mpi.h not found” error. We’ll gloss over the details of why that happens (that’d be an express ticket to nap town), but this syntax makes sure the compiler we just associated with our new OpenMPI build is used when we build hypre. 


Step 6) Installing ParFlow

Last, we’re going to build ParFlow and PFTools using the unified CMake approach. There are a lot of different setup options for ParFlow but for our “typical” setup we’re assuming you want to run in parallel, you might want to run CLM, you have Silo and Hypre, and a few other little things. All of this can be changed but I’d suggest building with these options first, then playing around after you have a ParFlow build that passes the verification tests.

First, we need to download the code. ParFlow is hosted on GitHub (link here), so to obtain the code:

>> cd ~/PFTree
>> git clone https://github.com/parflow/parflow.git --branch master --single-branch

which will create a folder called parflow in PFTree and now you’ve got the source code. You may want to rename the folder (I like to use the date I downloaded it so /parflow.20191124 for example) but you’ll need to update your PARFLOW_DIR too if you change that. I’m going to proceed assuming you have not renamed it. 

Next, we’ll create a directory for the incremental files of the build process called “build”

>> mkdir build
>> cd build

Note: You can also update an existing ParFlow build by navigating to the folder where the parflow source tree lives and using “git pull.” This command will synchronize you to the master repository. If you had a previous build the best thing to do is make sure to remove your old “build” directory and then do exactly what is listed here. If the same environment variables are used, the old binaries will be overwritten.

The next step will tell CMake how to configure ParFlow using the typical options (copy and paste will make you much happier than typing this all):
>> cmake ../parflow -DCMAKE_INSTALL_PREFIX=$PARFLOW_DIR -DHYPRE_ROOT=$HYPRE_DIR -DPARFLOW_AMPS_LAYER=mpi1 -DPARFLOW_AMPS_SEQUENTIAL_IO=true -DPARFLOW_ENABLE_TIMING=true -DSILO_ROOT=$SILO_DIR -DPARFLOW_HAVE_CLM=ON

Note that if your parflow source tree is NOT in a folder called parflow, you’ll need to change that right after “cmake” in the above command or this will go badly for you. The manual provides a few extra details on how you can customize an installation using the “ccmake” semi-GUI interface, but we don’t need that now. Next type:

>> make

and lots of stuff will happen. If the last line resemble has a [100%] resembling this:
[100%] Built target gmstinvertices

(the text might differ as changes are made in the ParFlow source tree). With that, you’re ready for the last command:

>> make install

The difference is that “make” builds everything in our temporary structure (the reason we created that build directory in the first), and “make install” copies the final build into our installation location. Think of it this way: cmake fully compiles the package in an isolated compartment, then, after that works, we copy it to the final destination.

And that’s it for the installation. The last step is to test the installation. Our old testing interface has been removed in favor of one that runs directly from the build folder. After your “make install” command, don’t navigate anywhere, instead just type:

>> make test

A lot of things will fly by as ParFlow verifies that it is working correctly with “Passed” or “Failed” for each. Some of these are designed to test the parallel performance and may use a higher number of processor than your system has. For example, default_richards_3_3_3.tcl will fail on any system that doesn’t have 27 processors so that’s not really an issue. For example, a 6-core Mac will fail 8 of the 155 tests and an 18-core Mac will fail 2. As long as all tests pass that are within the realm of the number of cores on your Mac, you’re in good shape.


Step 7) Some troubleshooting tips

If you are connected to an wifi network with enterprise level authentication you may experience errors when you try to run ParFlow. These will be cryptic and look something like:
    *** An error occurred in MPI_Comm_split
    *** reported by process [1866268673,0]
    *** on communicator MPI_COMM_WORLD
    *** MPI_ERR_ARG: invalid argument of some other kind
    *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
    ***    and potentially your MPI job)
Most users are able to overcome this by simply turning off wifi when they need to run ParFlow but a few folks have had other issues. This results from a bug in OpenMPI when running on MacOS and the OpenMPI developers are aware of the issue but we don’t have an estimate of when that fix will arrive. If you are having these kinds of problems we recommend using the Docker build of ParFlow, which can be found from the ParFlow GitHub page.

The single most common reasons that steps of the installation don’t go well is 1) not being in the correct path, 2) not defining the environment variables, and 3) failing to source .zshrc. You can always verify your location in the directory tree using the “pwd” command and each step tells you where you should be before you begin entering commands.

Your mac is also case-sensitive so pay attention to capitalization. That can really mess with you and be sure NOT to use spaces in any path names; drop in an underscore _ if you need to. If you’re getting a bunch of strange behaviors, it’s a good idea to stop and restart your system before trying too many things, the last of which may include tossing your computer out a window and getting a new one.

If you prefer to use the c-shell, or bash-shell, or some other shell, most of the steps are the same but your environment variables may be defined differently. The setup example in the ParFlow manual outlines the c-shell variations of all the steps we covered here; bash is basically identical to z-shell and is described in the Mojave blog post. If you really want to use another shell you’re probably just being stubborn, but if you're that stubborn you're aso probably experienced enough in said shell to figure out any differences. 

An issue was identified in a previous build where a system with more than one version of TCL/TK can result in memory allocation errors at runtime. If you have any issues with this, look at the following comment from the MacOS Mojave installation and apply a similar solution for your PATH variable.

Lastly, if you have MacPorts or any other “helper packages” installed, your system paths may have been altered from the defaults. Defining all the environment variables we need for ParFlow at the bottom of your .zshrc should take care of this, making sure that our PATH override comes last.

If you find an issue that isn't listed here that doesn't necessarily mean you've found a bug. There can be a lot of variability across machines because of the impacts of other programs/libraries. The majority of the time bugs in that install process aren't a problem in the source code but are instead machine-specific  configuration details, so we recommend that you contact the ParFlow user email list (search for the text "Parflow-Users" on the GitHub page) for help. That should always be your first step when troubleshooting. 


Step 8) Listen to some great music while you run ParFlow

And we all know that the best music for any modeling is early 90’s era alt rock and grunge. I have nothing against other music, but the Academy of Stochastic and Statistical Environmental Simulations did a comprehensive study and reached that conclusion, so it’s science and in no way did I just make that up... Go for some of the lesser played stuff like the Screaming Trees, check out some early origins with Mother Love Bone or Mudhoney, or stick with a classic album like Core by the Stone Temple Pilots. Best of all, there’s always a band name from those days that fits how ParFlow is making you feel; sometimes it’s the Crash Test Dummies, other times it’s Nirvana, but if you get to feeling like Rage Against the Machine maybe it’s time to take a break. 

Friday, March 15, 2019

Installing ParFlow under MacOS 10.14 Mojave

The time has come for a new installation guide for ye ole macintosh computer. Why so long? Well, lots of reasons (and a few lame excuses) but mainly because there have been some changes on both the ParFlow and MacOS side over the last year. We wanted to ensure everything actually worked on a good sampling of systems before this update, so there was a lot of testing to do.

This post is addressed to a new ParFlow user or to an experienced user setting up a new computer. If you don't fit in that category, this will still be useful to you but you'll have to exercise some creative license to adapt the instructions to your needs. 



This post will describe a “typical” ParFlow installation, that is one that uses the most common options and configurations on macOS using the default bash shell. We’ll primarily use the command line for this so go ahead and open a terminal window (if you don’t know where terminal is use spotlight search).

Some Preliminaries

For commands entered into the terminal window I’ll use Courier font and the following convention where the command is preceded with “>>” for example:

>> make                       

OK, first thing’s first. We need to define a few environment variables for the installation. This may seem abstract right now but it’ll come in handy later and we’ll explain everything as it get used in the process. Navigate to your home folder and create (or edit) your bash_profile:

>> cd ~
>> vim .bash_profile

In vim, hit “i” to enter insert mode and add the following lines, preferably at the bottom:

export MPI_DIR=~/ParF/ OMPI-4.0.0
export MPI_RUN_DIR=~/ParF/OMPI-4.0.0/bin
export PARFLOW_DIR=~/ParF/parflow
export HYPRE_DIR=~/ParF/hypre
export SILO_DIR=~/ParF/silo
export CC=clang
export CXX=clang++
export FC=gfortran
export F77=gfortran

After that, this next line needs to be at the very bottom of this file:

export PATH=$MPI_RUN_DIR:$PATH


then press “escape” followed by “:” then “x” to save the changes and exit. To apply the changes:

>> source .bash_profile

So why did that do? We’ve just defined some “environment variables” and shortcuts we’ll take advantage of later on that will be loaded automatically when you start your computer or open a new terminal window. You may need to update some of these depending on how you decide to configure installation and I’ll refer to these environment variables as we come to them and point out where changes might be needed.

Next, we’ll create a directory to hold all the various bits that comprise a typical installation. From your home directory, make a folder called “ParF” and enter it:

>> cd ~                                    
>> mkdir ParF
>> cd ParF

There are 6 “parts” to these instructions and we’re going to handle each separately: 1) install gcc/gfortran, 2) install cmake, 3) install OpenMPI, 4) Install Silo, 5) Install Hypre, 6) install ParFlow. All except step 6 are supporting libraries or programs that ParFlow needs for our typical installation. 

Step 1) Installing gfortran and gcc

Truth be told we only really need gfortran but gcc is nice to have, and since they’re often bundled it makes sense to knock both out. You could try building them from source code, but the easier way is to use a binary that is pre-built for MacOS. Those can be downloaded here: http://hpc.sourceforge.net

The option you want is usually the first entry right below the word “Binaries” after the “big” paragraph. Today, for me that is “gcc-8.1-bin.tar.gz” and clicking on that link put it in my “Downloads” folder. You don’t need to bother with the “gfortran” link because it’s already included in this one. In your terminal window, we first need to enable compiling programs on your mac. This is necessary after every major MacOS upgrade before you can compile new programs (old one will be fine though).

>> sudo xcode-select --install

A confirmation window will likely open and after you OK through all of that, we’re done with that. You might also see a message saying command line tools are already installed, which is also fine.

Next, navigate to wherever you put that download and extract it to /usr/local with:

>> cd ~/Downloads                  
>> sudo tar -xvf gcc-8.1-bin.tar -C /

A lot of text will fly by then you’ll get a command prompt. To make sure this has copied correctly, at the command prompt type:

>> gfortran -v

And you should get something like this:
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/x86_64-apple-darwin17.5.0/8.1.0/lto-wrapper
Target: x86_64-apple-darwin17.5.0
Configured with: ../gcc-8.1.0/configure --enable-languages=c++,fortran
Thread model: posix
gcc version 8.1.0 (GCC)

A potentially really important note:
There is one last critically important step that you may need to take that was not necessary in the past. Depending on your system, as you work through the installation steps below you might encounter an error that says "error: cannot run C compiled program" which causes the installation to fail and stop. Usually this happens when using GCC instead of clang (which can happen accidentally sometimes) but fortunately there is a fix. You'll need to install the package in MacOS that has some of the libraries that are needed in order to compile programs. These are located in “/Library/Developer/CommandLineTools/Packages/” and the easiest way to get there is to open a finder window, press command+shift+g, then paste in that path. There is only one package in this folder (macOS_SDK_headers_for_macOS_10.14.pkg), though its name might change, and to install it just double click on it and follow the instructions. If you get errors, you may also need run:
>> sudo --xcode-select --s /Library/Developer/CommandLineTools

to make sure the system is pointing to the right version. These headers used to be included in macOS but they're being phased out. Further, Apple may be depreciating the package in future macOS releases, but we’ll update you if/when that happens. 

OK, time to take a quick break while you restart your system right meow. Seriously, go ahead and do it; these instructions will still be here when you’re done with that, and this could save you from a lot of headaches later on.

All that being said, if you use "clang" and "clang++" with the instructions given here you probably won't see that error and can skip this little step, but it's here in case you need it. Or you can do it now anyway since installing the headers doesn't break anything, even if they aren't used.  

Step 2) Installing cmake

Past installation instructions have used the GNU autoconfig system. We’ll still use that for some of the supporting components, but ParFlow itself is now built using the CMake interface. This isn’t installed on your mac by default, so we’ll start by adding it. This step might seem like a pain, but it’s pretty painless and simplifies the ParFlow build a lot.

We’re going to download the latest version of the CMake source. There is also a GUI for CMake but we’re not going to use it because it's tougher to explain. Before proceeding, check here:https://cmake.org/download/ for the latest version. On the download page, go ahead and click on the link for the archive in the Unix/Linux source box and the download should complete shortly. Assuming the archive is in your ~/Downloads/ folder, we’ll copy it into ParF, the expand the archive:  As of this writing that file is “cmake-3.14.0-rc1.tar.gz

To move that download directly into the ParF folder, in your terminal window, be sure you’re in the ParF directory, then type:

>> cd ~/ParF/
>> cp ~/Downloads/cmake-3.14.0-rc1.tar .
>> tar -xvf cmake-3.14.0-rc1.tar

Note the period at the end of the second line. You will need to update the file name as newer versions of CMake are released, or if you used a different download or ParF path. 

Now we’ll enter the new directory and start building using cmake’s helper:

>> cd cmake-3.14.0-rc1
>> ./bootstrap                         
>> make                                  
>> sudo make install                 

Those middle two steps will take a long time. At the end of the make process you’ll see percentage signs on the left edge of your screen and if it makes it to 100 without errors, you’re all set.

Important: Remember that sudo is the “nuclear option” on unix-esque computers and should only be used when absolutely necessary. It is here so that cmake will be available across your entire system, but this and installing gcc are the only places we need it.

Step 3) Installing OpenMPI

OpenMPI is the message passing interface we use for ParFlow (note that this is not the same as OpenMP), and this is what makes it parallel. Note that this section is a slight departure from past installation instructions. Even if you have an older version of OpenMPI, I recommend following these instructions to add a newer version to your system because there have been some changes and the newer versions tend to be more reliable. The following instructions will not overwrite or replace any existing MPI installations; to return to compiling/running with your existing OpenMPI version (if you have one), simply comment out the “export PATH” line we added to your .bash_profile, then source the file and you’re good to go. However, our goal is to install a new version, which is 4.0.0 as of this writing.

That aside, navigate back to your ParF directory and create a run folder for OpenMPI to run from (this folder should be the same as the environment variable we defined called MPI_DIR):

>> cd ~/ParF
>> mkdir OMPI-4.0.0

Next we’re going to download openmpi, build the package, then install it to our MPI_RUN directory.
 The simplest way to do that is from the command line using the “curl” command, which is like “wget” for a mac. However, before doing so you might want to check to make sure you’re downloading the most recent version at: https://www.open-mpi.org/software/ompi/You’ll need to modify the version numbers if using a more recent release but otherwise it’ll be just like this: 

>> curl "https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.0.tar.gz" -o "openmpi-4.0.0.tar.gz"

(Note: When copying and pasting, sometimes quotes get messed up. If you get an error from that command about libcurl, or some other strange result, just delete each quotation mark then add them back manually once you’re in the terminal window.)

Next, we’ll decompress and extract the archive in one step:

>> gunzip -c openmpi-4.0.0.tar.gz | tar xf -

Now we’ll configure the installation package for our system. Navigate into the newly created openmpi-4.0.0 folder and type:

>> ./configure --prefix=$MPI_DIR CC=/usr/bin/clang CXX=/usr/bin/clang++

Those CC and CXX arguments force the computer to use a specific compiler, which is clang. On most systems this should install fine without those flags, but it doesn’t hurt to have them. After you hit return, lots of stuff will fly by your screen for a long time. Once that’s done, type:

>> make
>> make install

The first command builds OpenMPI within the tree of the archive we downloaded (which will take forever and a decade before it completes), and the second copies it into the place we’ll be running it from. With that, OpenMPI is now installed. If you want to check, type “mpirun” at the command line, but not much will happen since it can’t find anything to do, or, for a lot more info than you probably want, you can run “ompi_info” and see your version; there should be 4.0.0 up at the top of the list of things on your screen. If you want to update OpenMPI in the future you’ll follow these same steps and create a new MPI_DIR (and MPI_RUN_DIR variable) directory for your version, whatever that might be. This approach prevents overwriting of OpenMPI versions and also saves you from having to uninstall it.

Step 4) Installing Silo

Silo is a file format that is used as the default for viewing ParFlow outputs, but we also now have other options available. Many of the test cases use it so we always recommend installing it. The latest version of silo can be found at: https://wci.llnl.gov/simulation/computer-codes/silo/downloads

You could download the archive in a web browser and move it but instead I’m going to use curl again. Navigate to the ParF folder and the commands are:

>> cd ~/ParF
>> curl "https://wci.llnl.gov/content/assets/docs/simulation/computer-codes/silo/silo-4.10.2/silo-4.10.2.tar.gz" -o "silo-4.10.2.tar.gz"

Mimicking the process we used for OpenMPI, to expand and extract the archive type:

>> gunzip -c silo-4.10.2.tar.gz | tar xf -

Now we’ll rename the newly expanded directory to match our environment variable SILO_DIR:

>> mv silo-4.10.2 silo

Now we’ll navigate into the new directory, configure, and install:

>> cd silo
>> ./configure --disable-silex
>> make install
>> cd ..

And that completes the installation.

Note that if your version of Silo differs, you’ll need to update the appropriate file/directory names environment variable we set to reflect your installation path. 

Step 5) Installing the HYPRE library

HYPRE is a library of numerical solvers for high-performance computing applications and ParFlow hooks into it for some of its functionality. 

As before the simplest way to get this is curl. There were some issues with hypre version 10 and ParFlow but the current release (2.11.2) or newer should work just fine. You can check for those here: https://computation.llnl.gov/projects/hypre-scalable-linear-solvers-multigrid-methods/software

Installing this is nothing more than a minor variation of what we’ve already done:

>> cd ~/ParF
>> curl "https://computation.llnl.gov/projects/hypre-scalable-linear-solvers-multigrid-methods/download/hypre-2.11.2.tar.gz" -o "hypre-2.11.2.tar.gz"

>> gunzip -c hypre-2.11.2.tar.gz | tar xf –
>> mv hypre-2.11.2 hypre

(That would vary if the version number changes, and your HYPRE_DIR variable would need to reflect your version/directory names). Now onto the setup. 

>> cd ~/ParF/ hypre/src
>> ./configure --prefix=$HYPRE_DIR 
>> make install

This installation also takes a while to install, but it’s usually not as long as OpenMPI. You’re likely to see a lot of warnings about “could not find any symbols” but as long as there are no errorsyou should be able to move on to ParFlow. 

Step 6) Installing ParFlow

This step is where we depart significantly from the previous installation instructions, but in a good way. The ParFlow build process is now more streamlined because it uses CMake. Instead of separate builds for pfsimulator (the model itself) and pftools (the controlling interface), there is one unified build process. There are a lot of different setup options for ParFlow but for our “typical” setup we’re assuming you want to run in parallel, you might want to run CLM, you have Silo and Hypre, and a few other little things. 

First, we need to download the code. ParFlow is hosted on GitHub (https://github.com/parflow/parflow) , but to obtain the code take this approach:

>> cd ~/ParF
>> git clone https://github.com/parflow/parflow.git --branch master --single-branch

which will create a folder called parflow in ParF and now you’ve got the source code. You may want to rename the folder (I like to use the date I downloaded it so /parflow.20190315 for example) but you’ll need to update your PARFLOW_DIR too if you change that. I’m going to proceed assuming you have not renamed it. 

Next, we’ll create a directory for the incremental files of the build process called “build”

>> mkdir build
>> cd build

The next step will tell CMake how to configure ParFlow using the typical options (copy and paste will be your friend on this):

>> cmake ../parflow -DCMAKE_INSTALL_PREFIX=$PARFLOW_DIR -DHYPRE_ROOT=$HYPRE_DIR -DPARFLOW_AMPS_LAYER=mpi1 -DPARFLOW_AMPS_SEQUENTIAL_IO=true -DPARFLOW_ENABLE_TIMING=true -DSILO_ROOT=$SILO_DIR -DPARFLOW_HAVE_CLM=ON 

Note that if your parflow source tree is NOT in a folder called parflow, you’ll need to change that right after “cmake” in the above command or things will go badly for you. The manual provides a few extra details on how you can customize an installation using the “ccmake” semi-GUI interface but we don’t need that now. Next type:

>> make

and lots of stuff will happen. If the last two lines resemble:

[100%] Linking C shared library libpftools.dylib
[100%] Built target pftools

Then you’re ready for the last command:

>> make install

The difference is that “make” builds everything in our temporary structure (the reason we created that build directory in the first), and “make install” copies the final build into our installation location. Think of it this way: cmake fully compiles the package in an isolated compartment, then, after that works, we copy it to the final destination.


On a few computers we’ve found that after the “make” command you might see something like this:

collect2: error: ld returned 1 exit status
make[2]: *** [pftools/libpftools.dylib] Error 1
make[1]: *** [pftools/CMakeFiles/pftools.dir/all] Error 2
make: *** [all] Error 2

If that’s the case, just back up and add-DCMAKE_SHARED_LINKER_FLAGS=-ltcl8.5to the end of that long list for the “cmake” command, then “make” again and you should be all sorted out.


And that’s it for the installation. The last step is to test the installation. Navigate to:

>> cd ~/ParF/parflow/test                    

Then type:

>> make check

A lot of things will fly by as ParFlow verifies that it is working correctly. Most should pass, but some will likely fail if you have fewer processors than are being tested; for example, some of the tests need nine processors and these will fail on a system with only 4. You’ll see a report at the end and as long as most pass, you’re all done and ParFlow has been installed successfully.


Step 7) Some troubleshooting tips - What do I do if:

- ...things aren't being found by my computer? The single most common reason steps of the installation don’t go well is not being in the correct path. You can always verify your location in the directory tree using the “pwd” command and each step tells you where you should be before you begin entering commands.

- ...things still aren't being found by the computer? Your mac is also case-sensitive so pay attention to capitalization. That also tends to be a wrench in the gears of the setup process. If you’re getting a bunch of strange behaviors, it’s a good idea to pause and restart your system before trying too many things.

- ...my system uses a different shell? If you prefer to use the c-shell instead of the bash-shell most of the steps are the same but your environment variables are defined differently. Fortunately, the setup example in the ParFlow manual outlines the c-shell variations of all the steps we covered here.

- ...my thesis is due tomorrow? Um, now is probably not the time to be re-installing ParFlow.