Wednesday, March 31, 2010

Running ParFlow on Windows using VirtualBox (also updated instructions for installing ParFlow in Ubuntu)


-->
The title of this post may be a little misleading. This is not actually an instruction to run ParFlow on Windows, but a quick and easy way for Windows users to have access to ParFlow on Ubuntu Linux using virtualization. Sun Microsystems’ VirtualBox is a free and open source virtualization package that allows additional operating systems to run inside another operating system as an application.
This post is divided into three parts. The first section will detail the steps for setting up VirtualBox in Windows. The second part provides instructions for installing and configuring Ubuntu Linux on the virtual machine. The third section is an updated set of instructions for building ParFlow on an Ubuntu system, applicable to both native and virtual Ubuntu installations.
I completed this installation using Microsoft Windows XP SP3 (32-bit), VirtualBox 3.1.4, and Ubuntu 9.10 (32-bit). If you are running a 64-bit version of Windows, you can run a 32-bit or 64-bit version of Ubuntu in your VirtualBox. 64-bit Ubuntu will not work unless you are running a 64-bit version of Windows. When in doubt, install the 32-bit version. These instructions detail the 32-bit installation, but I will make note where differences appear for 64-bit. Excluding download time, setting up a virtual Ubuntu machine takes a little more than an hour.

Installing VirtualBox and setting up your virtual machine
  1. Download the following:
    • Sun VirtualBox – select the version for Windows hosts
    • Ubuntu – for a 64-bit version, Click “Alternative download options” and select “64-bit version” before starting your download. The file is around 700 MB.
  1. Install VirtualBox. This is a standard Windows installation wizard. On Windows XP, you may see several “Logo Testing” failures, which you may ignore.
  1. Start VirtualBox. You will be required to register the software. I did this three weeks ago, and they haven’t sent me any emails to date.
  1. Click “New” in the upper left corner to start the wizard. You may name it anything you like. Select “Linux” in the Operating System dropdown, and select “Ubuntu” in the Version dropdown (or select Ubuntu (64-bit) for a 64-bit install)
  1. Choose how much memory to allocate to Ubuntu. You may choose up to half of the total memory of your computer. 512 MB is the minimum recommended. I went with 1200 MB
  1. Create a Virtual Hard Disk when prompted by the wizard, and check “Boot Hard Disk.”
    • You will be given the choice between dynamically expanding storage or fixed-size storage. The wizard provides a good explanation of each. I picked dynamically expanding.
    • Next, set the size (or maximum size for dynamically expanding) of your virtual hard disk. 10.00 GB will cover you for the operating system, ParFlow and support software, and it will leave room for a small amount of data. I went with 20.00 GB to make sure I have plenty of space for small runs. I would not recommend going under 10.00 GB.
  1. After you finish the wizard, create a shared folder. This will allow you to transfer files between your Ubuntu virtual machine and your Windows machine.
    • In Windows Explorer, create a folder that you want to use as your shared folder.
    • In the VirtualBox window, select the virtual machine you just created, click Settings, then click Shared Folders
    • Click the icon to add a folder (looks like a folder with a green ‘plus’ sign), and enter the path (or click other and navigate) to the folder you just created. “Folder name” is not the name you gave it in Windows. It is the name that will identify it in Ubuntu later. I called it “shared_folder.” Don’t make it read only: that would make it pretty useless.
    • Click OK, click OK.
  1. You’re almost ready to start installing Ubuntu. If you have ever installed Windows, this is painless in comparison. First, we must mount the Ubuntu iso file. The virtual machine will recognize this as a CD.
    • In the VirtualBox window, click File>Virtual Media Manager.
    • Click the CD/DVD Images tab, then click Add.
    • Find and select the Ubuntu image file you downloaded from ubuntu.com, and click OK.
    • Select your virtual machine again and click settings, then click Storage. Under Storage Tree, click the icon that looks a bit like a CD.
    • Under Attributes, select the Ubuntu image file from the CD/DVD Device dropdown list.
    • While you’re dealing with settings, click Display and check the box to “Enable 3D Acceleration.” This will eventually allow you to work in full screen.
Now you’re ready to configure and install Ubuntu on your virtual machine.

Installing and configuring Ubuntu on a Virtual Machine
  1. In the VirtualBox window, select your virtual machine and click Start. This should start installing Ubuntu. Simply follow the on-screen prompts. When you get the part about partitioning, make sure the virtual hard disk you created earlier is selected. If you know what you’re doing, you can set up your partitions manually. If not, select “Erase and use entire disk.” Don’t worry; this won’t touch anything but the virtual hard disk you created earlier, which exists only as a single file on your Windows machine.
  1. Installation may take 45 minutes to an hour, depending on how fast your computer is. When it’s done, you will be prompted to restart. Go ahead and restart. When it tells you to remove the CD from the drive, close the window.
  1. Back in the VirtualBox window, select your virtual machine, click Settings, click storage. Select the CD icon under Storage Tree, then select “Empty” from the CD/DVD Device dropdown list under Attributes. Click OK.
  1. Start the virtual machine again. When it has finished booting into Ubuntu, start setting up Ubuntu to access the shared folder you created in Windows.
    • Go to Devices>Install Guest Additions. Cancel the dialog box that appears (don’t worry if it doesn’t appear).
    • Open a terminal window (Applications>Accessories>Terminal) and enter the following commands:
sudo apt-get install dkms [you will be prompted for your password]
cd /media/cdrom
sudo ./VBoxLinuxAdditions-x86.run
(for a 64-bit installation, enter “sudo ./VBoxLinuxAdditions-amd64.run” instead)
sudo reboot
*If you can't get this to work correctly, try restarting the virtual machine, then try again. If it still won't work, see the comment at the bottom of this post.
  • When Ubuntu finishes rebooting, open a terminal window again and create a directory for your Windows/Ubuntu shared folder. I created mine in the home directory and called it “send_to_XP.”
mkdir send_to_XP
  • In step 7 of installing VirtualBox above, you named a folder for use in Ubuntu. I called mine “shared_folder.” Let’s see if it works. It should. When you input your directory in Ubuntu in the following command, it is best to use the full absolute path, to avoid putting it in the wrong place or overwriting something important (I tell you this because it happened to me).
sudo mount –t vboxsf shared_folder /home/john/send_to_XP
  • If that worked, you now have access to a folder shared between Ubuntu and Windows. Try it out. Stick a file in the folder in Windows. Then open the directory in Ubuntu (Places>Home Folder) and see if the file is in the “send_to_XP” folder (or whatever you decided to call it).
  1. You will have to run the above command every time you start Ubuntu in order to use your shared folder. Here’s a way to mount it automatically so you don’t have to run that command every time:
    • Open the “fstab” file in a text editor (vi and gedit are installed by default).
sudo gedit /etc/fstab
    • Copy the following line to end of the “fstab” file, changing the directory names to the ones you used and "john" to your own username .
shared_folder /home/john/send_to_XP vboxsf defaults,uid=john,gid=john,fmask=111,dmask=022 0 0
  1. You can skip this step and come back to it after you’ve installed ParFlow. It involves up to 2 hours of unattended downloading and updating.
By now, “[Update Manager]” should appear at the bottom of the Ubuntu window. Click it. Click “Check.” You’ll be prompted for your password. When the check finishes, click “Install Updates.” When it is finished, you will probably be prompted to restart. Go ahead and restart.

Installing Parflow in Ubuntu
The instructions in this section are applicable to building ParFlow on both native and virtual installations of Ubuntu. These instructions should be much easier and more effective than those in my previous post about installing ParFlow on Ubuntu.
  1. Open a terminal window.
  1. You will first need to download and install C, C++ and Fortran compilers and Tcl/Tk. The download list below also includes optional, but recommended text editors. Ubuntu ships with vim-lite, which sucks. At the very least, install the full version of vim. Downloading and installing these is very easy. All you have to do is enter the following commands:
sudo apt-get install gcc
sudo apt-get install g++
sudo apt-get install gdc
sudo apt-get install gfortran
sudo apt-get install tcl-dev
sudo apt-get install tk-dev
Optional
sudo apt-get install vim [highly recommended]
sudo apt-get install jedit [my favorite text editor – allows vertical selection]
  1. Now, download software from the internet. OpenMPI is optional if you are using a machine with a single-core processor. Silo is optional if you do not intend to use the VisIt visualization package. VisIt is also optional, but recommended as it is a good visualization package that lends itself well to use with ParFlow.
    • ParFlow
    • Hypre – A powerful solver package that greatly enhances the performance of ParFlow
    • Silo
    • VisIt
Choose a version for Ubuntu 8.04 (which will work for 9.10 also). Be sure to pick the correct 32-bit or 64-bit version.
    • OpenMPI – if you intend to utilize a multi-core processor, you will need this in order to do so.
  1. Build a directory structure for your ParFlow installation. It really helps to keep this organized, I’ve found, and it’s easy to do. Make a directory in your home directory for ParFlow and its supporting software. I called mine “parflow.” Then place the downloads from step 3 into that directory. If you’re not already working in the terminal window, now you must. I’ll use my directory structure as the example here.
    • Move into the new parflow directory and unpack the downloaded *.tar.gz files. If you’re new to linux command line operations, the [tab] button auto-completes your typing on the command line. I’ll use that in the commands below.
cd [takes you to your home directory]
cd parflow
tar –xvf parflow[tab]
tar –xvf hypre[tab]
tar –xvf silo[tab]
tar –xvf visit[tab]
tar –xvf openmpi[tab]
  • Each “tar” command creates a directory with version numbers at the end of the name (except visit). I renamed the parflow, hypre and silo directories to simply “parflow,” “hypre,” and “silo” for simplicity.
  1. Set environmental variables. This makes the installation a breeze, when we get to it. You can do this on the command line, but it is better to set them in the .bash_aliases file. Then you only have to do it once.
Depending on your text editor of choice:
vi ~/.bash_aliases
jedit ~/.bash_aliases or
gedit ~/.bash_aliases
Add the following lines, adjusted for your own directory structure, of course.
export CC=gcc
export CXX=g++
export FC=gfortran
export F77=gfortran

export PARFLOW_DIR=/home/john/parflow/parflow

export SILO_DIR=/home/john/parflow/silo
export HYPRE_DIR=/home/john/parflow/hypre
export PATH=$PATH:/home/john/parflow/visit/bin

Now it’s time to start installing. But first, source your bash aliases
source ~/.bash_aliases
  1. Install OpenMPI
cd openmpi[tab]
sudo ./configure
sudo make all
sudo make check
sudo make all install
cd ..
  1. Install Silo
cd silo
./configure --prefix=$SILO_DIR --disable-silex

make all
make check
make all install
cd ..
  1. Install Hypre
cd hypre/src
./configure --prefix=$HYPRE_DIR --with-MPI [if you’re not using MPI, use --without-MPI]
make all
make check
make all install
cd ../..
  1. Install ParFlow and PFTools, run the test cases
cd parflow/pfsimulator
./configure --prefix=$PARFLOW_DIR --enable-timing --with-clm --with-silo=$SILO_DIR --with-hypre=$HYPRE_DIR --with-amps=mpi1 [if you’re not using MPI, leave the last flag off]
make install
cd ../pftools
./configure --prefix=$PARFLOW_DIR --with-silo=$SILO_DIR --with-amps=mpi1 [if you’re not using MPI, leave the last flag off]
make install
cd ../test
sudo ldconfig
make check

Wednesday, March 10, 2010

Building ParFlow on OSX 10.6.x Snow Leopard

Snow Leopard has a number of nice features but because it is 64 bit there are a few tricks to compiling ParFlow and associated libraries. Here's step-by-step tips to get everything built.

Compilers:
1) gcc, I use the gcc that comes w/ X-Code Version 3.2.1 available free from Apple.
Its' version statement is:
i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5646) (dot 1)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.


2) mpi, mpicc (an mpi-wrapper for c) comes packaged w/ X-Dev. I've found this works just fine with ParFlow.

3) gfortran, The only 64-bit compatible gfortran I've found, pre-built, is here:http://r.research.att.com/tools/
Its' version statement is:
GNU Fortran (GCC) 4.2.3
Copyright (C) 2007 Free Software Foundation, Inc.
GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of GNU Fortran
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING


4) Libraries, I download and build SILO and HYPRE libraries from scratch.
SILO: https://wci.llnl.gov/codes/silo/silo-4.7.2/silo-4.7.2.tar.gz
HYPRE: v. 2.4.0b https://computation.llnl.gov/casc/hypre/software.html


5) Configure Lines
I find that all packages need to be told to build in 64 bit. This is usually done w/ an '-arch x86_64' flag in the configure line.
For SILO:
setenv SILO_DIR path-to-silo/silo-4.7.2/
or for bash
export SILO_DIR=path-to-silo/silo-4.7.2/

./configure CC='gcc -arch x86_64' CXX='g++ -arch x86_64' F77='gfortran -arch x86_64' FC='gfortran -arch x86_64' --disable-silex
make install

For HYPRE
setenv HYPRE_DIR path-to-hypre/hypre-2.4.0b/
or for bash

export HYPRE_DIR=path-to-hypre/hypre-2.4.0b/

./configure CC='gcc -arch x86_64' CXX='g++ -arch x86_64' F77='gfortran -arch x86_64' FC='gfortran -arch x86_64' --prefix=$HYPRE_DIR
make install

PARFLOW
I'm assuming you've set environmental variables PARFLOW_DIR to point to where PF is installed, SILO_DIR to point to SILO and HYPRE_DIR to point to hypre as above.
1) cd $PARFLOW_DIR/pfsimulator
2) ./configure CC='gcc -arch x86_64' F77='gfortran -arch x86_64' FC='gfortran -arch x86_64' --prefix=$PARFLOW_DIR --with-clm --enable-timing --with-silo=$SILO_DIR --with-hypre=$HYPRE_DIR --with-amps=mpi1
3) make install

The compilation process goes fine to almost the end, then errors out w/
ld: duplicate symbol start in /usr/lib/crt1.o and /usr/lib/crt1.10.6.o

These steps will fix this:
1) cd config
2) manually edit Makefile.config to change crt1.o in LDLIBS to crt1.10.6.o
3) cd ..
4) make install

PFTOOLS
1) cd $PARFLOW_DIR/pftools
2) ./configure CC='gcc -arch x86_64' F77='gfortran -arch x86_64' FC='gfortran -arch x86_64' --prefix=$PARFLOW_DIR --enable-timing --with-silo=$SILO_DIR --with-amps=mpi1
3) make install
The build process is very similar, it errors out on same linking step/error duplicate libraries. Here is how to fix:
1) cd config
2) manually change Makefile.config to reflect crt1.10.5.o to crt1.10.6.o this time in two places in the LDLIBS line (it might also be crt1.o in two places as above depending on the system)
3) make install
This now almost seems to work, but errors with an odd message at the end during the linking step
4) Manually adding the -arch x86_64 to the end of the tools linking step will fix this, you can link manually by typing:

ld -dylib pftappinit.o printdatabox.o readdatabox.o databox.o error.o velocity.o head.o flux.o diff.o stats.o tools_io.o axpy.o getsubbox.o enlargebox.o load.o usergrid.o grid.o region.o file.o pftools.o top.o water_balance.o water_table.o sum.o -o ./bin/parflow.dylib -L/usr/lib -L$PARFLOW_DIR/lib -lparflow -L$SILO_DIR/lib -lsilo -ltcl -lc -arch x86_64

from the command line.
5) make install
Everything should now work and you can cd $PARFLOW_DIR/test and make check to test that everything is working.

-Reed

ParFlow Movie Library on YouTube

Hi Folks-

We have a ParFlow Channel on YouTube, it will include a wide range of simulations-- check it out:
http://www.youtube.com/user/TheParFlow

If you have things you would like to upload, please let us know.

-Reed