Tuesday, August 11, 2015

Single File Outputs for CLM

by Jen Jefferson & Nick Engdahl
Keywords: Output files, CLM

Manual Location
Manual version: July, 2014 v.693
Single file outputs – Chapter 6.3
CLM variables – Page 137

On many super computers there is often a file storage limit (limiting the total file size or file count). Or, in a more practical sense, you might be seeking a way to reduce the number of files you have to deal with to analyze model output. Using the single file output option within PF-CLM you can reduce the number of .pfb output files. A “single file” contains variables output from CLM as well as the upper soil layer temperatures. If you do not use the single file output option you will see an individual file for each CLM variable for each time step. For example, if you run your model for 1 year (8.760 hours) and output data (i.e., 13 CLM variables) every hour you will end up with a total of 113,880 files (not even counting the other PF output files)! If you choose to use the single file output option you would have (only!) 8,760 files that would contain the same information as all of the individual files. Combined with some of the write suppressing options, like Solver.CLM.WriteLogs, this can be hugely beneficial for output file management.

You must add the following key to your TCL script to activate the single file output option:
     pfset Solver.CLM.SingleFile           True

Single files will appear with a “C” in the .pfb file name and are output for each time step (e.g., runname.out.clm_output.00001.C.pfb). Structurally, these files are formatted like any other PFB but the contents of each layer corresponds to a 2-D output parameter from CLM. Below is an example of single file output from a single column model simulation (1m x 1m x 10 m, dz = 0.1). Note: Because the model is a single column, there is only one value per row; when you have larger grids each row will contain nx by ny values.

1 1 23           nx=1, ny=1, nlayer=23 
                                            (# of variables contained in single file output)
4.805467e+01            eflx_lh_tot
3.818246e+02            eflx_lwrad_out
-6.762173e+00         eflx_sh_tot  b
1.199176e+01            elfx_soil_grnd
1.914223e-05     qflx_evap_tot
1.914223e-05     qflx_evap_grnd
1.914223e-05     qflx_evap_soi
0.000000e+00     qflx_evap_veg
0.000000e+00     qflx_tran_veg
-1.270223e-05    qflx_infl
0.000000e+00     swe_out
2.870696e+02            t_grnd
0.000000e+00     qflux_qirr
2.819986e+02            Temperature of bottom soil layer; recall that the initial ground temperature   in CLM is set to 300K, so if this value is 300K the model hasn’t been   “heated” enough for the deeper layers to change temperature.
2.820123e+02     Temperature of soil layer
2.820283e+02            Temperature of soil layer
2.820340e+02            Temperature of soil layer
2.820463e+02            Temperature of soil layer
2.821715e+02            Temperature of soil layer
2.826579e+02            Temperature of soil layer
2.838075e+02            Temperature of soil layer
2.855920e+02            Temperature of soil layer
2.870696e+02            Temperature of top soil layer; 
                                             This temperature should match t_grnd

There are a several approaches to access or see data from the C.pfb file (pseudocode shown below):

1.    Read the C.pfb files using a .pfb reader (readpfb) in R.

runname = "sfo_example"

for(k in 1:tottime){
   fin1=sprintf("%s.out.clm_output.%05d.C.pfb", runname,k)
   data_clm[k,1]=data1[1,1,5]     #qflx_evap_tot (mm/s)
   data_clm[k,2]=data1[1,1,12]    #t_grnd (K)
   data_clm[k,3]=data1[1,1,23]    #temperature of top soil layer

             [,1]     [,2]     [,3]
[1,] 1.914223e-05 287.0696 287.0696
[2,] 1.595737e-05 284.1664 284.1664
[3,] 1.116288e-05 282.7117 282.7117
[4,] 1.170877e-05 281.7334 281.7334

2.    Import the C.pfb into Matlab. In the ../parflow/pftools/prepostproc folder is a Matlab.m file that is setup for .pfb files. All you need to do in order for it to work is insert the contents of the file into an m-file and replace ‘PARFLOW.out.FILE.pfb’ with the file name of your output.

3.    Convert the C.pfb to a silo output file and view in VisIT.

4.    Convert the C.pfb to .txt file using TCL script.

lappend   auto_path $env(PARFLOW_DIR)/bin
package   require parflow
namespace import Parflow::*

pfset     FileVersion    4

set runname [lindex $argv 0]

puts "Convert PFB to TXT"
set Tend 4

#loop through variables and  write text files
for {set j 1} {$j <=$Tend} {incr j} {
            puts "File: $j"
            set fin [format $runname.out.clm_output.%05d.C.pfb $j]
            puts $fin
            set input [pfload $fin]
            set fout [format $runname.out.clm_output.%05d.C.txt $j]
            pfsave $input  -sa $fout
            pfdelete $input
puts "...Done with year $runname pressure file conversions"

Text files (.txt) will be created and contents will look similar to the example data displayed above.

5.    Use pfload and pfgetelt to query specific locations within the single file outputs. The location of the first entry in the single file (e.g., 23.729, from above) has a location of 0,0,0 (not 1,1,1) so you will need to subtract one from the layer number. Below is an example of a TCL script to obtain evapotranspiration data (i.e., qflx_evap_tot, which is in layer 5 (layer index of 4) in the single file output) for 4 time steps. If you want to see a different variable/layer just replace $layer for the layer of the CLM variable you want (minus one, of course).

lappend auto_path $env(PARFLOW_DIR)/bin
package require parflow
namespace import Parflow::*

pfset FileVersion 4

set runname sfo_example

# should be index - 1
set layer 4

set t_min 1
set t_max 4
set t_inc 1

# ------------------------------------------------------------
puts "----- Obtain Value from Single File Output -----"
puts "Total Evaporation for 4 time steps:"

for { set i $t_min } { $i <= $t_max } { incr i $t_inc } {
     set inname  [format "%s.out.clm_output.%05d.C.pfb" $runname $i]
     set data [pfload $inname]

     set v   [pfgetelt $data 0 0 $layer]

puts [format "%05d %f" $i $v]

The following information is displayed in the terminal window:

----- Obtain Value from Single File Output -----
Total Evaporation for 4 time steps:
00001 0.000019
00002 0.000016
00003 0.000011
00004 0.000012



Karthika Shree said...

Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us
Matlab Training in Chennai

Unknown said...

I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well. In fact your creative writing abilities has inspired me to start my own BlogEngine blog now. Really the blogging is spreading its wings rapidly. Your write up is a fine example of it.

click here

Selenium Training in Bangalore|
Selenium Training in Chennai

rohini said...

For Mobile phone,Laptop,Apple Products repairs.
Just visit our webpage.
apple service center chennai|lenovo service center chennai|acer service center in chennai|apple service center chennai

Thank You

manjuprabhu59 said...

Great blog! This blog gives good information to know more about mobile and other gadget products. Need more information like this for good technical information.
Motorola Service Center in Chennai
Honor Service Center in Chennai

Vijiaajith said...


Vijiaajith said...






merlin said...

It is really a great work and the way in which u r sharing the knowledge is excellent.Thanks for your informative article.

Selenium Training in chennai | Selenium Training in annanagar | Selenium Training in omr | Selenium Training in porur | Selenium Training in tambaram | Selenium Training in velachery