User Tools

Site Tools


wiki:grappolo_grass

HPC geographical data using GRASS

We created for GRASS a similar data processing scheme we used from the previous exercise ( HPC gdal data processing). Now the submitter script stay the same while the template GRASS script has a set of variable and GRASS environment settings and variable exports needed for accessing GRASS functions from each CPU.

The submitter script

This is needed for splitting data processing in multiple CPU.

| Grass_submitter.sh
#!/bin/bash
SCRIPTDIR=~/ost4sem/experiment/scripts
for file in  ~/usbstore1/input/20*.tif ; do
qsub -v file=$file $SCRIPTDIR/Grass_template.sh
done

The data processing template script

This is the chore script which is used for defining the actual data processing tasks.

| Grass_template.sh
#$ -S /bin/bash
#$ -o /home/pi/ost4sem/experiment/stdout
#$ -e /home/pi/ost4sem/experiment/stderr
 
export INPUTDIR=/home/pi/usbstore1/input
export OUTPUTDIR=/home/pi/usbstore2/output
export FILENAME=$(basename $file .tif)
export GRASSDB=/home/pi/ost4sem/grassdb
export LOCATION=location3035
export MAPSET=mapset_$FILENAME
 
################################  Setting GRASS variables for GRASS bash job
echo "LOCATION_NAME: $LOCATION"           >  $HOME/.grassrc6_$$
echo "MAPSET: PERMANENT"                  >> $HOME/.grassrc6_$$
echo "DIGITIZER: none"                    >> $HOME/.grassrc6_$$
echo "GRASS_GUI: text"                    >> $HOME/.grassrc6_$$
echo "GISDBASE: $GRASSDB"                 >> $HOME/.grassrc6_$$
###############################  path to GRASS binaries and libraries:
export GISBASE=/usr/lib/grass64
export PATH=$PATH:$GISBASE/bin:$GISBASE/scripts
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$GISBASE/lib"
export GISRC=~/.grassrc6_$$
export GIS_LOCK=$$  #   use process ID (PID) as lock file number
##############################   Start processing using grass
 
# gdalinfo $file > $OUTPUTDIR/$(basename $file .tif).txt
# gdalwarp -cutline $INPUTDIR/poly_fr.shp -crop_to_cutline $file $OUTPUTDIR/$CROPFILE
# gdalinfo $OUTPUTDIR/$CROPFILE > $OUTPUTDIR/$(basename $CROPFILE .tif).txt
 
g.mapset -c mapset=$MAPSET
r.external input=$file output=virtual_grass_map --overwrite
gdalinfo $file > $OUTPUTDIR/$FILENAME"before.txt"
g.region res=10000
r.mapcalc resampled=virtual_grass_map
r.out.gdal    input=resampled  format=GTiff output=$OUTPUTDIR/"test_GRASS_"$FILENAME.tif
gdalinfo  $OUTPUTDIR/"test_GRASS_"$FILENAME.tif > $OUTPUTDIR/$FILENAME"after.txt"
g.mapset mapset=PERMANENT
rm -r $GRASSDB/$LOCATION/$MAPSET
In between the r.external and the r.out gdal functions we could add a long procedure of data procesing routines.
The amount of data process we need to perform here is our processing chain bottle neck.
wiki/grappolo_grass.txt · Last modified: 2017/12/05 22:53 (external edit)