User Tools

Site Tools


wiki:grappolo_grass

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

wiki:grappolo_grass [2017/12/05 22:53] (current)
Line 1: Line 1:
 +====== HPC geographical data using GRASS ======
 +We created for GRASS a similar data processing scheme we used from the previous exercise ( HPC gdal data processing). Now the submitter script stay the same while the template GRASS script has a set of variable and GRASS environment settings and variable exports needed for accessing GRASS functions from each CPU.\\
  
 +===== The submitter script =====
 +This is needed for splitting data processing in multiple CPU.\\
 +
 +<code bash | Grass_submitter.sh>​
 +#!/bin/bash
 +SCRIPTDIR=~/​ost4sem/​experiment/​scripts
 +for file in  ~/​usbstore1/​input/​20*.tif ; do
 +qsub -v file=$file $SCRIPTDIR/​Grass_template.sh
 +done
 +</​code>​
 +
 +===== The data processing template script =====
 +This is the chore script which is used for defining the actual data processing tasks.\\
 +
 +<code bash | Grass_template.sh>​
 +#$ -S /bin/bash
 +#$ -o /​home/​pi/​ost4sem/​experiment/​stdout
 +#$ -e /​home/​pi/​ost4sem/​experiment/​stderr
 +
 +export INPUTDIR=/​home/​pi/​usbstore1/​input
 +export OUTPUTDIR=/​home/​pi/​usbstore2/​output
 +export FILENAME=$(basename $file .tif)
 +export GRASSDB=/​home/​pi/​ost4sem/​grassdb
 +export LOCATION=location3035
 +export MAPSET=mapset_$FILENAME
 +
 +################################ ​ Setting GRASS variables for GRASS bash job
 +echo "​LOCATION_NAME:​ $LOCATION" ​          > ​ $HOME/​.grassrc6_$$
 +echo "​MAPSET:​ PERMANENT" ​                 >> $HOME/​.grassrc6_$$
 +echo "​DIGITIZER:​ none" ​                   >> $HOME/​.grassrc6_$$
 +echo "​GRASS_GUI:​ text" ​                   >> $HOME/​.grassrc6_$$
 +echo "​GISDBASE:​ $GRASSDB" ​                >>​ $HOME/​.grassrc6_$$
 +############################### ​ path to GRASS binaries and libraries:
 +export GISBASE=/​usr/​lib/​grass64
 +export PATH=$PATH:​$GISBASE/​bin:​$GISBASE/​scripts
 +export LD_LIBRARY_PATH="​$LD_LIBRARY_PATH:​$GISBASE/​lib"​
 +export GISRC=~/​.grassrc6_$$
 +export GIS_LOCK=$$ ​ #   use process ID (PID) as lock file number
 +############################## ​  Start processing using grass
 +
 +# gdalinfo $file > $OUTPUTDIR/​$(basename $file .tif).txt
 +# gdalwarp -cutline $INPUTDIR/​poly_fr.shp -crop_to_cutline $file $OUTPUTDIR/​$CROPFILE
 +# gdalinfo $OUTPUTDIR/​$CROPFILE > $OUTPUTDIR/​$(basename $CROPFILE .tif).txt
 +
 +g.mapset -c mapset=$MAPSET
 +r.external input=$file output=virtual_grass_map --overwrite
 +gdalinfo $file > $OUTPUTDIR/​$FILENAME"​before.txt"​
 +g.region res=10000
 +r.mapcalc resampled=virtual_grass_map
 +r.out.gdal ​   input=resampled ​ format=GTiff output=$OUTPUTDIR/"​test_GRASS_"​$FILENAME.tif
 +gdalinfo ​ $OUTPUTDIR/"​test_GRASS_"​$FILENAME.tif > $OUTPUTDIR/​$FILENAME"​after.txt"​
 +g.mapset mapset=PERMANENT
 +rm -r $GRASSDB/​$LOCATION/​$MAPSET
 +
 +</​code>​
 +
 +
 +
 +
 +<note tip>
 +In between the **r.external** and the **r.out gdal** functions we could add a long procedure of data procesing routines. ​
 +</​note>​
 +
 +<note important>​
 +The amount of data process we need to perform here is our processing chain bottle neck.
 +</​note>​
wiki/grappolo_grass.txt ยท Last modified: 2017/12/05 22:53 (external edit)