User Tools

Site Tools


Mapping the schools of Cornwall from online web pages

This is an example of computational social science applied to spatial analysis.

Computational social science is the science that investigates social phenomena through the medium of computing and related advanced information processing technologies.
There is an increasing interest of this discipline and related applications in a broad fields of interests.
Here we show how to use the web to withdrowal a list of schools in a particular area, cratate a list of post codes, link post codes to lat long data to produce a map.

Computational implementation

mkdir ~/school_sh
cd ~/school_sh
wget --output-document=/home/ste/school_sh/school.html
awk -F"<tr class=\"odd\">" '{if(NF>10)print $0}' school.html > cleaned.txt
awk   '{gsub (/<td class="txt"><a /,"\n") ; print $0}' cleaned.txt > tmp1.txt
awk   -F"::" '{gsub (/href="/,"::") ;  gsub (/">/,"::") ; gsub (/<\/a>/,"::") gsub(/<\/td><td/,"::")  ;  if($6!="") print NR","$2","$3","$6  }' tmp1.txt > tmp2.txt
for x in $(seq 2 $(wc -l tmp2.txt | awk '{print $1}')) ; do 
mylink=$(awk -v x=$x -F"," '{ if($1==x)  print $2}' tmp2.txt) 
myschool=$(awk -v x=$x -F"," '{ if($1==x)  print $3}' tmp2.txt)
mytown=$(awk -v x=$x -F"," '{ if($1==x)  print $4}' tmp2.txt)
wget --output-document=/home/ste/school_sh/tempfile.txt  $mylink
postID=$(grep "<br />" tempfile.txt | awk -F, '{gsub(/</,","); gsub(/>/,",") ; if(NR==4) print $3}')
export POST=$(echo $postID | awk '{print $1 $2}')
/usr/bin/php << 'EOF'
$input = getenv("POST");
$data = file_get_contents("$key&p=$input&output=text&want=ll-wgs84");
$file = 'latlong.txt';
file_put_contents($file, $data );
latlong=$(awk -F, '{print$3","$4}' latlong.txt)
echo $latlong,$POST,$myschool,$mytown,$mylink >> school_cornwall.csv
to get YOURKEYNUMBER you need to access and request it
wiki/exercise_cornwallschool.txt · Last modified: 2017/12/05 22:53 (external edit)