User Tools

Site Tools


Running R with multiple cores

If you need to recursively process an R function or an R script n times, you can extract the for loop in bash and process each R function or script with a different CPU. How to do it:

  1. built a template R script
  2. built a template bash script
  3. use xargs bash fonction in a loop for processing each step of the loop with a different CPU

Create a ~/ost4sem/exercise/multicore and a …/exercise/multicore/output folders if you do not have them.

1. Example of a template R script

Save the R script below as ~/ost4sem/exercise/multicore/templateR.R

mod.t = Sys.getenv(c('mod','t','path'))
mod = as.numeric(mod.t[1])
t = as.numeric(mod.t[2])
path = as.character(mod.t[3])
my_output=paste("We have processed  mod ",mod," and t ",t," ---- output path is   ",path,sep=" ")

2. Example of a bash script to exchange variables with R

Save the template bash script below as ~/ost4sem/exercise/multicore/

cd ~/ost4sem/exercise/multi_core
export mod
export t
export path
R --vanilla --no-readline  -q    <    template_R.R
- - vanilla
is used to combine –no-save, –no-restore, –no-site-file, –no-init-file and –no-environ

3. Process the R script using the template bash within a xargs loop

cd ~/ost4sem/exercise/multi_core
for mod in 1 2 3 4 ; do 
for t in `seq 1 5` 
do echo $mod $t 
done| xargs -n 2 -P 2 bash
in xargs
-n 2 refer to the number of variables exchanged between xargs in the bash loop and the script
-P 2 indicate the number of procssors you want to use. The
script need to be in your working environment( ~/ost4sem/exercise/multi_core ). </note>
wiki/cluster_r_xargs.txt · Last modified: 2021/01/20 20:36 (external edit)