Error in Unserialize(Socklist[[N]]):Error Reading from Connection on Unix

How can I print when using %dopar%

There are a number of good solutions posted here, but I find it easiest to log to a socket and use a separate process to output the log calls in a console.

I use the following function:

log.socket <- make.socket(port=4000)

Log <- function(text, ...) {
  msg <- sprintf(paste0(as.character(Sys.time()), ": ", text, "\n"), ...)
  cat(msg)
  write.socket(log.socket, msg)
}

You can then place log statements in the code such as:

Log("Processing block %d of %d", i, n.blocks)

Log output can viewed in real-time using any simple socket listening tool. For example, using netcat on Linux:

nc -l 4000

The above log statement would display in the netcat terminal as:

2014-06-25 12:30:45: Processing block 2 of 13

This method has the advantage of working remotely and provides as detailed output as you care to log.

p.s. For those on Windows, see Jon Craton's netcat port.

p.p.s I'm guessing the write.socket R function probably isn't thread-safe, but unless you're logging at high frequency, you're unlikely to run into any issue. Something to be aware of though.

Get randomForest regression faster in R

Since you are using caret, you could use the method = "parRF". This is an implementation of parallel randomforest.

For example:

library(caret)
library(randomForest)
library(doParallel)

cores <- 3
cl <- makePSOCKcluster(cores)
registerDoParallel(cl)

dataset <- read.csv("/home/anonimo/Modelli/total_merge.csv", header=TRUE)
dati <- data.frame(dataset)
attach(dati)

trainSet <- dati[2:107570,]
testSet <- dati[107570:480343,]

# 3 times cross validation.
my_control <- trainControl(method = "cv", number = 3 )

my_forest <- train(Clip_pm25 ~ e_1 + Clipped_so + Clip_no2 + t2m_1 + tp_1 + Clipped_nh  +  Clipped_co + Clipped_o3 + ssrd_1 + Clipped_no + Clip_pm10 + sp_1, , 
                   data = trainSet,
                   method = "parRF",
                   ntree = 250,
                   trControl=my_control)

Here is a foreach implementation as well:

foreach_forest <- foreach(ntree=rep(250, cores), 
                          .combine=combine, 
                          .multicombine=TRUE, 
                          .packages="randomForest") %dopar%
   randomForest(Clip_pm25 ~ e_1 + Clipped_so + Clip_no2 + t2m_1 + tp_1 + Clipped_nh  +  Clipped_co + Clipped_o3 + ssrd_1 + Clipped_no + Clip_pm10 + sp_1, 
                   data = trainSet, ntree=ntree)

# don't forget to stop the cluster
stopCluster(cl)

Remember I didn't set any seeds. You might want to consider this as well. And here is a link to a randomforest package that also runs in parallel. But I have not tested this.

Related Topics