Saving Multiple Outputs of Foreach Dopar Loop

Saving multiple outputs of foreach dopar loop

Don't try to use side-effects with foreach or any other parallel program package. Instead, return all of the values from the body of the foreach loop in a list. If you want your final result to be a list of two lists rather than a list of 100,000 lists, then specify a combine function that transposes the results:

comb <- function(x, ...) {
  lapply(seq_along(x),
    function(i) c(x[[i]], lapply(list(...), function(y) y[[i]])))
}

oper <- foreach(i=1:10, .combine='comb', .multicombine=TRUE,
                .init=list(list(), list())) %dopar% {
  list(i+2, i+3)
}

oper1 <- oper[[1]]
oper2 <- oper[[2]]

Note that this combine function requires the use of the .init argument to set the value of x for the first invocation of the combine function.

outptut two objects using foreach

If you need to return two objects from the body of the foreach loop, you must bundle them into one object somehow or other, and a list is the most general way to do that. The trick is to provide an appropriate combine function to achieve the desired final result. If you want to combine all of the vec1 objects with cbind, and also all of the vec2 objects with cbind, the mapply function is quite handy. I think this is what you want:

comb <- function(...) {
  mapply('cbind', ..., SIMPLIFY=FALSE)
}

Here's a little test program for this combine function:

result <- foreach(i=1:100, .combine='comb', .multicombine=TRUE) %dopar% {
  vec1 <- rep(i, 10)
  vec2 <- rep(2*i, 10)
  list(vec1, vec2)
}

This will return a list containing two, 10 X 100 matrices, but the same combine function can be used if vec1 and vec2 are data frames.

R: doParallel foreach with several data frames outputs

It seems that your problem has nothing to do with parallelism, but rather about combining the results.

An example of solution of how I would do it (which I think is the most efficient way to do it):

library(foreach)
tmp <- foreach(i = seq_len(32)) %do% {
  list(iris[i, ], mtcars[i, ], iris[i, ])
}

lapply(purrr::transpose(tmp), function(l) do.call(rbind, l))

Save multiple results of foreach loops in R

concerning store results(error, model1_b and model2_b) in a foreach loop, the answer can be found at Saving multiple outputs of foreach dopar loop

and %dorng% might be used instead of using %dopar%.

R - is there a way to create a nested %dopar% foreach loop?

Foreach maintainer here.

Use the %:% operator:

registerDoParallel(60)

out2 <- foreach(i = 1:length(number_of_daily_images)) %:% 
  foreach(j = 1:length(number_of_pixels_in_each_image)) %dopar% {

    # perform some calculations
    something(i, j)

  }

R foreach() - how to define combine function for multiple outputs when using itertools chunks?

The idea is that you can use .combine=c to append the lists returned in chunks
(so that you don't get nested lists),
and then adjust the structure in the way you were doing without itertools
(but simplified a bit):

lists <- foreach(thisIter=isplitIndices(10L, chunks=num_cores), .combine=c) %dopar% {
    lapply(thisIter, function(i) {
        c(i * 1L, 
          i * 10L,
          i * 100L)
    })
}

first_output_list <- lapply(lists, "[", 1L)
second_output_list <- lapply(lists, "[", 2L)
third_output_list <- lapply(lists, "[", 3L)

---

Related Topics