Simplest way to do parallel replicate
You can just use the parallel versions of lapply
or sapply
, instead of saying to replicate this expression n
times you do the apply on 1:n
and instead of giving an expression, you wrap that expression in a function that ignores the argument sent to it.
possibly something like:
#create cluster
library(parallel)
cl <- makeCluster(detectCores()-1)
# get library support needed to run the code
clusterEvalQ(cl,library(MASS))
# put objects in place that might be needed for the code
myData <- data.frame(x=1:10, y=rnorm(10))
clusterExport(cl,c("myData"))
# Set a different seed on each member of the cluster (just in case)
clusterSetRNGStream(cl)
#... then parallel replicate...
parSapply(cl, 1:10000, function(i,...) { x <- rnorm(10); mean(x)/sd(x) } )
#stop the cluster
stopCluster(cl)
as the parallel equivalent of:
replicate(10000, {x <- rnorm(10); mean(x)/sd(x) } )
How do I parallelize a simple Python loop?
Using multiple threads on CPython won't give you better performance for pure-Python code due to the global interpreter lock (GIL). I suggest using the multiprocessing
module instead:
pool = multiprocessing.Pool(4)
out1, out2, out3 = zip(*pool.map(calc_stuff, range(0, 10 * offset, offset)))
Note that this won't work in the interactive interpreter.
To avoid the usual FUD around the GIL: There wouldn't be any advantage to using threads for this example anyway. You want to use processes here, not threads, because they avoid a whole bunch of problems.
What is the easiest way to parallelize a task in java?
I would recommend taking a look at ExecutorService.
In particular, something like this:
ExecutorService EXEC = Executors.newCachedThreadPool();
List<Callable<Result>> tasks = new ArrayList<Callable<Result>>();
for (final Object object: objects) {
Callable<Result> c = new Callable<Result>() {
@Override
public Result call() throws Exception {
return compute(object);
}
};
tasks.add(c);
}
List<Future<Result>> results = EXEC.invokeAll(tasks);
Note that using newCachedThreadPool
could be bad if objects
is a big list. A cached thread pool could create a thread per task! You may want to use newFixedThreadPool(n)
where n is something reasonable (like the number of cores you have, assuming compute()
is CPU bound).
Here's full code that actually runs:
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
public class ExecutorServiceExample {
private static final Random PRNG = new Random();
private static class Result {
private final int wait;
public Result(int code) {
this.wait = code;
}
}
public static Result compute(Object obj) throws InterruptedException {
int wait = PRNG.nextInt(3000);
Thread.sleep(wait);
return new Result(wait);
}
public static void main(String[] args) throws InterruptedException,
ExecutionException {
List<Object> objects = new ArrayList<Object>();
for (int i = 0; i < 100; i++) {
objects.add(new Object());
}
List<Callable<Result>> tasks = new ArrayList<Callable<Result>>();
for (final Object object : objects) {
Callable<Result> c = new Callable<Result>() {
@Override
public Result call() throws Exception {
return compute(object);
}
};
tasks.add(c);
}
ExecutorService exec = Executors.newCachedThreadPool();
// some other exectuors you could try to see the different behaviours
// ExecutorService exec = Executors.newFixedThreadPool(3);
// ExecutorService exec = Executors.newSingleThreadExecutor();
try {
long start = System.currentTimeMillis();
List<Future<Result>> results = exec.invokeAll(tasks);
int sum = 0;
for (Future<Result> fr : results) {
sum += fr.get().wait;
System.out.println(String.format("Task waited %d ms",
fr.get().wait));
}
long elapsed = System.currentTimeMillis() - start;
System.out.println(String.format("Elapsed time: %d ms", elapsed));
System.out.println(String.format("... but compute tasks waited for total of %d ms; speed-up of %.2fx", sum, sum / (elapsed * 1d)));
} finally {
exec.shutdown();
}
}
}
Simplest way to run three methods in parallel in C#
See the TPL documentation. They list this sample:
Parallel.Invoke(() => DoSomeWork(), () => DoSomeOtherWork());
So in your case this should just work:
Parallel.Invoke(
() => results.LeftFront.CalcAi(),
() => results.RightFront.CalcAi(),
() => results.RearSuspension.CalcAi(geom,
vehDef.Geometry.LTa.TaStiffness,
vehDef.Geometry.RTa.TaStiffness));
EDIT: The call returns after all actions have finished executing. Invoke()
is does not guarantee that they will indeed run in parallel, nor does it guarantee the order in which the actions execute.
Using -snowfall- (or other packages) to parallel repeating a user-defined function
sfSapply
expects fun
to be a function, but you hand over the result of one call to my.fun
. That is, you want to hand over my.fun
, not my.fun ()
.
Parallel Processing in python
A good simple way to start with parallel processing in python is just the pool mapping in mutiprocessing -- its like the usual python maps but individual function calls are spread out over the different number of processes.
Factoring is a nice example of this - you can brute-force check all the divisions spreading out over all available tasks:
from multiprocessing import Pool
import numpy
numToFactor = 976
def isFactor(x):
result = None
div = (numToFactor / x)
if div*x == numToFactor:
result = (x,div)
return result
if __name__ == '__main__':
pool = Pool(processes=4)
possibleFactors = range(1,int(numpy.floor(numpy.sqrt(numToFactor)))+1)
print 'Checking ', possibleFactors
result = pool.map(isFactor, possibleFactors)
cleaned = [x for x in result if not x is None]
print 'Factors are', cleaned
This gives me
Checking [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
Factors are [(1, 976), (2, 488), (4, 244), (8, 122), (16, 61)]
Related Topics
How to Make Object Created Within Function Usable Outside
Predicting Lda Topics for New Data
Kruskal-Wallis Test with Details on Pairwise Comparisons
Remove Spacing Around Plotting Area in R
Differences in Heatmap/Clustering Defaults in R (Heatplot Versus Heatmap.2)
Piecewise Regression with R: Plotting the Segments
Adding Custom Image to Geom_Polygon Fill in Ggplot
How to Save a Data Frame as CSV to a User Selected Location Using Tcltk
Avoid Rbind()/Cbind() Conversion from Numeric to Factor
Compute Rolling Sum by Id Variables, with Missing Timepoints
Comparing Two Columns in a Data Frame Across Many Rows
Handling Special Characters E.G. Accents in R
Loops in R - Need to Use Index, Anyway to Avoid 'For'
Obtaining Separate Summary Statistics by Categorical Variable with Stargazer Package