How to Parallelize a Foreach Loop in Java

Java 8: Parallel FOR loop

Read up on streams, they're all the new rage.

Pay especially close attention to the bit about parallelism:

"Processing elements with an explicit for-loop is inherently serial. Streams facilitate parallel execution by reframing the computation as a pipeline of aggregate operations, rather than as imperative operations on each individual element. All streams operations can execute either in serial or in parallel."

So to recap, there are no parallel for-loops, they're inherently serial. Streams however can do the job. Take a look at the following code:

    Set<Server> servers = getServers();
    Map<String, String> serverData = new ConcurrentHashMap<>();

    servers.parallelStream().forEach((server) -> {
        serverData.put(server.getIdentifier(), server.fetchData());
    });

How to parallelize for loop in java?

If you replace your for loops with Int Streams, it is very easy to make it run in parallel:

Example:

  IntStream.range(0, 10).parallel().forEach(i -> {
    System.out.println(i);
  });

Is there a easy way to parallelize a foreach loop in java?

A solution could be to launch every task in a Thread as follows:

new Thread(() -> object.doSomethingThatCanBeDoneInParallel(stuff, andStuff)).start();

but this is not a relevant solution as Thread creation is costly, so there are mechanisms and tools to help you: the Executors class to build some pools.

Once you have the instance that will manage this, you provide it with tasks, which will run in parallel, on the number of threads you choose:

void someFunction(SomeType stuff, SomeType andStuff) {
    ExecutorService exe = Executors.newFixedThreadPool(4);   // 4 can be changed of course
    for (Object object : lotsOfObjects) {
        exe.submit(() -> object.doSomethingThatCanBeDoneInParallel(stuff, andStuff));
    }

    // Following lines are optional, depending if you need to wait until all tasks are finished or not
    exe.shutdown();
    try {
        exe.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }   
}

Java 8: How can I convert a for loop to run in parallel?

I used the following code in Java 8 and it did the work. I was able to reduce the batch job to run from 28 minutes to 3:39 minutes.

IntStream.range(0, 100000).parallel().forEach(i->{
     restTemplate.exchange(url, HttpMethod.GET, request, String.class);
}
});

How to convert normal for loop to parallel stream in java

The compilation error in your current version is happening because a lambda expression is not allowed to make use of a local variable that is not either final or effectively final at the point that the lambda is created.

You could try fix it like this:

final List<FinPlan> = new ArrayList<>();

...

finPlanIdsList.parallelStream().forEach( planId -> {
        newList.add(retrieveFinPlan(planId));
});

... but that has the more subtle flaw that the code won't be thread-safe. The newList object is potentially going to be updated from different threads ... and ArrayList is not a thread-safe class.

A correct solution is this:

List<FinPlan> newList = 
    finPlanIdsList.parallelStream()
                  .map(planId -> retrieveFinPlan(planId))
                  .collect(Collectors.toList());

If it is essential that the result is an ArrayList, you can replace the Collectors.toList() expression with Collections.toCollection(ArrayList::new).

In general, it is a bad idea for a stream to rely on side-effects. In some cases you will get away with it. In others ... no.

Parallel.For for Java?

I guess the closest thing would be:

ExecutorService exec = Executors.newFixedThreadPool(SOME_NUM_OF_THREADS);
try {
    for (final Object o : list) {
        exec.submit(new Runnable() {
            @Override
            public void run() {
                // do stuff with o.
            }
        });
    }
} finally {
    exec.shutdown();
}

Based on TheLQ's comments, you would set SUM_NUM_THREADS to Runtime.getRuntime().availableProcessors();

Edit: Decided to add a basic "Parallel.For" implementation

public class Parallel {
    private static final int NUM_CORES = Runtime.getRuntime().availableProcessors();

    private static final ExecutorService forPool = Executors.newFixedThreadPool(NUM_CORES * 2, new NamedThreadFactory("Parallel.For"));

    public static <T> void For(final Iterable<T> elements, final Operation<T> operation) {
        try {
            // invokeAll blocks for us until all submitted tasks in the call complete
            forPool.invokeAll(createCallables(elements, operation));
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }

    public static <T> Collection<Callable<Void>> createCallables(final Iterable<T> elements, final Operation<T> operation) {
        List<Callable<Void>> callables = new LinkedList<Callable<Void>>();
        for (final T elem : elements) {
            callables.add(new Callable<Void>() {
                @Override
                public Void call() {
                    operation.perform(elem);
                    return null;
                }
            });
        }

        return callables;
    }

    public static interface Operation<T> {
        public void perform(T pParameter);
    }
}

Example Usage of Parallel.For

// Collection of items to process in parallel
Collection<Integer> elems = new LinkedList<Integer>();
for (int i = 0; i < 40; ++i) {
    elems.add(i);
}
Parallel.For(elems, 
 // The operation to perform with each item
 new Parallel.Operation<Integer>() {
    public void perform(Integer param) {
        System.out.println(param);
    };
});

I guess this implementation is really more similar to Parallel.ForEach

Edit
I put this up on GitHub if anyone is interested. Parallel For on GitHub

How do I parallellise a for loop, and save the results of each repetition in a matrix?

You can try something like parallel streams to do parallelism like one of the below lines.

Stream.of(allResults).parallel().forEach(i -> processData(i));

Arrays.stream(allResults).parallel().forEach(i -> processData(i));