Does Concurrency Happen Even When Only One Thread Is in a Thread Pool

Does concurrency happen even when only one thread is in a thread pool?

Your assumption that multiple threads must be running concurrently just because the connection pool is being exhausted is not correct. Just because a connection is still 'checked out' from the connection pool doesn't mean that a query is currently being executed on the checked-out connection in a thread, it simply means that the thread's connection hasn't been checked back in. The thread could be sitting idle but still holding onto a connection from the connection pool as long as it hasn't been explicitly terminated.

Since ActiveRecord Connections are thread-local, you can exhaust the connection pool by running ActiveRecord queries on multiple threads, as you are doing in this case. (Every time Concurrent::FixedThreadPool.new(1) is called, a new thread is created.) Even if you're only running queries on a single thread at a time, by default a connection will still be held open on every thread until they are terminated.

To avoid this, you can either manually check in connections after using them, or ensure your threads are terminated (killed) so that their connections can be recovered (reaped) by the pool.

• To manually check in connections, refer to the ConnectionPool documentation for your options. The easiest way is to wrap your ActiveRecord code in a with_connection block:

  Concurrent::Promise.execute(executor: pool) do
    ActiveRecord::Base.connection_pool.with_connection do
      # update_attributes, etc
    end
  end
  

• To ensure all threads are terminated, call #shutdown followed by #wait_for_termination on the thread pool after you're finished using it:

  values = promises.map(&:value!)
  pool.shutdown
  pool.wait_for_termination
  

Why does thread pool takes tasks independenty and not concurrently?

Why not concurrently? Why not just time-slice those 10 tasks?

You can have a thread pool that is able to perform ten concurrent tasks. You just need to configure it to have at least ten worker threads. "Time-slicing" tasks is what threads do. What thread pools do is:

• Allow your program to control the number of threads that it uses to perform "background" tasks, and
• Allow your program to re-use threads, which can be much more efficient than creating a new thread for each new task, and then destroying the thread when the task is complete.

Thread concurrency issue even within one single command?

The "++" operator is not atomic -- it doesn't happen in one uninterruptible cycle. Think of it like this:

1. Fetch the old value
2. Add one to it
3. Store the new value back

So imagine that you get this sequence:

Thread A: Step 1
Thread B: Step 1
Thread A: Step 2
Thread B: Step 2
Thread A: Step 3
Thread B: Step 3

Both threads think they've incremented the variable, but its value has only increased by one! The second "store back" operation effectively cancels out the result of the first.

Now, truth is, when you add in multiple levels of cache, far weirder things can actually happen; but this is an easy explanation to understand. You can fix these kinds of issues by synchronizing access to the variable: either the whole run() method, or the inside of the loop using a synchronized block. As Jon suggests, you could also use some of the fancier tools in java.util.concurrent.atomic.

How can threads of execution be running concurrently when there is a thread scheduler?

However how can they be running concurrently with the existence of a thread scheduler?

They are not always running concurrently, the scheduler's job is to swap the running threads around so that they appear to be running concurrently. i.e. too fast for you to see.

The scheduler uses a time slice which is 0.1 ms. You can only see a flicker of 10 - 25 ms, so this is too fast for your to see, but it is quickly swapping threads so it appears there is concurrency.

e.g. you don't see movies jumping from one frame to the next. Each frame is changed every 1/42nd of a second so you think you see movement when actually to a high speed camera the screen would look jumpy.

If you have one logical CPU, all the thread are being swapped to one CPU. If you have multiple logical CPUs, a small set can be running at once and the rest have to wait.

What is the difference between concurrency and parallelism?

Concurrency is when two or more tasks can start, run, and complete in overlapping time periods. It doesn't necessarily mean they'll ever both be running at the same instant. For example, multitasking on a single-core machine.

Parallelism is when tasks literally run at the same time, e.g., on a multicore processor.

---

Quoting Sun's Multithreaded Programming Guide:

• Concurrency: A condition that exists when at least two threads are making progress. A more generalized form of parallelism that can include time-slicing as a form of virtual parallelism.

• Parallelism: A condition that arises when at least two threads are executing simultaneously.

Custom thread pool in Java 8 parallel stream

There actually is a trick how to execute a parallel operation in a specific fork-join pool. If you execute it as a task in a fork-join pool, it stays there and does not use the common one.

final int parallelism = 4;
ForkJoinPool forkJoinPool = null;
try {
    forkJoinPool = new ForkJoinPool(parallelism);
    final List<Integer> primes = forkJoinPool.submit(() ->
        // Parallel task here, for example
        IntStream.range(1, 1_000_000).parallel()
                .filter(PrimesPrint::isPrime)
                .boxed().collect(Collectors.toList())
    ).get();
    System.out.println(primes);
} catch (InterruptedException | ExecutionException e) {
    throw new RuntimeException(e);
} finally {
    if (forkJoinPool != null) {
        forkJoinPool.shutdown();
    }
}

The trick is based on ForkJoinTask.fork which specifies: "Arranges to asynchronously execute this task in the pool the current task is running in, if applicable, or using the ForkJoinPool.commonPool() if not inForkJoinPool()"

Java threads and number of cores

Processes vs Threads

In days of old, each process had precisely one thread of execution, so processes were scheduled onto cores directly (and in these old days, there was almost only one core to schedule onto). However, in operating systems that support threading (which is almost all moderns OS's), it is threads, not processes that are scheduled. So for the rest of this discussion we will talk exclusively about threads, and you should understand that each running process has one or more threads of execution.

Parallelism vs Concurrency

When two threads are running in parallel, they are both running at the same time. For example, if we have two threads, A and B, then their parallel execution would look like this:

CPU 1: A ------------------------->

CPU 2: B ------------------------->

When two threads are running concurrently, their execution overlaps. Overlapping can happen in one of two ways: either the threads are executing at the same time (i.e. in parallel, as above), or their executions are being interleaved on the processor, like so:

CPU 1: A -----------> B ----------> A -----------> B ---------->

So, for our purposes, parallelism can be thought of as a special case of concurrency*

Scheduling

But we are able to produce a thread pool(lets say 30) with a larger number than the number of cores that we posses(lets say 4) and have them run concurrently. How is this possible if we are only have 4 cores?

In this case, they can run concurrently because the CPU scheduler is giving each one of those 30 threads some share of CPU time. Some threads will be running in parallel (if you have 4 cores, then 4 threads will be running in parallel at any one time), but all 30 threads will be running concurrently. The reason you can then go play games or browse the web is that these new threads are added to the thread pool/queue and also given a share of CPU time.

Logical vs Physical Cores

According to my current understanding, a core can only perform 1 process at a time

This is not quite true. Due to very clever hardware design and pipelining that would be much too long to go into here (plus I don't understand it), it is possible for one physical core to actually be executing two completely different threads of execution at the same time. Chew over that sentence a bit if you need to -- it still blows my mind.

This amazing feat is called simultaneous multi-threading (or popularly Hyper-Threading, although that is a proprietary name for a specific instance of such technology). Thus, we have physical cores, which are the actual hardware CPU cores, and logical cores, which is the number of cores the operating system tells software is available for use. Logical cores are essentially an abstraction. In typical modern Intel CPUs, each physical core acts as two logical cores.

can someone explain how this works and also recommend some good reading on this?

I would recommend Operating System Concepts if you really want to understand how processes, threads, and scheduling all work together.

• The precise meanings of the terms parallel and concurrent are hotly debated, even here in our very own stack overflow. What one means by these terms depends a lot on the application domain.

What is the difference between concurrency, parallelism and asynchronous methods?

Concurrent and parallel are effectively the same principle as you correctly surmise, both are related to tasks being executed simultaneously although I would say that parallel tasks should be truly multitasking, executed "at the same time" whereas concurrent could mean that the tasks are sharing the execution thread while still appearing to be executing in parallel.

Asynchronous methods aren't directly related to the previous two concepts, asynchrony is used to present the impression of concurrent or parallel tasking but effectively an asynchronous method call is normally used for a process that needs to do work away from the current application and we don't want to wait and block our application awaiting the response.

For example, getting data from a database could take time but we don't want to block our UI waiting for the data. The async call takes a call-back reference and returns execution back to your code as soon as the request has been placed with the remote system. Your UI can continue to respond to the user while the remote system does whatever processing is required, once it returns the data to your call-back method then that method can update the UI (or handoff that update) as appropriate.

From the User perspective, it appears like multitasking but it may not be.

---

EDIT

It's probably worth adding that in many implementations an asynchronous method call will cause a thread to be spun up but it's not essential, it really depends on the operation being executed and how the response can be notified back to the system.

Threading and Concurrency Within A Servlet

Nonsense.

The JEE spec has lots of "should nots" and "thou shant's". The Servlet spec, on the other hand, has none of that. The Servlet spec is much more wild west. It really doesn't dive in to the actual operational aspects like the JEE spec does.

I've yet to see a JEE container (either a pure servlet container ala Tomcat/Jetty, or full boat ala Glassfish/JBoss) that actually prevented me from firing off a thread on my own. WebSphere might, it's supposed to be rather notorious, but I've not used WebSphere.

If the concept of creating unruly, self-managed threads makes you itch, then the full JEE containers internally have a formal "WorkManager" that can be used to peel threads off of. They just all expose them in different ways. That's the more "by the book-ish" mechanism for getting a thread.

But, frankly, I wouldn't bother. You'll likely have more success using the Executors out of the standard class library. If you saturate your system with too many threads and everything gets out of hand, well, that's on you. Don't Do That(tm).

As to whether an async solution is even appropriate, I'll punt on that. It's not clear from your post whether it is or not. But your question was about threads and Servlets.

Just Do It. Be aware it "may not be portable", do it right (use an Executor), take responsibility for it, and the container won't be the wiser, nor care.

Redis is single-threaded, then how does it do concurrent I/O?

Well it depends on how you define concurrency.

In server-side software, concurrency and parallelism are often considered as different concepts. In a server, supporting concurrent I/Os means the server is able to serve several clients by executing several flows corresponding to those clients with only one computation unit. In this context, parallelism would mean the server is able to perform several things at the same time (with multiple computation units), which is different.

For instance a bartender is able to look after several customers while he can only prepare one beverage at a time. So he can provide concurrency without parallelism.

This question has been debated here:
What is the difference between concurrency and parallelism?

See also this presentation from Rob Pike.

A single-threaded program can definitely provide concurrency at the I/O level by using an I/O (de)multiplexing mechanism and an event loop (which is what Redis does).

Parallelism has a cost: with the multiple sockets/multiple cores you can find on modern hardware, synchronization between threads is extremely expensive. On the other hand, the bottleneck of an efficient storage engine like Redis is very often the network, well before the CPU. Isolated event loops (which require no synchronization) are therefore seen as a good design to build efficient, scalable, servers.

The fact that Redis operations are atomic is simply a consequence of the single-threaded event loop. The interesting point is atomicity is provided at no extra cost (it does not require synchronization). It can be exploited by the user to implement optimistic locking and other patterns without paying for the synchronization overhead.

---

Related Topics