Impact of Hundreds of Idle Threads

Impact of hundreds of idle threads

Each thread has overhead - most importantly each one has its own stack and TLS. Performance is not that much of a problem since they will not get any time slices unless they actually do anything. You may still want to consider using thread pools.

Threads sitting idle = bad?

Asynchronous I/O basically means that your application does most of the thread scheduling. Instead of letting the OS randomly suspend your thread and schedule another one, you have only as many threads as there are CPU cores and yield to other tasks at the most appropriate points—when the thread reaches an I/O operation, which will take some time.

The above seems like a clear win from the performance standpoint, but the asynchronous programming model is much more complex in several regards:

1. it can't be expressed as a single function call so the workflow is not obvious, especially when transfer of control flow due to exceptions is considered;
2. without specifically targetted support from the programming language the idioms are very messy: spaghetti code and/or extremely weak signal-to-noise ratio are the norm;
3. mostly due to 1. above debugging is much more difficult as the stack trace does not represent the progress within a unit of work as a whole;
4. execution jumps from thread to thread within a pool (or even several pools, where each layer of abstraction has its own) so profiling and monitoring with the usual tools is rendered virtually useless.

On the other hand, many favorable improvements and optimizations have happened to the modern OS's which mostly eliminate the performance downsides of synchronous I/O programming:

• the address space is huge, so space reserved for stacks isn't a problem;
• the actual physical RAM load of call stacks is not very large as only the part of the stack actually used by a thread is committed to RAM, and a call stack doesn't normally exceed 64K;
• context switching, which used to be prohibitively expensive for larger thread counts, has been improved to the point where its overhead is negligible for all practical purposes.

A classical paper going through much of the above and some other points is a good complement to what I am saying here:

https://www.usenix.org/legacy/events/hotos03/tech/full_papers/vonbehren/vonbehren_html/index.html

Drawbacks to an idling fixed threadpool

No, there are no drawbacks.

The threads won't be actually running, they'll be parked until a new task is submitted. So it does not affect CPU. Also you say that you will use this pool again, so in your case there's no point of shutting it down and recreating again.

As to the memory - yes, idle threads will consume some amount of memory, but that's not an issue as well, until you have hundreds (thousands?) of threads.

Also, a piece of advice. Do not do premature optimizations. That's the root of all evil. Analyze a problem once you have real performance issues, using special utilities for that and detecting bottlenecks.

Multi-Threading: At what point have you created too many threads?

You've created too many threads when the overall performance of your application degrades or the impact on other applications running on that same box are negatively affected to an unacceptable level.

The point is that there is no absolute answer.

One application I've been working on uses a thread pool of 1000 threads and for what we're doing, that seems to be the right number. In one configuration we didn't limit it and it went up to 30,000+ and basically brought the machine to a grinding halt.

You basically have to performance test it and have enough monitoring/instrumentation to determine the overall throughput of your application, resource usage, thread utilization and know how idle threads were and how long work was waiting on queues to be picked up. You then tune as necessary.

One cautionary note: think very carefully before you add another layer of threads. As I'm sure you know, writing multithreaded code is hard. Try to keep it as simple as possible. Adding another layer is a risky step.

---

Related Topics