Ruby on Rails with IMAP IDLE for multiple accounts
There is no need to spawn a new thread for each IMAP session. These can be done in a single thread.
Maintain an Array (or Hash) of all users and their IMAP sessions. Spawn a thread, in that thread, send IDLE keep-alive to each of the connections one after the other. Run the loop periodically. This will definitely give you far more concurrency than your current approach.
A long term approach will be to use EventMachine. That will allow using many IMAP connections in the same thread. If you are processing web requests in the same process, you should create a separate thread for Event Machine. This approach can provide you phenomenal concurrency. See https://github.com/ConradIrwin/em-imap for Eventmachine compatible IMAP library.
Is it possible to thread pool IMAP connections?
IMAP typically uses SSL over TCP/IP. And a TCP/IP connection will need to be maintained per IMAP client connection, meaning that there will be many simultaneous open connections.
These multiple simultaneous connections can easily be maintained in a non-threaded (single thread) implementation without affecting the state of the TCP connections. You'll have to have some sort of a flow concept per IMAP TCP/IP connection, and store all of the flows in a container (a c++ STL map for instance) using the TCP/IP five-tuple (or socketFd) as a key. For each data packet received, lookup the flow and handle the packet accordingly. There is nothing about this approach that will affect the TCP nor IMAP connections.
Considering that this will work in a single-thread environment, adding a thread pool will only increase the throughput of the application, since you can handle data packets for several flows simultaneously (assuming its a multi-core CPU) You will just need to make sure that 2 threads dont handle data packets for the same flow at the same time, which could cause the packets to be handled out of order. An approach could be to have a group of flows per thread, maybe using IP pools or something similar.
Mature IMAP library
Sometimes you just need to know the right place to look. Many of the programming languages that run on Linux have their own website that lists open source libraries.
Here are some of the main sites, along some IMAP libraries that I found (for some languages there are many, many libraries available):
- Erlang (CEAN): erlmail
- Perl (CPAN): Net::IMAP::Client
- Python (PyPI): IMAPClient
- Ruby (RubyGems.org): imapget
- Haskell (HackageDB): HaskellNet
- Javascript/Node.js (NPM): imap
I can't offer any advice as to the quality of these libraries, and you may need to browse through the libraries available for each language before you find one that has the features you need, but hopefully these links will lead you to a solution.
Related Topics
Ruby Object Prints Out as Pointer
Iterate Over Days, Starting from X Date Through an End Date
How to Create a Symbol from a String That Has Whitespaces
How to Write a Rails Mixin That Spans Across Model, Controller, and View
How to Get the File Extension from a Url
How to Store Enum as String to Database in Rails
How to Detect User Agent in Rails 3.1
Deleting the Current Session with Rack::Session::Cookie
Convert Hash to Openstruct Recursively
Convert a Partial to Method/Block for Speed
Display Base64 Encoded Image in Rails
Ruby on Rails - Is Params a Method or a Hash
Elegantly Selecting Attributes from Has_Many :Through Join Models in Rails