Ruby: Any Gems for Threadpooling

Ruby: Any gems for threadpooling?

From my experience forking/process pooling is much more effective than thereadpooling in Ruby (assuming you do not need much in terms of thread communication). Some time ago I created a gem called process_pool, which is a very basic process pool with a file based job queue (you can check it out here: http://github.com/psyho/process_pool).

Ruby thread pool

You need to check the thread pool size , clean old thread etc.
Give the code below a try. That's the general way but as suggested above , https://rubygems.org/gems/threadpool/versions/0.1.2 should be used instead of writing something inefficient like the example below.

class ThreadsPool
  attr_accessor :threads, :limit
  def initialize(limit = 1, sleep_time = 0.1)
    @limit      = limit
    @threads    = []
    @sleep_time = sleep_time
  end

  def can_add_thread?
    @threads.size < @limit
  end

  def wait_for_spot
    until can_add_thread?
      clean_threads
      sleep @sleep_time
    end
  end

  def add_thread(thread)
    thread.abort_on_exception = true
    @threads << thread
  end

  def clean_threads
    @threads.delete_if { |t| !t.alive? }
  end

  def join_all
    @threads.each(&:join)
  end
end

stack_asgs = [5,6,7]

thread_pool = ThreadsPool.new(2)
stack_asgs.each do |this_asg|
  thread_pool.wait_for_spot
  puts "this_asg -> #{this_asg}"
  thread = Thread.new { sleep this_asg ; puts "end of sleep #{this_asg} | #{DateTime.now}" }
  thread_pool.add_thread(thread)
end
thread_pool.join_all

Ruby Thread Pooling - What am I doing wrong?

I've used the ruby-thread gem which add's pool support like so:

require 'thread/pool'

pool = Thread.pool(50)

Content.all.each do |content|
  pool.process do
    grab_and_store(content)
  end
end

pool.shutdown

Ruby 1.9 thread pools

You are correct in that the default C Ruby interpreter only executes one thread at a time (other C based dynamic languages such as Python have similar restrictions). Because of this restriction, threading is not really that common in Ruby and as a result there is no default threadpool library. If there are tasks to be done in parallel, people typically uses processes since processes can scale over multiple servers.

If you do need to use threads, I would recommend you use https://github.com/meh/ruby-threadpool on the JRuby platform, which is a Ruby interpreter running on the JVM. That should be right up your alley, and because it is running on the virtual machine it will have true threading.

Why doesn't Ruby have a ThreadPool built-in?

Most likely the reason is because ruby doesn't have "real" threads. It has what are called Green threads. The ruby interpreter takes care of scheduling execution threads without using any underlying OS threads. This effectively makes Ruby single threaded.

Ruby threading/forking with API (Sinatra)

I believe that better way to do it - is to use background jobs. While your worker executes some long-running tasks, it is unavailable for new requests. With background jobs - they do the work, while your web-worker can work with new request.

You can have a look at most popular backgroung jobs gems for ruby as a starting point: resque, delayed_jobs, sidekiq

UPD: Implementation depends on chosen gem, but general scheme will be like this:

# Controller
post '/items' do
  # Processing data
  MyAwesomeJob.enqueue # here you put your job into queue
  head :ok # or whatever
end

In MyAwesomejob you implement your long-runnning task

Next, about Mongoid and background jobs. You should never use complex objects as job arguments. I don't know what kind of task you are implementing, but there is general answer - use simple objects.

For example, instead of using your User as argument, use user_id and then find it inside your job. If you will do it like that, you can use any DB without problems.

---

Related Topics