Resque Multiple Workers in Development Mode

Resque multiple workers in development mode

You need to add a COUNT environment variable and then change resque:work to resque:workers. For example, to start 3 workers:

bundle exec env rake resque:workers QUEUE='*' COUNT='3'

Resque with multiple workers: What order?

See https://github.com/resque/resque/blob/master/lib/resque/multi_queue.rb

If it's working in blocking mode (which it does by default), then it uses Redis' BLPOP method, which accepts a list of queues, and will pop and return a value from the first queue to have data.

Redis' BLPOP enqueues clients on a first-come, first-served basis. When given multiple queue keys, it simply iterates through them and sets up blocking per key. See https://github.com/antirez/redis/blob/unstable/src/t_list.c#L781-815

Resque is going to build a list of queues to test by just getting something like SMEMBERS queues, which means that queues will be prioritized in the order that the SMEMBERS command returns them in. That's a set operation, which means that its order is undefined; you're mostly at Redis' mercy.

Concurrency with Resque on Heroku (running multiple workers per node)

Update: The solution below works, but is not recommended. For resque concurrency on heroku use the resque-pool gem.

---

It is possible if you use the COUNT=* option. Your procfile will look something like:

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
resque: env TERM_CHILD=1 COUNT=2 RESQUE_TERM_TIMEOUT=6 QUEUE=* bundle exec rake resque:workers

It is impotant to note that the rake task in the Procfile is resque:workers and not resque:work.

---

Update Explanation

There are major problems with the COUNT=* option and the rake resque:workers invocation in production on heroku. Because of the way resque starts up the multiple workers using threads, all of the SIGTERM, SIGKILL, etc. handling that allows workers to stop the current job, re-enqueue job, and shut down properly (including de-registering) will ever happen. This is because the the signals are handled by the main process not trapped by the threads. This can cause phantom workers to remain in the worker list long after they've been killed. This is probably why there is a comment in the resque code that warns that the resque:workers should only be used in development mode.

Starting resque workers while started development server

Are you sure that you manage to run this command sometime back successfully

QUEUE=* rake environment resque:work rails s

because the way I know rails s is a list of rails command not a rake task

you can run consecutive rake separating spaces like

QUEUE=* rake environment rake1 rake2 rake3

but you cant run a rake and rails command they way you mention above

what I see from your trace above that rake(resque rake) is running now instead of passing a second rake you specified a rails command to start the server rake is considering that as rake task (which is not true)

I believe you are looking for this

QUEUE=* rake environment resque:work && rails s

But I dont believe that what you have mention would ever work please let me know if something conflict over here

Hope It make sense

Rails Resque Concurrency

You can specify the worker count when running the command, like so:

$ COUNT=2 QUEUE='converter' rake resque:workers

---

Related Topics