Use Delayed::Job to Manage Multiple Job Queues

Use Delayed::Job to manage multiple job queues

Since then http://github.com/collectiveidea/delayed_job has reached v3.0 and includes named queues! Excellent for simple grouped queue needs.

Delayed_job - Multiple parallel queues?

With bundler in production:

RAILS_ENV=production bundle exec script/delayed_job -n 4 start

or without bundler

ruby script/delayed_job -n 4 start

Practical use of delayed background job when dealing with many users

Resque is a background processing engine that can support multiple queues.

Ways you could use this:

1. Group your tasks into queues that make sense on their priority. If you need fast response times, use it in a 'foreground' queue. Slow? (like sending/receiving emails) can be in the 'background' queue
2. Have one queue per user (you will need to have many many workers for this)

This SO question also gives a way to use delayed_jobs with multiple queues/tables

How can I 'consume' jobs from a Delayed Job queue in an app besides the one that 'produces' them

Actually, what you're describing above is correct:
You want to queue an id, and sometimes other required information to identify the record you want to pull from the database, and NOT an object, especially if it's an ActiveRecord::Base object.

Queuing an object is problematic and can lead to issues. Check my answer at:

https://stackoverflow.com/a/21751030/226255

You can test if your second application is pulling correctly by doing the following in console: (make sure to push a Delayed::Job to queue first)

Delayed::Job.first.invoke_job

An affordable way to use multiple Delayed::Job queues

Moving to my own server was the way to go. I'm paying under $300/month for a machine that can run 40 workers easily. The downside is having to learn about server administration, but it's not too bad.

Delayed Job One worker for one task (synchronous call)

I can solve this by doing that way

When it assign next same job to another worker, we can add run_at one second ahead, so multiple workers does not save object same time and this fix our issue

latest_job = Delayed::Job.where(queue: "order_item_kit").last

run_at = latest_job ? latest_job.run_at + 1.seconds : 1.seconds.from_now

kit_sku.delay(:run_at => run_at,:queue =>"order_item_kit").update_ordere_item

delayed_job: One job per tenant at a time?

Your best bet is probably to spin a custom solution that implements a distributed lock - essentially, the workers all run normally and pull from the usual queues, but before performing work check with another system (Redis, RDBMS, API, whatever) to verify that no other worker is yet performing a job for that tenant. If that tenant is not being worked, then set the lock for the tenant in question and work the job. If the tenant is locked, don't perform the work. It's your call on a lot of the implementation details like whether to move on to try another job, re-enqueue the job at the back of the queue, whether to consider it a failure and bind it to your retry limits, or do something else entirely. This is pretty open-ended, so I'll leave the details to you, but here are some tips:

• Inheritance will be your friend; define this behavior on a base job and inherit from it on the jobs you expect your workers to run. This also allows you to customize the behavior if you have "special" cases for certain jobs that come up without breaking everything else.
• Assuming you're not running through ActiveJob (since it wasn't mentioned), read up on delayed_job hooks: https://github.com/collectiveidea/delayed_job/#hooks - they may be an appropriate and/or useful tool
• Get familiar with some of the differences and tradeoffs in Pessimistic and Optimistic locking strategies - this answer is a good starting point: Optimistic vs. Pessimistic locking
• Read up on general practices surrounding the concept of distributed locks so you can choose the best tools and strategies for yourself (it doesn't have to be a crazy complicated solution, a simple table in the database that stores the tenant identifier is sufficient, but you'll want to consider the failure cases - how to you manage locks that are abandoned, for example)

Seriously consider not doing this; is it really strictly required for the system to operate properly? If so, it's probably indicative in an underlying flaw in your data model or how you've structured transformations around that data. Strive for ACIDity in your application when thinking about operations on the data and you can avoid a lot of these problems. There's a reason it's not a commonly available "out of the box" feature on background job runners. If there is an underlying flaw, it won't just bite you on this problem but on something else - guaranteed!

How do I separate workers into pools of jobs with delayed job + heroku?

It's in the README for Delayed Job 3:

DJ 3 introduces Resque-style named queues while still retaining DJ-style priority. The goal is to provide a system for grouping tasks to be worked by separate pools of workers, which may be scaled and controlled individually.

Jobs can be assigned to a queue by setting the queue option:

object.delay(:queue => 'tracking').method

Delayed::Job.enqueue job, :queue => 'tracking'

handle_asynchronously :tweet_later, :queue => 'tweets'

script/delayed_job can be used to manage a background process which will start working off jobs.

To do so, add gem "daemons" to your Gemfile and make sure you’ve run rails generate delayed_job.

You can then do the following:

$ RAILS_ENV=production script/delayed_job start
$ RAILS_ENV=production script/delayed_job stop

# Runs two workers in separate processes.
$ RAILS_ENV=production script/delayed_job -n 2 start
$ RAILS_ENV=production script/delayed_job stop

# Set the --queue or --queues option to work from a particular queue.
$ RAILS_ENV=production script/delayed_job --queue=tracking start
$ RAILS_ENV=production script/delayed_job --queues=mailers,tasks start

Work off queues by setting the QUEUE or QUEUES environment variable.

QUEUE=tracking rake jobs:work
QUEUES=mailers,tasks rake jobs:work

On Heroku, In your procfile, create two entries:

worker1: QUEUE=tracking rake jobs:work
worker2: QUEUES=mailers,tasks rake jobs:work

and scale them individually:

heroku ps:scale worker1=2 worker2=1 

etc

concurrency in delayed_jobs

No, one worker cannot run two jobs concurrently, you need more than one process running for that.

In the example you are describing, you are starting a worker that is running in the foreground (rake job:work), but what you could do instead, is to start them as background workers, by running bin/delayed_job instead (script/delayed_job for earlier versions). That command has multiple options that you can use to specify how you want delayed_job to function.

One of the options is -n or --number_of_workers=workers. That means you can start two workers by running the following command:

bundle exec bin/delayed_job --number_of_workers=2 start

It is also possible to dedicate certain workers to only run jobs from a specific queue, or only high priority jobs.

---

Related Topics