Preventing Delayed_Job Background Jobs from Consuming Too Much CPU on a Single Server

Preventing delayed_job background jobs from consuming too much CPU on a single server

If I'm not mistaken, delayed_job uses worker processes that will handle all the background jobs. It should be easily possible to alter the OS scheduling priority of the process when you start it.

So instead of, for example:

ruby script/delayed_job -e production -n 2 start

try:

nice -n 15 ruby script/delayed_job -e production -n 2 start

How to manage a pool of background servers in Rails

My personal preference is Sidekiq. I'd be a little concerned about "several hour" jobs and what happens if they fail in the middle. By default Sidekiq will try and re-run them. You can change that, but you definitely want to think through the the scenario. This of course will be true for whatever background job processing system you use though. IMHO I'd try to find a way to break those big jobs up into smaller jobs. Even if it's just "job part 1 runs then enqueues job part 2, etc".

As for scalability Sidekiq's only real limit is Redis. See here for some options on that: https://github.com/mperham/sidekiq/wiki/Sharding

As for load balancing, Sidekiq does it by default. I run two sidekiq servers now that pull from a single Redis instance. 25 workers on each with about 12 queues. Works amazingly well.

Running large amount of long running background jobs in Rails

It sounds like you are limited by memory on the number of workers that you can run on your DigitalOcean host.

If you are worried about scaling, I would focus on making the workers as efficient as possible. Have you done any benchmarking to understanding where the 900MB of memory is being allocated? I'm not sure what the nature of these jobs are, but you mentioned large files. Are you reading the contents of these files into memory, or are you streaming them? Are you using a database with SQL you can tune? Are you making many small API calls when you could be using a batch endpoint? Are you assigning intermediary variables that must then be garbage collected? Can you compress the files before you send them?

Look at the job structure itself. I've found that background jobs work best with many smaller jobs rather than one larger job. This allows execution to happen in parallel, and be more load balanced across all workers. You could even have a job that generates other jobs. If you need a job to orchestrate callbacks when a group of jobs finishes there is a DelayedJobGroup plugin at https://github.com/salsify/delayed_job_groups_plugin that allows you to invoke a final job only after the sibling jobs complete. I would aim for an execution time of a single job to be under 30 seconds. This is arbitrary but it illustrates what I mean by smaller jobs.

Some hosting providers like Amazon provide spot instances where you can pay a lower price on servers that do not have guaranteed availability. These pair well with the many fewer jobs approach I mentioned earlier.

Finally, Ruby might not be the right tool for the job. There are faster languages, and if you are limited by memory, or CPU, you might consider writing these jobs and their workers in another language like Javascript, Go or Rust. These can pair well with a Ruby stack, but offload computationally expensive subroutines to faster languages.

Finally, like many scaling issues, if you have more money than time, you can always throw more hardware at it. At least for a while.

delayed_job processes being killed silently

Well I am a ridiculous person. I did NOT know about the "run" option, where I could run delayed_job in the foreground and see exactly what was happening. My database setup is unconventional for rails, and my delayed_job table is not in the same schema as is listed in my config/database.yml table.

The reason this worked before is that I had monkey-patched a vendored delayed_job gem to hardcode the schema where my delayed_jobs table lived. When I upgraded Rails, I upgraded and re-vendored several gems, thus overwriting my previous hard-coding.

Which is why I shouldn't be doing that, I know. Lesson learned.

Running tasks in the background with lower CPU priority

If your CMS is running on linux system, then you can do this using the "nice" command. "nice" will start a process at a lower priority. Easiest way to use it is to just put nice in front of your command. So if you were running the command like

unzip uploaded-images.zip

instead run

nice unzip uploaded-images.zip

This will cause the unzip process to get lower CPU priority, letting other processes run first. See the man page for more options, like how to adjust the priority level.

Running delayed_job inside the main web process

Officially

No, there is no supported way to run delayed_jobs asynchronously within the web framework. From the documentation on running jobs, it looks like the only supported way to run a job is to run a rake task or the delayed job script. Also, it seems conceptually wrong to bend a Rack server, which was designed to handle incoming client requests, to support pulling tasks off of some queue somewhere.

The Kludge

That said, I understand that saving money sometimes trumps being conceptually perfect. Take a look at these rake tasks. My kludge is to create a special endpoint in your Rails server that you hit periodically from some remote location. Inside this endpoint, instantiate a Delayed::Worker and call .start on it with the exit_on_complete option. This way, you won't need a new dyno or command.

Be warned, it's kind of a kludgy solution and it will tie up one of your rails processes until all delayed jobs are complete. That means unless you have other rails processes, all incoming requests will block until this queue request is finished. Unicorn provides facilities to spawn worker processes. Whether or not this solution will work will also depend on your jobs and how long they take to run and your application's delay tolerances.

Edit

With the spawn gem, you can wrap your instantiation of the Delayed::Worker with a spawn block, which will cause your jobs to be run in a separate process. This means your rails process will be available to serve web requests immediately instead of blocking while delayed jobs are run. However, the spawn gem has some dependencies on ActiveRecord and I do not know what DB/ORM you are using.

Here is some example code, because it's becoming a bit hazy:

class JobsController < ApplicationController
  def run
    spawn do
      @options = {} # youll have to get these from that rake file
      Delayed::Worker.new(@options.merge(exit_on_complete: true)).start
    end
  end
end

Run an application in background

AppHarbor has background workers and Heroku has worker dynos.

---

Related Topics