Spinning Background Tasks in Rails

Spinning Background Tasks in Rails

Ryan Bates created three great screencasts that might really help you:

1. Rake in Background
2. Starling and Workling
3. Custom Daemon

He talks about the various pros and cons for using each one. This should help you get started.

Running large amount of long running background jobs in Rails

It sounds like you are limited by memory on the number of workers that you can run on your DigitalOcean host.

If you are worried about scaling, I would focus on making the workers as efficient as possible. Have you done any benchmarking to understanding where the 900MB of memory is being allocated? I'm not sure what the nature of these jobs are, but you mentioned large files. Are you reading the contents of these files into memory, or are you streaming them? Are you using a database with SQL you can tune? Are you making many small API calls when you could be using a batch endpoint? Are you assigning intermediary variables that must then be garbage collected? Can you compress the files before you send them?

Look at the job structure itself. I've found that background jobs work best with many smaller jobs rather than one larger job. This allows execution to happen in parallel, and be more load balanced across all workers. You could even have a job that generates other jobs. If you need a job to orchestrate callbacks when a group of jobs finishes there is a DelayedJobGroup plugin at https://github.com/salsify/delayed_job_groups_plugin that allows you to invoke a final job only after the sibling jobs complete. I would aim for an execution time of a single job to be under 30 seconds. This is arbitrary but it illustrates what I mean by smaller jobs.

Some hosting providers like Amazon provide spot instances where you can pay a lower price on servers that do not have guaranteed availability. These pair well with the many fewer jobs approach I mentioned earlier.

Finally, Ruby might not be the right tool for the job. There are faster languages, and if you are limited by memory, or CPU, you might consider writing these jobs and their workers in another language like Javascript, Go or Rust. These can pair well with a Ruby stack, but offload computationally expensive subroutines to faster languages.

Finally, like many scaling issues, if you have more money than time, you can always throw more hardware at it. At least for a while.

Practical use of delayed background job when dealing with many users

Resque is a background processing engine that can support multiple queues.

Ways you could use this:

1. Group your tasks into queues that make sense on their priority. If you need fast response times, use it in a 'foreground' queue. Slow? (like sending/receiving emails) can be in the 'background' queue
2. Have one queue per user (you will need to have many many workers for this)

This SO question also gives a way to use delayed_jobs with multiple queues/tables

Rails background job processing

I start up my workling and starling processes in one call using the "god" gem. It also allows you to setup CPU and memory thresholds. You might find it useful.

http://railscasts.com/episodes/130-monitoring-with-god

How to manage a pool of background servers in Rails

My personal preference is Sidekiq. I'd be a little concerned about "several hour" jobs and what happens if they fail in the middle. By default Sidekiq will try and re-run them. You can change that, but you definitely want to think through the the scenario. This of course will be true for whatever background job processing system you use though. IMHO I'd try to find a way to break those big jobs up into smaller jobs. Even if it's just "job part 1 runs then enqueues job part 2, etc".

As for scalability Sidekiq's only real limit is Redis. See here for some options on that: https://github.com/mperham/sidekiq/wiki/Sharding

As for load balancing, Sidekiq does it by default. I run two sidekiq servers now that pull from a single Redis instance. 25 workers on each with about 12 queues. Works amazingly well.

ruby on rails background application to run jobs automaically at a time dynamically defined by users?

I would advise to use Sidekiq. With it you can use scheduled jobs to tell sidekiq when to perform the jobs.

Example :
MyWorker.perform_at(3.hours.from_now, 'mike', 1)

EDIT : worker example

#app/workers/restart_device_worker.rb
class RestartDeviceWorker
  include Sidekiq::Worker

  def perform(params)
    # Do the job
    # ...
    # update in DB
  end
end

---

Related Topics