Exception_Notification for Delayed_Job

How to make ExceptionNotifier work with delayed_job in Rails 3?

I do this with Rails 3.2.6, delayed_job 3.0.3 and exception_notification 2.6.1 gem

# In config/environments/production.rb or config/initializers/delayed_job.rb

# Optional but recommended for less future surprises.
# Fail at startup if method does not exist instead of later in a background job 
[[ExceptionNotifier::Notifier, :background_exception_notification]].each do |object, method_name|
  raise NoMethodError, "undefined method `#{method_name}' for #{object.inspect}" unless object.respond_to?(method_name, true)
end

# Chain delayed job's handle_failed_job method to do exception notification
Delayed::Worker.class_eval do 
  def handle_failed_job_with_notification(job, error)
    handle_failed_job_without_notification(job, error)
    # only actually send mail in production
    if Rails.env.production?
      # rescue if ExceptionNotifier fails for some reason
      begin
        ExceptionNotifier::Notifier.background_exception_notification(error)
      rescue Exception => e
        Rails.logger.error "ExceptionNotifier failed: #{e.class.name}: #{e.message}"
        e.backtrace.each do |f|
          Rails.logger.error "  #{f}"
        end
        Rails.logger.flush
      end
    end
  end 
  alias_method_chain :handle_failed_job, :notification 
end

It's probably a good idea to load this code in all environments to catch errors after bundle update etc before they reach production. I do this by having a config/initializers/delayed_job.rb file but you could duplicate the code for each config/environments/* environment.

Another tip is to tune the delayed job config a bit as default you may get a lot of duplicate exception mails when job fails.

# In config/initializers/delayed_job_config.rb
Delayed::Worker.max_attempts = 3

Update I had some problems with the delayed_job daemon silently exiting and it turned out to be when ExceptionNotifier fails to send mail and no one rescued the exception. Now the code rescues and log them.

exception_notification for delayed_job

I've done something like this in the past for delayed job rake tasks:

require 'action_mailer'
class ExceptionMailer < ActionMailer::Base
  def setup_mail
    @from = ExceptionNotifier.sender_address
    @sent_on = Time.now
    @content_type = "text/plain"
  end

  def exception_message(subject, message)
    setup_mail
    @subject = subject
    @recipients = ExceptionNotifier.exception_recipients
    @body = message
  end
end

namespace :jobs do
desc "sync the local database with the remote CMS"
task(:sync_cms => :environment) do
  Resort.sync_all!
  result = Delayed::Job.work_off
  unless result[1].zero?
    ExceptionMailer.deliver_exception_message("[SYNC CMS] Error syncing CMS id: #{Delayed::Job.last.id}", Delayed::Job.last.last_error)
  end
end

end

Configuring Exception Notification with Delayed Jobs in Rails 4

Since exception_notification 4.0 there's a new way to handle manual notify. Try to change

ExceptionNotifier::Notifier.background_exception_notification(error)

for

ExceptionNotifier.notify_exception(error)

Requeue or evaluate delayed job from inside?

As you've said, if the Delayed Job runner gets to the end of the perform queue then it will be considered as a successful run and removed from the queue. So you just have to stop it from getting to the end. There isn't a requeue -- even if there was it'd be a new record with new attributes. So you may rethink whatever it is that is causing the job to notify you about exceptions. You could, for example, add a condition upon which to notify you...

Potential Solutions

You can get the default JOBS_MAX (as you pseudo-coded it) with Delayed::Worker.max_attempts or you can set your own per-job by defining a method, e.g.: max_attempts.

# Fail permanently after the 10th failure for this job
def max_attempts
  10
end

That is, this method will be usable given the following:

You can also make use of callback hooks. Delayed Job will callback to your payload object via the error method if it is defined. So you may be able to use the error method to notify you of actual exceptions beyond a given attempt number. To do that...

Within the callback, the Delayed::Job object itself is returned as the first argument:

def error(job, exception)
  job.attempts # gives you the current attempt number
  # If job.attempts is greater than max_attempts then send exception notification
  # or whatever you want here...
end

So you can use the callbacks to start adding logic on when to notify yourself and when not to. I might even suggest making a base set of functionality that you can include into all of your payload objects to do these things... but that's up to you and your design.

Rails exception notifier in rake tasks

Airbrake gem patches Rake to allow rescuing, so it already does what I'm asking...

ActionMailer::Base::NullMail when trying exception_notification in development

Likely you are using an old version of the ExceptionNotifier and a newer version of ActiveMailer::Base. Not calling the mail command within the email functionality will result in the ActionMailer::Base::NullMail instance returned rather than a Mail instance.

From documentation:

class Notifier < ActionMailer::Base
  default :from => 'no-reply@example.com',
         :return_path => 'system@example.com'

  def welcome(recipient)
    @account = recipient
    mail(:to => recipient.email_address_with_name,
         :bcc => ["bcc@example.com", "Order Watcher <watcher@example.com>"])
  end
end

401 unauthorized errors on Heroku with Delayed Job

Okay, I figured this out by - of course - browsing the source of the various libraries that were invoked in the backtrace.

The most relevant line:

vendor/bundle/ruby/1.9.1/gems/heroku-api-0.3.5/lib/heroku/api/processes.rb:9:in `get_ps'

This uses the heroku-api gem to go get some worker processes. Browsing the source of the heroku-api gem, I see that when you make a request via the API, it initializes a connection like so:

@api_key = options.delete(:api_key) || ENV['HEROKU_API_KEY']
if !@api_key && options.has_key?(:username) && options.has_key?(:password)
  @connection = Excon.new("#{options[:scheme]}://#{options[:host]}", options.merge(:headers => HEADERS))
  @api_key = self.post_login(options[:username], options[:password]).body["api_key"]
end

My environment did not have ENV['HEROKU_API_KEY'] set. I did have the HEROKU_PASSWORD set, which contains the same data, but that's not what this gem is looking for. Thus, I was getting 401 errors on this.

I will submit an update to the Heroku documentation and ask them to include this as one of the steps, since it is not in there now.

To fix this, I simply did:

heroku config:add HEROKU_API_KEY=KEY

Where KEY is the same as the value for HEROKU_PASSWORD. (You can view all config vars with heroku config).

---

Related Topics