How to Catch Top Level Failures on an Eventmachine Server

How to catch top level failures on an EventMachine server?

If you want a catch-all error handler, try EM.error_handler. Example from the docs:

EM.error_handler{ |e|
  puts "Error raised during event loop: #{e.message}"
}

You may also want more fine-grained error handling, in which case you can use the errback mechanism (see Deferrable). So for example you could have in your reactor loop:

EventMachine::run do
  server = EventMachine::start_server('0.0.0.0', PORT, MyServer)
  server.errback { # handle error thrown by server here  }
end

For this to work, include Deferrable in your MyServer, then whenever you want to raise an error, call fail.

Debugging websocket 1006

Went back to basic Ruby and inserted this code at the server's "on close" event:

if @debug
 puts "MyserverBackend>> Close entered. Last error:#{$!.class}:#{$!.to_s};Module:#{$0};"
 $@.each { |backtrace| puts backtrace }
 exit
end

This identifies where the failure is occurring and enhances debugging. For example, when I forced the error by issuing "xputs" instead of "puts" in the "on message" event, I now receive this clear information:

MyserverBackend>> Close entered.  Last error:NoMethodError:undefined method `xputs' for #<Myserver::MyserverBackend:0x4c9b2e8>;Module:.../ruby/bin/rackup;Line:2;
.../projects/myserver/middlewares/myserver_backend.rb:45:in `block (2 levels) in call'
.../projects/myserver/middlewares/myserver_backend.rb:43:in `each'
.../projects/myserver/middlewares/myserver_backend.rb:43:in `block in call'
...

Before this, my client would just terminate with a 1006 and the server would give me no notice at all of what had happened. Hope this helps someone else debug those nasty, ubiquitous 1006 errors.

ADDED OPTION:
I found another option. I execute this method as the very first statement of each WS method such as open/message/close/error. It is related to EventMachine, running under the covers within faye-websocket. I found it here.

def catch_error
  EventMachine.error_handler do |e|
    puts "Catch_error: #{caller[0]}: Last error:#{$!.class}:#{$!.to_s}; Module:#{$0}; Line:#{$.}; Error raised during event loop: Error:#{e.message}" #.red
    raise
  end
end

Node.js Best Practice Exception Handling

Update: Joyent now has their own guide. The following information is more of a summary:

Safely "throwing" errors

Ideally we'd like to avoid uncaught errors as much as possible, as such, instead of literally throwing the error, we can instead safely "throw" the error using one of the following methods depending on our code architecture:

• For synchronous code, if an error happens, return the error:

  // Define divider as a syncrhonous function
  var divideSync = function(x,y) {
      // if error condition?
      if ( y === 0 ) {
          // "throw" the error safely by returning it
          return new Error("Can't divide by zero")
      }
      else {
          // no error occured, continue on
          return x/y
      }
  }
  
  // Divide 4/2
  var result = divideSync(4,2)
  // did an error occur?
  if ( result instanceof Error ) {
      // handle the error safely
      console.log('4/2=err', result)
  }
  else {
      // no error occured, continue on
      console.log('4/2='+result)
  }
  
  // Divide 4/0
  result = divideSync(4,0)
  // did an error occur?
  if ( result instanceof Error ) {
      // handle the error safely
      console.log('4/0=err', result)
  }
  else {
      // no error occured, continue on
      console.log('4/0='+result)
  }
  

• For callback-based (ie. asynchronous) code, the first argument of the callback is err, if an error happens err is the error, if an error doesn't happen then err is null. Any other arguments follow the err argument:

  var divide = function(x,y,next) {
      // if error condition?
      if ( y === 0 ) {
          // "throw" the error safely by calling the completion callback
          // with the first argument being the error
          next(new Error("Can't divide by zero"))
      }
      else {
          // no error occured, continue on
          next(null, x/y)
      }
  }
  
  divide(4,2,function(err,result){
      // did an error occur?
      if ( err ) {
          // handle the error safely
          console.log('4/2=err', err)
      }
      else {
          // no error occured, continue on
          console.log('4/2='+result)
      }
  })
  
  divide(4,0,function(err,result){
      // did an error occur?
      if ( err ) {
          // handle the error safely
          console.log('4/0=err', err)
      }
      else {
          // no error occured, continue on
          console.log('4/0='+result)
      }
  })
  

• For eventful code, where the error may happen anywhere, instead of throwing the error, fire the error event instead:

  // Definite our Divider Event Emitter
  var events = require('events')
  var Divider = function(){
      events.EventEmitter.call(this)
  }
  require('util').inherits(Divider, events.EventEmitter)
  
  // Add the divide function
  Divider.prototype.divide = function(x,y){
      // if error condition?
      if ( y === 0 ) {
          // "throw" the error safely by emitting it
          var err = new Error("Can't divide by zero")
          this.emit('error', err)
      }
      else {
          // no error occured, continue on
          this.emit('divided', x, y, x/y)
      }
  
      // Chain
      return this;
  }
  
  // Create our divider and listen for errors
  var divider = new Divider()
  divider.on('error', function(err){
      // handle the error safely
      console.log(err)
  })
  divider.on('divided', function(x,y,result){
      console.log(x+'/'+y+'='+result)
  })
  
  // Divide
  divider.divide(4,2).divide(4,0)
  

Safely "catching" errors

Sometimes though, there may still be code that throws an error somewhere which can lead to an uncaught exception and a potential crash of our application if we don't catch it safely. Depending on our code architecture we can use one of the following methods to catch it:

• When we know where the error is occurring, we can wrap that section in a node.js domain

  var d = require('domain').create()
  d.on('error', function(err){
      // handle the error safely
      console.log(err)
  })
  
  // catch the uncaught errors in this asynchronous or synchronous code block
  d.run(function(){
      // the asynchronous or synchronous code that we want to catch thrown errors on
      var err = new Error('example')
      throw err
  })
  

• If we know where the error is occurring is synchronous code, and for whatever reason can't use domains (perhaps old version of node), we can use the try catch statement:

  // catch the uncaught errors in this synchronous code block
  // try catch statements only work on synchronous code
  try {
      // the synchronous code that we want to catch thrown errors on
      var err = new Error('example')
      throw err
  } catch (err) {
      // handle the error safely
      console.log(err)
  }
  

  However, be careful not to use try...catch in asynchronous code, as an asynchronously thrown error will not be caught:

  try {
      setTimeout(function(){
          var err = new Error('example')
          throw err
      }, 1000)
  }
  catch (err) {
      // Example error won't be caught here... crashing our app
      // hence the need for domains
  }
  

  If you do want to work with try..catch in conjunction with asynchronous code, when running Node 7.4 or higher you can use async/await natively to write your asynchronous functions.

  Another thing to be careful about with try...catch is the risk of wrapping your completion callback inside the try statement like so:

  var divide = function(x,y,next) {
      // if error condition?
      if ( y === 0 ) {
          // "throw" the error safely by calling the completion callback
          // with the first argument being the error
          next(new Error("Can't divide by zero"))
      }
      else {
          // no error occured, continue on
          next(null, x/y)
      }
  }
  
  var continueElsewhere = function(err, result){
          throw new Error('elsewhere has failed')
  }
  
  try {
          divide(4, 2, continueElsewhere)
          // ^ the execution of divide, and the execution of 
          //   continueElsewhere will be inside the try statement
  }
  catch (err) {
          console.log(err.stack)
          // ^ will output the "unexpected" result of: elsewhere has failed
  }
  

  This gotcha is very easy to do as your code becomes more complex. As such, it is best to either use domains or to return errors to avoid (1) uncaught exceptions in asynchronous code (2) the try catch catching execution that you don't want it to. In languages that allow for proper threading instead of JavaScript's asynchronous event-machine style, this is less of an issue.

• Finally, in the case where an uncaught error happens in a place that wasn't wrapped in a domain or a try catch statement, we can make our application not crash by using the uncaughtException listener (however doing so can put the application in an unknown state):

  // catch the uncaught errors that weren't wrapped in a domain or try catch statement
  // do not use this in modules, but only in applications, as otherwise we could have multiple of these bound
  process.on('uncaughtException', function(err) {
      // handle the error safely
      console.log(err)
  })
  
  // the asynchronous or synchronous code that emits the otherwise uncaught error
  var err = new Error('example')
  throw err
  

---

Related Topics