How to Monitor Uptime of 20 Websites (Ping or Http) in Node.Js/Ror

How to Monitor Uptime of 20 Websites (Ping or HTTP) in Node.js/RoR

Github

I really like node.js and I would like to tackle this problem and hopefully soon share some code on github to achieve this. Keep in mind that I only have a veryy basic setup right now hosted at https://github.com/alfredwesterveld/freakinping

What's the best way to ping a list of
20 websites every 5 minutes (for
example) in order to know if the site
responds with HTTP 202 or not?

PING(ICMP)

First I would like to know if you want to really do a ping(ICMP) or if you just want to know if the website returns with code 200(OK) and measure the time it takes. I believe from the context that you don't really want to do a ping, but just an http request and measure the time. I ask this because(I believe) pinging from node.js/ruby/python can't be done from normal user because we need raw sockets(root user) to do the pinging(ICMP) from programming language. I for example found this ping script in python(I also believe I saw a simple ruby script somewhere although I am not a really big ruby programmer) but requires root access. I don't believe there is even yet a ping module out there for node.js.

Message Queue

Also, is there better but no-brainer
solution for this? I'm afraid the list
can grow to 20000 websites and then
there's not enough time to ping them
all in the 5 minutes I need to be
pinging.

Basically, I'm describing how PingDom,
UptimeRobot, and the likes work.

What you need to achieve this kind of scale is to use a message queue like for example redis, beanstalkd or gearmand. At the scale of PingDom one worker process is not going to cut it, but in your case it(I assume) one worker will do. I think(assume) redis will be the fastest message queue because of the C(node.js) extension but then again I should benchmark it against beanstalkd, which is another popular message queue(but does not yet have a C extension).

I'm afraid the list can grow to 20000
websites

If you get at that scale you might have to have host multiple boxes(a lot of worker threads/processes) to handle the load but you aren't at that scale yet and node.js is insane fast. It might even be able to handle that load with even one single box, although I don't know for sure(you need to do/run some benchmarks).

Datastore/Redis

I think this could be achieved pretty easily in node.js(I really like node.js). The way I would do this is use redis as my datastore because it is INSANE FAST!

PING: 20000 ops 46189.38 ops/sec 1/4/1.082
SET: 20000 ops 41237.11 ops/sec 0/6/1.210
GET: 20000 ops 39682.54 ops/sec 1/7/1.257
INCR: 20000 ops 40080.16 ops/sec 0/8/1.242
LPUSH: 20000 ops 41152.26 ops/sec 0/3/1.212
LRANGE (10 elements): 20000 ops 36563.07 ops/sec 1/8/1.363
LRANGE (100 elements): 20000 ops 21834.06 ops/sec 0/9/2.287

using node_redis(with hredis(node.js) c library). I would Add the URLs to redis using sadd.

Run tasks every 5 minutes

This could be achieved without barely any effort. I would use the setInterval(callback, delay, [arg], [...]) to repeatedly test response time of servers. Get all URLs on callback from redis using smembers. I would put all the URLs(messages) on the message queue using rpush.

Checking Response (Time)

However, what happen when one doesn't
answers? What happens to the ones
after that?

I might not completely understand this sentence but here it goes. If one fails it just fails. You could try to check response(time) again in 5 seconds or something to see if it is online. A precise algorithm for this should be devised. The ones after that should not have anything to do with previous URLs unless the are to the same server. Also something you clearly think about I guess because then you should not ping all those URLs to the same server at the same time but queue them up or something.

Processing URL

From the worker process(for now just one would be suffice) fetch message(URL) from redis using brpop command. check response time for URL(message) and fetch next URL(message) from the list. I would probably do a couple of request simultaneous to speed up the process.

How To Set Up A Ping Route (HEAD) For New Relic In Restify/Express

EDIT:

The parameters are mislabeled so the res.send() is actually trying to call next().send() which would be undefined. Removing the error parameter and shifting everything over fixed the code as discovered by the OP.

---

As per the restify documentation, you need to call return next() in your callback function:

http://mcavage.me/node-restify/#Routing

server.head('/ping', function (req, res) {
    res.send('hello');
});

If you would like to respond immediately and not continue down the chain, you can pass false as a parameter in your call to next()

Alternative to New Relic for pinging Heroku application

Pingdom should work for that. The Heroku add-on catalog has lots of other monitoring add-ons that should also work great.

Node.js programming workflow - Tests, Code, Tests

My testing methodology isn't up the snuff as in for example Java/Junit and I should really work more on this(improve). I should really practice TDD more.

I played a little bit with expresso and liked to the fact that you could generate code coverage reports. What I thought was missing was something like @before @beforeclass @after which you can find in java.

I also played a bit with nodeunit which does have setup/teardown. I still like to play a little bit more with this framework.

I don't like the vowjs syntax, but it is very popular BDD framework, so maybe I should use it (more) to get sold like a lot of other users. But for now I am going to dismiss vowjs.

I also played with zombie.js a litle bit which is also pretty cool. I also lately saw another cool testing framework which I can't remember the name, but there are luckily enough options to do testing in node.js.

The only thing I don't like is that the integration with IDE is not up to snuff in my opinion. The IDE I had for Java cannot be compared with what I have found for node.js, but I think with a little bit effort I can make a more useful programming environment. I will try and keep you guys informed about this progress.

P.S: But what I do like a lot is the npm package manager. When you compare it to for example maven you just say wow. I still has some minor bugs because it is still a young project. But still npm is very good in my opinion!

---

Related Topics