Hacker Notes

Real Time Monitoring : Know Everything

Instagram released an awesome overview of how they handled the incredible scaling issues associated with 40+ million people using the service. Their theory of monitoring everything was exactly what I’ve been trying to do at @EggDropApp.

A great tool they highlighted was Statsd, an open source monitoring server from Etsy. It can collect basic stats from any service in your system with minimum overhead and generate easy-to-use graphs on the fly. The basic stats can be simple counters or timing that can be averaged over selectable periods. The interface is UDP-based and a breeze to use.

This is something that any startup can use. Given that, I tossed together an EC2 AMI that launches a standalone Statsd server ready for action. Launch it in your environment and all you need to worry about is integrating stats. Grab the link below and give it a whirl.

https://aws.amazon.com/amis/statsd-on-amazon-linux-2012-03-ami

Check out Instagram’s incredible post and the original blog post from Etsy.

http://instagram-engineering.tumblr.com/post/20541814340/keeping-instagram-up-with-over-a-million-new-users-in

http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

* Minor note on logging into the Graphite web:

Log into the box

$ cd /opt/graphite/webapp/graphite
$ sudo python manage.py createsuperuser stats

Follow the steps to create your new user

Open up the site in your browser and log in.