After publishing Ben's blog post about "Memory Monitoring with LXC" we realized there is a lot of interest in articles about Monitoring. I got in touch with Jehiah Czebotar, Head of Engineering at bitly, and asked him if we could republish his blog post about things bitly forgot to monitor.
Jehiah originally published his blog post on the bitly engeneering blog. You can find Jehiah on twitter and on his personal page. Definitely check out his "Personal Annual Reports".
There is always a set of standard metrics that are universally monitored (Disk Usage, Memory Usage, Load, Pings, etc). Beyond that, there are a lot of lessons that we’ve learned from operating our production systems that have helped shape the breadth of monitoring that we perform at bitly.
One of my favorite all-time tweets is from @DevOps_Borat
Law of Murphy for devops: if thing can able go wrong, is mean is already wrong but you not have Nagios alert of it yet.
— DevOps Borat (@DEVOPS_BORAT) December 19, 2012
What follows is a small list of things we monitor at bitly that have grown out of those (sometimes painful!) experiences, and where possible little snippets of the stories behind those instances.
1 - Fork Rate
We once had a problem where IPv6 was intentionally disabled on a box via options ipv6 disable=1
and alias ipv6 off
in /etc/modprobe.conf
. This caused a large issue for us: each time a new curl object was created, modprobe
would spawn, checking net-pf-10
to evaluate IPv6 status. This fork bombed the box, and we eventually tracked it down by noticing that the process counter in /proc/stat
was increasing by several hundred a second. Normally you would only expect a fork rate of 1-10/sec on a production box with steady traffic.
https://gist.github.com/jehiah/8511258.js
2 - flow control packets
TL;DR; If your network configuration honors flow control packets and isn’t configured to disable them, they can temporarily cause dropped traffic. (If this doesn’t sound like an outage, you need your head checked.)
#!ruby $ /usr/sbin/ethtool -S eth0 | grep flow_control rx_flow_control_xon: 0 rx_flow_control_xoff: 0 tx_flow_control_xon: 0 tx_flow_control_xoff: 0
Note: Read this to understand how these flow control frames can cascade to switch-wide loss of connectivity if you use certain Broadcom NIC’s. You should also trend these metrics on your switch gear. While at it, watch your dropped frames.
3 - Swap In/Out Rate
It’s common to check for swap usage above a threshold, but even if you have a small quantity of memory swapped, it’s actually the rate it’s swapped in/out that can impact performance, not the quantity. This is a much more direct check for that state.
https://gist.github.com/jehiah/8511306.js
4 - Server Boot Notification
Unexpected reboots are part of life. Do you know when they happen on your hosts? Most people do not. We use a simple init script that triggers an ops email on system boot. This is valuable to communicate provisioning of new servers, and helps capture state change even if services handle the failure gracefully without alerting.
https://gist.github.com/jehiah/8511374.js
5 - NTP Clock Offset
If not monitored, yes, one of your servers is probably off. If you’ve never thought about clock skew you might not even be running ntpd
on your servers. Generally there are 3 things to check for. 1) That ntpd
is running, 2) Clock skew inside your datacenter, 3) Clock skew from your controller time servers to an external source.
We use check_ntp_time for this check.
6 - DNS Resolutions
Internal DNS - It’s a hidden part of your infrastructure that you rely on more than you realize. The things to check for are 1) Local resolutions from each server, 2) If you have local DNS servers in your datacenter, you want to check resolution, and quantity of queries, 3) Check availability of each upstream DNS resolver you use.
External DNS - It’s good to verify your external domains resolve correctly against each of your published external nameservers. At bitly we also rely on several CC TLD’s and we monitor those authoritative servers directly as well (yes, it’s happened that all authoritative nameservers for a TLD have been offline).
7 - SSL Expiration
It’s the thing everyone forgets about because it happens so infrequently. The fix is easy, just check it and get alerted with enough timeframe to renew your SSL certificates.
#!ruby define command{ command_name check_ssl_expire command_line $USER1$/check_http --ssl -C 14 -H $ARG1$ } define service{ host_name virtual service_description bitly_com_ssl_expiration use generic-service check_command check_ssl_expire!bitly.com contact_groups email_only normal_check_interval 720 retry_check_interval 10 notification_interval 720 }
8 - DELL OpenManage Server Administrator (OMSA)
We run bitly split across two data centers, one is a managed environment with DELL hardware, and the second is Amazon EC2. For our DELL hardware it’s important for us to monitor the outputs from OMSA. This alerts us to RAID status, failed disks (predictive or hard failures), RAM Issues, Power Supply states and more.
9 - Connection Limits
You probably run things like memcached and mysql with connection limits, but do you monitor how close you are to those limits as you scale out application tiers?
Related to this is addressing the issue of processes running into file descriptor limits. We make a regular practice of running services with ulimit -n 65535
in our run scripts to minimize this. We also set Nginx worker_rlimit_nofile.
10 - Load Balancer Status.
We configure our Load Balancers with a health check which we can easily force to fail in order to have any given server removed from rotation.We’ve found it important to have visibility into the health check state, so we monitor and alert based on the same health check. (If you use EC2 Load Balancers you can monitor the ELB state from Amazon API’s).
Various Other things to watch
New entries written to Nginx Error Logs, service restarts (assuming you have something in place to auto-restart them on failure), numa stats, new process core dumps (great if you run any C code).
We want to thank Jehiah for making his original article available to our readers. Is there something missing on this list in your opportunity? Let us know in the comments!