Measuring server load with collectd, Part I

Jump to Part II

14 Jun 2010 by Misha Dragojevic

If you ever asked yourself one of the following questions, then read on:

  1. Want to know how much server resources are you using?
  2. Do you have enough CPU power?
  3. Is your RAM fully utilized?
  4. What’s network traffic load?
  5. And by the way what is server load overnight vs. daily?
  6. When is the peak load?

To answer above questions, you would need to collect various system samples at regular intervals, save those in a DB of some sort (so that you can run reports afterwards), and finally present results in a nice graphical format.

Thanks to collectd, system statistics collection daemon, all of these are possible: you can set automatic sample collection via cron, configure what do you want to measure and track, record results in RRD data base and finally present reports in a flexible graphical format! Collectd has a very small footprint, it is fast and reliable.

This post if part 1 of 2 and explains how to setup “collectd”, version 4, for your server farm. Next post, part 2, explains how to expose results via web server in graphical format.

Now is a good time to ask yourself: do I really need this? It is fair to note that if you are already in Amazon EC2 cloud, you can buy CloudWatch service (estimated cost for 10-server farm is $108.00 per month). Other solutions are possible. For example, RightScale will monitor your server farm, but that service is not free (you need to have a paid account). If you follow instructions below it will cost you nothing to operate and you can potentially save $$$ …

Use case scenario: you are running a server farm (in the cloud or somewhere else) and need to collect performance data at regular intervals. Architecture: we will configure a single server that will collect information from all other servers on the farm and expose usage graphs via web browser. In addition to the server, we will configure all other systems to collect information at regular intervals and then send upstream to the server.

Let’s start with “collectd” installation. I’m assuming you are running Ubuntu Linux. If not, please check official site as to how to build from source.

Step 1: Install collectd on all servers

*Step 2*: Install perl modules on server machine *only*

Step 3: Configuration

Configuration files on Ubuntu are in /etc/collectd/, the most important is /etc/collectd/collectd.conf.

Server configuration file:

 Notes: # Server will listen on *all* interfaces, port 12345 (see line 11). This could be a security problem: you can listen only on "internal" interface (if in cloud environment) and / or you can adjust port number and then fine tune security rules. If you are using Amazon EC2 cloud above configuration would be OK by default. In addition, you can enhance security by requesting Username/ Password or by HMAC signature. # Server listens for UDP traffic (not TCP). Make sure that client can connect to server on designated port using UDP protocol. # Collected data is saved in DataDir, see line 31; you can change this. # You can have multiple servers and / or multiple ports. *Client* configuration file is very similar to server configuration file, with few exceptions:

Notes:

  1. Client sends data to the server, see line 11; make sure to provide correct server name / port (must match server configuration, above).
  2. Client needs to be able to connect to server on designated port via UDP.

After you configure server and client, you need to restart “collectd” daemon so that new settings take effect.

After restart, clients will start collecting data and sending those to the server.

Next post, Part 2 explains how to configure web server for graphical data presentation.

blog comments powered by Disqus