Optimize VPS server for Drupal Hosting | Using Varnish

Written by
Date: 2012-04-08 14:26:00 00:00


Introduction

In the past, and for some years I've run my blog with the help of Drupal, in that time and in the first months, I've got Slashdoted, and Dugg three times, all three times my server went down.

Since Then I've become obsessed with tweaking my server configuration to support the load of Slashdot, Digg and the like.

I'm not running my blog over Drupal anymore, but I still like Drupal a lot, and this weekend I've been playing with Drupal 7 and Varnish, to see how it performs, under heavy load.

I was trying to figure out a way to optimize the Drupal configuration without having to tweak too much into the Drupal or server configuration, and without the need to add too many "performance" modules.

The environment

Here is my configuration details:

  • Arch Linux 2011.10
  • RackSpace VPS
  • 256 RAM
  • Apache/PHP/MySQL/Varnish

Configuration

Drupal

I'm using the basic Drupal 7 installation, with core cache turned ON.

LAMP

LAMP is the standard available in Arch Linux by the time of this writing, and no special configuration to any of the components. Except that Apache is listening to port 8080 instead of port 80. So it can server pages internally to Varnish.

Varnish

From Wikipedia:

Varnish is an HTTP accelerator designed for content-heavy dynamic web sites. In contrast to other HTTP accelerators, such as Squid, which began life as a client-side cache, or Apache and nginx, which are primarily origin servers, Varnish was designed from the ground up as an HTTP accelerator. Varnish is focused exclusively on HTTP, unlike other proxy servers that often support FTP, SMTP and other network protocols

Varnish is going to support the load, but once again the configuration is pretty basic:

/etc/conf.d/varnish listing:

VARNISHD_OPTS="-a 0.0.0.0:80 \
           -b localhost:8080 \
           -T localhost:6082 \
           -s malloc,64M \
       	   -s file,/var/cache/$INSTANCE/varnish_storage.bin,1G \
           -u nobody -g nobody"

VARNISH_CFG="/etc/varnish/default.vcl"

/etc/varnish/default.vcl listing:

backend default {
.host = "127.0.0.1";
.port = "8080";
}


sub vcl_recv {
  // Remove has_js and Google Analytics __* cookies.
  set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|	has_js)=[^;]*", "");
  // Remove a ";" prefix, if present.
  set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
  // Remove empty cookies.
  if (req.http.Cookie ~ "^\s*$") {
    unset req.http.Cookie;
  }

  // Cache all requests by default, overriding the
  // standard Varnish behavior.
  // if (req.request == "GET" || req.request == "HEAD") {
  //   return (lookup);
  // }
}

sub vcl_hash {
  if (req.http.Cookie) {
    set req.hash += req.http.Cookie;
  }
}

Testing

I've used ab tool to test, as this is a test to prove the Drupal site will be able to manage a spike in traffic from Digg or John Grubber, then ab is OK. If you plan to have thousands of pages and ten thousands pages views per hour, distributed all across the content, this may not be for you, but if only one or a few pages are popular at a time, this is the right place to be.

This is the command:

ab -n 10000 -c 100 http://10.179.138.178/drupal/?q=node/2

Where:

-n: Number of requests -c: Number of concurrent sessions

The results:

Server Software:        Apache/2.2.22
Server Hostname:        10.179.138.x
Server Port:            80

Document Path:          /drupal/?q=node/2
Document Length:        8866 bytes

Concurrency Level:      100
Time taken for tests:   39.358 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      95050000 bytes
HTML transferred:       88660000 bytes
Requests per second:    254.08 [#/sec] (mean)
Time per request:       393.577 [ms] (mean)
Time per request:       3.936 [ms] (mean, across all concurrent requests)
Transfer rate:          2358.43 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1  115 205.8    101    4696
Processing:     3  277 139.6    297    2946	
Waiting:        1  141 129.8    147    2546
Total:          6  392 242.9    399    4948

Percentage of the requests served within a certain time (ms)
  50%    399
  66%    400
  75%    400
  80%    401
  90%    448
  95%    450
  98%    499
  99%   1399
 100%   4948 (longest request)

After this, I've transfer that same page to an Nginx server running on a mirror Arch Linux powered server.

I've done that using curl

curl 10.179.138.178/drupal/?q=node/2 > /srv/http/drupal.html

And then run ab against Nginx with the static page, the result was:

Server Software:        nginx/1.0.14
Server Hostname:        10.179.129.x
Server Port:            8080

Document Path:          /drupal.html
Document Length:        8866 bytes

Concurrency Level:      100
Time taken for tests:   38.353 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      91860000 bytes
HTML transferred:       88660000 bytes
Requests per second:    260.73 [#/sec] (mean)
Time per request:       383.533 [ms] (mean)
Time per request:       3.835 [ms] (mean, across all concurrent requests)
Transfer rate:          2338.97 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1  129 332.9    101    9051
Processing:     1  253  80.0    249     749
Waiting:        1  130  63.7    141     617
Total:          4  382 340.4    350    9300

Percentage of the requests served within a certain time (ms)
  50%    350
  66%    399
  75%    400
  80%    400
  90%    401
  95%    450
  98%    652
  99%    901
 100%   9300 (longest request)

As you can see even though Drupal is not using boost, and it is full dynamic content, Varnish is making it equivalent to a static site. The results are almost the same in both tests.

Just to let you see how it performs without Varnish, here is what happens when Varnish is taken aside and Apache/PHP/MySQL support the full load.

Well: with the same load, MySQL hanged up, and all Operating System halted. I had to reboot the server from the Console.

So lowering the load:

ab -n 1000 -c 20 http://10.179.138.178/drupal/?q=node/2

The result is:

Server Software:        Apache/2.2.22
Server Hostname:        10.179.138.178
Server Port:            80

Document Path:          /drupal/?q=node/2
Document Length:        8866 bytes

Concurrency Level:      20
Time taken for tests:   13.022 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      9445000 bytes
HTML transferred:       8866000 bytes
Requests per second:    76.79 [#/sec] (mean)
Time per request:       260.443 [ms] (mean)
Time per request:       13.022 [ms] (mean, across all concurrent requests)
Transfer rate:          708.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.9      1      23
Processing:    32  259 757.4    109    6489
Waiting:       28  243 732.0    101    6305
Total:         34  260 757.4    110    6490

Percentage of the requests served within a certain time (ms)
  50%    110
  66%    120
  75%    127
  80%    132
  90%    162
  95%   1042
  98%   3192
  99%   4713
 100%   6490 (longest request)

Conclusion

As you can see, it is just a matter of install varnish with a very simple and basic configuration to improve server performance a lot. Being able to handle 250+ request per second in a 256 MB RAM server with Drupal CMS is not that difficult.

Once again, this is only valid for anonymous users, that is if you have a blog or a news or tutorial site, where your visitors does not need to be logged in to interact with your content. If you need this level of performance for logged users, then you need to look at memcached, APC and the like.

Note: All tests were run from another dedicated Cloud Server using internal IPs to access the Nginx and Apache servers, so there is no Bandwidth Limitation.