Optimize VPS server for Drupal Hosting | Using Varnish
Written by Guillermo Garron
Date: 2012-04-08 14:26:00 00:00
Introduction
In the past, and for some years I've run my blog with the help of Drupal, in that time and in the first months, I've got Slashdoted, and Dugg three times, all three times my server went down.
Since Then I've become obsessed with tweaking my server configuration to support the load of Slashdot, Digg and the like.
I'm not running my blog over Drupal anymore, but I still like Drupal a lot, and this weekend I've been playing with Drupal 7 and Varnish, to see how it performs, under heavy load.
I was trying to figure out a way to optimize the Drupal configuration without having to tweak too much into the Drupal or server configuration, and without the need to add too many "performance" modules.
The environment
Here is my configuration details:
- Arch Linux 2011.10
- RackSpace VPS
- 256 RAM
- Apache/PHP/MySQL/Varnish
Configuration
Drupal
I'm using the basic Drupal 7 installation, with core cache turned ON.
LAMP
LAMP is the standard available in Arch Linux by the time of this writing, and no special configuration to any of the components. Except that Apache is listening to port 8080 instead of port 80. So it can server pages internally to Varnish.
Varnish
From Wikipedia:
Varnish is an HTTP accelerator designed for content-heavy dynamic web sites. In contrast to other HTTP accelerators, such as Squid, which began life as a client-side cache, or Apache and nginx, which are primarily origin servers, Varnish was designed from the ground up as an HTTP accelerator. Varnish is focused exclusively on HTTP, unlike other proxy servers that often support FTP, SMTP and other network protocols
Varnish is going to support the load, but once again the configuration is pretty basic:
/etc/conf.d/varnish listing:
VARNISHD_OPTS="-a 0.0.0.0:80 \
-b localhost:8080 \
-T localhost:6082 \
-s malloc,64M \
-s file,/var/cache/$INSTANCE/varnish_storage.bin,1G \
-u nobody -g nobody"
VARNISH_CFG="/etc/varnish/default.vcl"
/etc/varnish/default.vcl listing:
backend default {
.host = "127.0.0.1";
.port = "8080";
}
sub vcl_recv {
// Remove has_js and Google Analytics __* cookies.
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+| has_js)=[^;]*", "");
// Remove a ";" prefix, if present.
set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
// Remove empty cookies.
if (req.http.Cookie ~ "^\s*$") {
unset req.http.Cookie;
}
// Cache all requests by default, overriding the
// standard Varnish behavior.
// if (req.request == "GET" || req.request == "HEAD") {
// return (lookup);
// }
}
sub vcl_hash {
if (req.http.Cookie) {
set req.hash += req.http.Cookie;
}
}
Testing
I've used ab
tool to test, as this is a test to prove the Drupal site will be able to manage a spike in traffic from Digg or John Grubber, then ab
is OK. If you plan to have thousands of pages and ten thousands pages views per hour, distributed all across the content, this may not be for you, but if only one or a few pages are popular at a time, this is the right place to be.
This is the command:
ab -n 10000 -c 100 http://10.179.138.178/drupal/?q=node/2
Where:
-n: Number of requests -c: Number of concurrent sessions
The results:
Server Software: Apache/2.2.22
Server Hostname: 10.179.138.x
Server Port: 80
Document Path: /drupal/?q=node/2
Document Length: 8866 bytes
Concurrency Level: 100
Time taken for tests: 39.358 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 95050000 bytes
HTML transferred: 88660000 bytes
Requests per second: 254.08 [#/sec] (mean)
Time per request: 393.577 [ms] (mean)
Time per request: 3.936 [ms] (mean, across all concurrent requests)
Transfer rate: 2358.43 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 115 205.8 101 4696
Processing: 3 277 139.6 297 2946
Waiting: 1 141 129.8 147 2546
Total: 6 392 242.9 399 4948
Percentage of the requests served within a certain time (ms)
50% 399
66% 400
75% 400
80% 401
90% 448
95% 450
98% 499
99% 1399
100% 4948 (longest request)
After this, I've transfer that same page to an Nginx server running on a mirror Arch Linux powered server.
I've done that using curl
curl 10.179.138.178/drupal/?q=node/2 > /srv/http/drupal.html
And then run ab
against Nginx with the static page, the result was:
Server Software: nginx/1.0.14
Server Hostname: 10.179.129.x
Server Port: 8080
Document Path: /drupal.html
Document Length: 8866 bytes
Concurrency Level: 100
Time taken for tests: 38.353 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 91860000 bytes
HTML transferred: 88660000 bytes
Requests per second: 260.73 [#/sec] (mean)
Time per request: 383.533 [ms] (mean)
Time per request: 3.835 [ms] (mean, across all concurrent requests)
Transfer rate: 2338.97 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 1 129 332.9 101 9051
Processing: 1 253 80.0 249 749
Waiting: 1 130 63.7 141 617
Total: 4 382 340.4 350 9300
Percentage of the requests served within a certain time (ms)
50% 350
66% 399
75% 400
80% 400
90% 401
95% 450
98% 652
99% 901
100% 9300 (longest request)
As you can see even though Drupal is not using boost, and it is full dynamic content, Varnish is making it equivalent to a static site. The results are almost the same in both tests.
Just to let you see how it performs without Varnish, here is what happens when Varnish is taken aside and Apache/PHP/MySQL support the full load.
Well: with the same load, MySQL hanged up, and all Operating System halted. I had to reboot the server from the Console.
So lowering the load:
ab -n 1000 -c 20 http://10.179.138.178/drupal/?q=node/2
The result is:
Server Software: Apache/2.2.22
Server Hostname: 10.179.138.178
Server Port: 80
Document Path: /drupal/?q=node/2
Document Length: 8866 bytes
Concurrency Level: 20
Time taken for tests: 13.022 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 9445000 bytes
HTML transferred: 8866000 bytes
Requests per second: 76.79 [#/sec] (mean)
Time per request: 260.443 [ms] (mean)
Time per request: 13.022 [ms] (mean, across all concurrent requests)
Transfer rate: 708.30 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 0.9 1 23
Processing: 32 259 757.4 109 6489
Waiting: 28 243 732.0 101 6305
Total: 34 260 757.4 110 6490
Percentage of the requests served within a certain time (ms)
50% 110
66% 120
75% 127
80% 132
90% 162
95% 1042
98% 3192
99% 4713
100% 6490 (longest request)
Conclusion
As you can see, it is just a matter of install varnish with a very simple and basic configuration to improve server performance a lot. Being able to handle 250+ request per second in a 256 MB RAM server with Drupal CMS is not that difficult.
Once again, this is only valid for anonymous users, that is if you have a blog or a news or tutorial site, where your visitors does not need to be logged in to interact with your content. If you need this level of performance for logged users, then you need to look at memcached, APC and the like.
Note: All tests were run from another dedicated Cloud Server using internal IPs to access the Nginx and Apache servers, so there is no Bandwidth Limitation.