Official HHVM packages are only distributed for Ubuntu and Debian, but thankfully some enterprising people have packaged them up for CentOS. So with a quick stand up of a Virtual Machine and some additional packages, we were good to go. To test, we chose a fairly complex site that we developed last year, built on the usual Panels and Display Suite with significant relationships between the content. This meant that on any given page, Drupal was likely to load and render several entities beyond just the one on the page.
I took the current PHP and database for the site and stood it up on a local VM with 4 cores and 6.5 GB of memory. We did some minor optimization of FPM to set the number of servers, along with some increases to the caches in MySQL. Other than that, it’s pretty much a vanilla installation of PHP 5.3 with XCache, Percona 5.5, and nginx. The goal was to provide as much of an “apples to apples” comparison of the two interpreters without as much regard for making everything as fast possible. We then spidered the site and sampled out 1000 URLs at random and ran JMeter to generate 30 concurrent requests against the site running with HHVM and PHP-FPM. We recorded time and load on the server.
Starting off, we ran the test against a cold start of the site. We cleared the Drupal cache and restarted nginx, PHP-FPM and MySQL. We then hit the home page with a single request to build the persistently cached items. As the requests started ramping up, the time to complete each one went up as expected.
What was happening on the server mirrors that.
We checked the actual processes that were generating the CPU and memory usage, in this case only pulling out mysqld since tracking each process from PHP-FHM was challenging.
MySQL was using anywhere from 40 – 60%, with a few spikes to just over 100% of one core. All the other cores were entirely used by PHP-FPM. Similarly, MySQL was using, on average, about 300 MB of memory. I’m not entirely sure what was causing the areas where processing pauses and response times spike. I saw them on all scenarios, and my guess is that they were due to some sort of I/O blocking, maybe from MySQL or nginx.
As a second test, we ran the exact same URLs against the system, clearing the Drupal cache to see if a burn in iteration for XCache and MySQL would help. Overall, it reduced the median response time by a whole 400 milliseconds and increased the throughput by a whopping 2.081 pages per minute. The response times in general were a little more choppy with fewer of the peaks and valleys from the previous run, but the results were pretty consistent.
After that we restarted the machine, switching out PHP-FPM for HHVM. Thankfully, it supports fastCGI, so it was a simple matter of altering the nginx configuration slightly. The setup was the exact same as for PHP-FPM; we made sure that the Drupal cache was cleared and restarted MySQL and nginx. We then hit the homepage with a single request and started up JMeter.
The first thing we noticed was that it completed in just under half the time, 6:28 as opposed to 13:21. Every part of the response graph was better, and both the peaks and valleys were significantly lower. The little hiccup at the very start before it dropped, was the JIT compiler running.
Looking at the server stats, those were improved as well.
HHVM runs as a single process, so we were able to capture CPU and memory usage separately for them.
Just looking at these graphs, it’s pretty easy to tell a couple of things. Namely, both CPU and memory usage for HHVM are improved over PHP-FPM. Peak memory usage for HHVM was just about 320 MB for a total system usage of approximately 20% compared to the 25 – 26% under PHP-FPM. Likewise, total CPU usage was lower with only a couple of spikes to over 90% and the median closer to 60%, compared to consistent spike to 95% CPU and a median closer to 70 – 75%.
Similar to the PHP-FPM test, we also ran the scenario against a warm start of HHVM. Like PHP-FPM, there was very little difference, only about a 200-millisecond difference in median response time.
If we are to look at the high level, HHVM compares very well to PHP-FPM.
We saw more than double the throughput and less than half the average response time. Combined with the decrease in system resources needed, it’s a pretty compelling argument to switch to HHVM.
There are some important considerations, however. The biggest is that while the HHVM team is attempting to get as close to Zend PHP as possible, they aren’t there yet. As of the latest reports, HHVM passes 99.83% of the unit tests for Drupal, but there’s no idea how whatever idiosyncrasies exist in various contributed modules will affect it. For instance, we couldn’t get GD to work at all, despite all indications that it should. Thankfully, it’s easily replaceable with ImageMagick – for most manipulations. In fact, we didn’t run into any pages during our test that failed with HHVM, but you never know what might not work until you actually run into it. In addition, the testing was done entirely on the front end. While we went through a couple of common scenarios on the administrative side, we didn’t test that thoroughly. Some PHP modules have been ported, such as APC and memcache, and there is work by third parties to add others, such as MongoDB, Ice and Redis, but many modules haven’t and probably will never be.
It’s also a moving target. Right now the HHVM team is looking at around an 8-week release cycle. Presumably, there won’t be significant regressions as they move forward, but you never know. Similarly, they are targeting Ubuntu and Debian for official packages, so if you’re running Fedora or CentOS you have to either build from source or depend on a third party repository that may not be up to date.
The performance results shown above are from running a production site on a very much ‘non-production’ virtual machine running on MacBook Pro with a 2.3 GHz Core i7 processor. There was very little tuning on any portion of the stack to ensure best performance. Load testing was performed from a separate machine over a wireless network, albeit one that was not being used for any other purpose. The load testing did not include any wait time or requests for non-PHP assets and was not intended to simulate real usage, merely to benchmark the performance of the PHP interpreters.