VMs are OUT, Odroid is IN Dec 2015
There are three main internet-facing web servers for JeeLabs - the weblog (jeelabs.org), the community site (jeelabs.net), and the shop (jeelabs.com). The shop is hosted elsewhere and managed by Digital Smarties, the other two have been running each in a separate VM.
This has been the setup until now:
- one VM running Nginx for serving some static data and as reverse proxy for the rest
- one VM running WordPress, Apache, PHP, and MySQL - for the weblog
- one VM running Redmine, Apache, Ruby On Rails, and MySQL - for forums & wikis
Note that three VMs help to cleanly compartmentalise these different systems. Nginx has proven to be rock solid and very low overhead, which is great as front-end for the rest. And when either WordPress or Redmine get into trouble, they don’t affect the rest much - even a runaway process will only be able to load down one of the dual CPU’s on the Mac Mini.
The drawback of 3 VMs is that you end up with 3 copies of Linux, each with their own memory management and mapping, each idling on the same host, and each running off separate virtual disk drives. Linux containers and Docker would be the new way to do this.
As a result, the Mac Mini (a 2010-vintage 2.66 GHz Core 2 Duo) had to be upgraded to 8 GB RAM. It was also upgraded to use an 128 GB SSD as root disk a few years ago.
But for some time now, WordPress has regularly been failing due to an out-of-memory error (in MySQL, it seems). Apparently 750 MB RAM plus 500 MB swap space is not enough for what is a fairly basic website, with no special plugins and comments disabled. Or - more likely - there’s probably a badly-configured setting somewhere in there…
The server stability over the past year amounts to 66 outages, for a total of 19 hours down, which translates to an uptime of 99.8%. Most of the failures were caused by WordPress.
The irony is that when you think about it, WordPress is doing almost nothing in this use case, other than serving pages and images, a few of which change once a day. That’s it!
So why not just generate the entire weblog once, as static website?
All that would be needed is to re-generate the website when something changes. No more “database-backed website”, with MySQL handling database requests for each page, and PHP turning these pages into HTML. No need to add a clever cache to make things snappy.
There is a growing trend to do just that, for sites which are static enough to allow such an approach. Or even for static parts of websites, in some cases. See this interesting article.
Here is the effect of switching all of JeeLabs.org from WordPress to a static site:
(the red lines mark when WordPress fell over again… leading to the decision to switch)
Somewhere around Dec 12th, the new setup went live. In fact, you’re reading this article on the new static setup. As you can see, all the response spikes are gone. Nginx is amazingly good at serving static files. It needs less than 40 MB of RAM to do its thing, and it eats up well under 1% of the CPU’s processing power.
If you look closely, you’ll see another drop early Dec 14th: this was when the proxying setup was removed. It’s now a single Nginx install, directly accessed from the internet router. Ping times in Europe are around 50..100 ms. US and elsewhere take a bit longer - because there are more “hops” involved (and because you can’t get around the laws of physics!).
So what’s gone: no more WordPress + Apache + PHP + MySQL. And 700 MB less RAM.
What remains (or will, soon, as jeelabs.net hasn’t been migrated yet): Redmine, Ruby-On-Rails, MySQL. It will use a new Ruby 2.x setup, which is considerably faster than 1.9.3.
This adds a nice display (which can also take over the console on boot-up), as well as room for a 2.5” SATA drive. There’s a fan, which only kicks in when needed. And best of all, this thing costs less than €200, including a fast 16 GB eMMC-5 card, but excluding a 2.5” drive. It’s all standard stuff, available from shops in Asia, the US, and Europe.
The eMMC card is not strictly needed, since there’s also a µSD slot, but it’s considerably faster as root disk (a quick test with
hdparm reports 98 MB/sec raw read access speeds).
There are lower-cost alternatives, such as any of the Raspberry Pi models, or the Odroid C1+. But the features and raw performance of the XU4 are quite appealing. The XU4 has 8 CPU cores, 2 GB RAM, gigabit Ethernet, and USB3. In fact it exceeds all the specs of the Mac Mini it’s replacing, except for RAM size. For a fraction of the price - that’s progress!
Here’s a performance comparison (by Hardkernel, so please take this with a grain of salt):
Meanwhile, the new server appears to be working nicely. CPU load is only 1..2 percent. Power consumption was measured to be about 4W of AC mains (1W for the SSD?).
Reliability is a major concern. But instead of going for premium hardware, we can in fact get similar results via redundancy and backup. Here is the
crontab setup on the XU4:
$ crontab -l # m h dom mon dow command B = /ssd/backups 0 22 * * 1 rsync -ax --delete / $B/xudroid-1/ && touch $B/xudroid-1/ 0 22 * * 2 rsync -ax --delete / $B/xudroid-2/ && touch $B/xudroid-2/ 0 22 * * 3 rsync -ax --delete / $B/xudroid-3/ && touch $B/xudroid-3/ 0 22 * * 4 rsync -ax --delete / $B/xudroid-4/ && touch $B/xudroid-4/ 0 22 * * 5 rsync -ax --delete / $B/xudroid-5/ && touch $B/xudroid-5/ 0 22 * * 6 rsync -ax --delete / $B/xudroid-6/ && touch $B/xudroid-6/ 0 22 * * 7 rsync -ax --delete / $B/xudroid-7/ && touch $B/xudroid-7/
Once every day, a full backup of the entire root disk is updated, with a separate copy kept for each day of the week. Due to the magic of
rsync, these backups take under a minute to update. The
touch command then tags the directory with the backup completion time.
This will keep a copy on a different disk in case the eMMC root disk fails, but that’s just part of the story. We still need a way to quickly switch over to some fallback server. And we need a backup outside of this main server, in case it decides to wipe itself, or whatever…