Computing stuff tied to the physical world


Server upgrade

In Hardware on Jun 3, 2011 at 00:01

The JeeLabs server is running out of steam…

No, not CPU load:

Screen Shot 2011 05 30 at 12.41.31

It’s idling at under 10%, and drawing 10..15W, exactly as planned.

It’s running out of RAM, all friggin’ 4 Gbytes of ’em!

There are 4 VM’s running so far, of which Drupal gets 768 Mb, and WordPress / Redmine each get 512 Mb. With Parallels, VM’s take up a bit more memory than the raw amount, I suppose there’s some caching going on.

It’s bordering on the ridiculous. The entire WordPress MySQL database is probably not more than a few dozen Mb by now (which in itself is silly, but that’s another story). Yet it needs more than an order of magnitude more RAM to serve it all up as web pages?

I can’t help but think that a fully RAM-based database (with disk image backup) could dramatically reduce the memory needs. But hey, I’m not in the business of redoing the database world – well, not these days anyway ;)

The shining exception is the nginx reverse proxy I use: it runs in 128 Mb, and would probably run just as fine with 64 Mb. It serves lots of static pages, and handles all the re-routing for the rest of the traffic. Brilliant software.

So I bought a fresh pair of 4 Gb memory sticks off eBay (thx, BWired):

Dsc 2541

With the latest Mac Mini models, memory upgrading is – at last – trivial. You might have seen the backSoon server for a few minutes, which is just a JeeNode USB with EtherCard presenting this (dynamic!) web page:

Screen Shot 2011 05 30 at 18.56.17

So there you go. All the JeeLabs .org and .net sites (and a few more) now run with a bit more breathing space. We’re solidly back in the green again:

Screen Shot 2011 05 30 at 19.01.55

Onwards!

Update – Looks like software always expands to consume all available memory: this time it is Apache, btw. I probably need to tweak it, it just grows and grows and grows! (the Drupal VM is now 1.5 Gb)

Screen Shot 2011 06 03 at 11.15.42

  1. It’s exactly as expected.

    MySQL is a toy. In a real production environment it will break your neck by choking on it’s own overhead.

    Seen this in far too many startups.

    Really like your setup though!

    anti 3 June 2011 at 10am

  2. You can probably make apache do miracles by cutting down on the maximum number of threads it creates. You can do this by editing the httpd-mpm.conf

    Vliegendehuiskat 3 June 2011 at 1pm

    • Thx. The Drupal VM is running Debian Squeeze, and I see /etc/apache2/apache2.conf has all the settings. Surprisingly, I only see 3 apache processes, and they are not even that huge:

        PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30841 www-data  20   0  492m 361m  16m S  0.0 24.0   1:13.94 apache2
31146 www-data  20   0  300m 262m  16m S  0.0 17.4   0:38.92 apache2
31628 www-data  20   0  112m  74m  16m S  0.0  5.0   0:07.00 apache2
  685 mysql     20   0  146m  32m 2944 S  0.0  2.1   6:38.63 mysqld 
31319 root      20   0  4452 3208 1400 S  0.0  0.2   0:00.21 bash

      Apache is usign prefork, with these (default?) settings:

          StartServers          2
    MinSpareServers       1
    MaxSpareServers       3
    MaxClients           10
    MaxRequestsPerChild 500

      Free mem tells me:

                   total       used       free     shared    buffers     cached
Mem:       1543712    1509308      34404          0       5108      35624
-/+ buffers/cache:    1468576      75136
Swap:      1179396     230828     948568

      I wonder what’s going on. Suggestions welcome.

      jcw 3 June 2011 at 2pm

  3. I’m starting to think it’s VM related. Parallels gives 1.5 GB of ram to each vm at max. So if it would buffer everything to ram, you’d have 6GB fully used. The portions that are unused (blue) seem to indicate that this memory is reserved but indeed unused by the VM’s.

    However, your guess will probably be as good as (or maybe even better than) mine.

    Vliegendehuiskat 3 June 2011 at 4pm

    • Wait – not all VMs get 1.5 Gb… the problem I am trying to understand is why within the Dropal 7 VM all memory gets used up.

      jcw 3 June 2011 at 4pm

    • Excuse me, but my knowledge about parallels seems to be a bit outdated, since it can allocate up to 8GB (max) of ram per VM today, but since you’ve apparently set the max to 1.5 GB on the VM’s my previous guess still stands.

      Vliegendehuiskat 3 June 2011 at 4pm

  4. Is there any background information about your setup? I’m interested to host some virtual machines and make available some services for vpn, filestorage, mails, sync between pc, netbook and mobilephone or just tinkering online. Although i would host it in VirtualBox on a Windows machine i would like to know how you interfaced all those VMs to each other and which role the nginx plays. If you have some good sources for information on this topic, this would be also fine. Thank you in advance, Philipp

    Philipp 3 June 2011 at 11pm

    • The Nginx server is set up as a reverse proxy, i.e. it serves a couple of static sites, and forwards all other domains to different IP addresses, each to a separate VM – normally running on the same box. For the VM images I use TurnkeyLinux, which has ready-made setups for all sorts of apps, and which includes a fantastic backup system to the cloud (I use Amazon S3).

      What this gives me, is the flexibility to move VMs around within JeeLabs, but also to run some or even all of them as Amazon EC2 instances. I hope it’s never going to be needed, but that means complete server migration is possible without pulling my hairs out.

      (Correction: TurnkeyLinux, not TurkeyLinux :)

      jcw 3 June 2011 at 11pm

  5. Update: Drupal VM memory use is down again. I tweaked some Apache and MySQL settings and rebooted. The forum is still responsive, but using only about 140 Kb of RAM so far – over 1 Gb less than before.

    jcw 4 June 2011 at 12am

  6. thanks for the background information ;)

    Philipp 5 June 2011 at 2am

Comments are closed.