Computing stuff tied to the physical world

DIY versus outsourcing

In Musings on Jan 28, 2012 at 00:01

Currently on the front page of the JeeLabs shop:

Screen Shot 2012 01 27 at 16 15 38

The benefit of doing everything yourself, is that you can make things work exactly as you want them.

The drawback of doing everything yourself, is that you have to do everything yourself…

Having become pretty independent in my work areas, my hobbies, and my income streams over the years, I know all about those trade-offs. Or at least I think I know about most aspects of this DIY-vs-outsourcing range.

It’s a bit like trying to stay on your feet with a floor covered with marbles…

Example: I used to rent a web server (a real physical one, with full root access and Linux on it). No worries about hardware outages or connectivity details. Being housed at an ISP with thousands of servers, means they’ll have round-the-clock watchdogs and support staff, and will jump into action the minute something is seriously wrong.

At the same time, I had total control over the web server software and operating system configuration. With a Linux distribution such as Debian, maintenance was delightfully simple (“apt-get update && apt-get upgrade”).

The flip side is that I had to choose and configure a web server (“lighty” / lighttpd at the time), and technologies to create dynamic database-driven websites (I built my own back then, based on Metakit – my own database).

Did it work? Sure. Did it evolve? Nope. Too busy. Didn’t want to risk breaking anything.

Only thing that setup did was track security updates (automatically). I had two break-ins over the 10 years that this went on. Learned more about rootkits than I care about (they’re evolving to amazingly sophisticated levels).

Did I learn a lot? You bet. And some of that knowledge is priceless and timeless. Big, big benefit.

But I also had to learn lots of stuff I really care very little about. For me, network routing, package installation dependencies, mail server configuration, and lighttpd configuration were a waste of time. The latter because lighttpd wasn’t really kept up to date very actively by its developer(s). Other options became more practical, meaning that all that lighttpd-specific knowledge is now useless to me.

The story is repeating itself right now. Redmine, which I use on http://jeelabs.net/ is not up to date, because I haven’t found a simple upgrade path. The difference is that it’s not just me not updating my stuff, I now have the same stagnant state with stuff from others. So what’s the point of Redmine? As far as I’m concerned, it’s a dead end (luckily, everything in there is stored in Markdown format – a solid long-term standard which I also use for the forum and the weblog).

With the forum, running on Drupal, it’s different again. Module updates are automated more or less, so I tend to track them from time to time. But Drupal itself is a little harder to update. And sure enough, it’s falling behind… With Drupal, I’m also running into not being knowledgeable enough to put it to really good use.

But the reason for writing this post is a different one – see the message at the top.

For the web shop, I use the Shopify web store service. They have the servers (at Rackspace – very good ops, I’ve used them for a couple of years). And Shopify develop and run the web store software (using Ruby on Rails).

They take care of dealing with nasty things such as possible DoS attacks, heavy data security, financial gateway interfaces – lots of important issues I no longer need to worry about. So far so good.

But they have their own agenda:

  • some things don’t change, and that’s good: it works, the shop is operational
  • some things don’t change, but that’s bad: years have gone by, and they still haven’t got a clue about VAT
  • some things change, and that’s good: improvements to the service, new features for customers
  • some things change, but that’s bad: they change their API and their XML data structures

That last one is what bites me now. I created a little scripted setup whereby I always pull information about orders from their shop database, to fill my database here with all the details, so I can generate paper invoices, and do the fulfillment of orders here. Doing this here was necessary to be able to do the Value Added Tax thing properly, as required by law and as my accountant wants it, of course.

So to summarize, the choices are:

  1. do everything yourself (and pay in time)
  2. outsource everything (and pay in money)
  3. choose a mix (and deal with the interface changes)

Everything is a trade-off, of course. In my case, I’m moving more and more to #1 as far as operational choices are concerned (own server, own fiber connection), and #2 as far as day-to-day software use is concerned (solid, but actively developed open source software, and Apple hardware + Mac OSX for my main workplaces). These choices are optimal for me, in terms of cost and stability.

The choice to host my own servers was made a lot simpler because I’m running VM’s for the different sites, built from ready-to-run images from TurnKey Linux. What makes them (and others, like Bitnami) different, is that all VMs are automatically backed up to the cloud (Amazon S3 in my case). The way TKL does this is really clever, reducing the amount of data in incremental backups, even for all the records stored in MySQL. So not only are my VM’s pre-configured and run out of the box, they automatically self-update and they automatically self-backup – if anything goes completely wrong, I can switch to cloud-based instances and be up and running again in no time.

TurnKey Linux is an example of using third-party stuff to side-step (and in fact avoid) a massive amount of effort, while retaining maximum operational flexibiity. My Amazon S3 bill is a whopping $1.01 per month…

But the web shop setup at Shopify is far from optimal. It was supposed to be choice #2, but ended up being #3 due to the mismatch between what I need (a European shop with correct VAT handling) and what they offer (flashy stuff, aimed at the masses). In hindsight, it was a bad choice, but I really don’t want to do this myself.

Oh well, I’ll suffer the consequences – will fix my scripts and get everything going again by next Monday!

PS. My little presentation yesterday at HackersNL #5 can be found here (PDF) – for those who read Dutch.

  1. I’ve already been to a demo like this from you before. :-)

  2. What exactly is the problem with Redmine? I didn’t get this 100%, and as I plan to introduce it for my workplace this kind of information would be helpful before going with the wrong tool.

    PS: Switching fast paced through the slides I realized how much different Nodes and Plugs you already developed.. it’s just amazing!

    • To work, I need software which can easily be kept up to date. For web-based systems, having to type in a couple of commands on the command line is tedious, if it’s more than 1 or 2 commands.

      WordPress is the shining example: notifications in the admin web back-end when new WP or plugin versions are available, a button in the web interface, click, and I’m done. Minor updates, major updates, anything. I’ve kept WP and the plugins I use up to date for years. I don’t have to remember anything.

      Drupal: almost as good – for plugins, that is. Drupal itself: command-line only, N steps to type, don’t forget to save/restore X, etc. In the real world, that’s called a mine field.

      Redmine: self-updating? Forget it. Go through a large list of things to do, must use the command line, please backup first, can only upgrade from a specific X to a specific Y version, and for the two plugins I use, the process is virtually non-existent. I can pick a new TurnKey Linux image, but then I’ve got to somehow import all my data again (database & images). And it doesn’t solve anything, because one year from now I have to do it again.

      In terms of features, Redmine is fine. In terms of install-and-forget, not so good.

      This might not seem like a big deal. But over the years, I end up wishing I could do everything with TurnKey Linux and WordPress: both are solid + secure + feature-full + evolving + up-to-date + effortless.

    • Ah, I see.. Thanks for the clarifications. This should not be such a problem for our environment. What is more annoying in my viewpoint is that all the best ones of every kind of tool seem to use another programming language.

Comments are closed.