A lot has changed under the hood since my last post about the servers that power my sites. In particular, the goal of reducing the mental load of managing the servers (and the code that runs on them) has been important to me because of the fragmented time I have to work on them, and thanks to a variety of great open source tools, I've slowly settled into a process that seems sane and solid.

Here are a few changes that happened over that time, in no particular order:

• Migration from Amazon (AWS) to Digital Ocean DO's servers are fast (they all have SSD), affordable, and generally reliable. Their web management interface is simple and elegant, and they even have a great API for managing all your cloud instances. Since what I do doesn't really require the scalability of EC2, and since the cost of a medium server on DO was still less than the reserved pricing of an EC2 small instance, I decided to move everything over.

• Adopting Ansible Ansible is awesome and has been the cornerstone of all my processes and allows me to spin up new cloud instance and push code to the servers in a sane, reproducible and idempotent way. The YAML scripting takes some getting used to, but writing the scripts forces you to break down your actions in a self-describing sort of way. I can write individual scripts to spin up webservers or db servers (though I don't have one anymore) and be explicit in what I want on each machine.

  Changes are never made directly on the servers, but only as a part of a push from Git using Ansible. Another interesting side effect of breaking down the actions is that you really distinguish what user each action runs under, and in practices, you should do as little as technically possible as root. One thing that I should track better though is the exact versions of my dependencies, especially if they are just built from source.

• Python 100% I had started out with PHP, which with the right frameworks (ie. Kohana) can be pretty solid and well architected. But I found myself context switching too much between C++/Java at work, Python for other projects, and PHP for web projects, and decided to consolidate to a Python web code base for all personal projects (where appropriate).

  In addition to the simplicity of the language, Python has a plethora of libraries to do all sorts of things, including a beautifully simple web framework called bottle.py, which powers all my sites. Migrating from Kohana to Bottle was a slow, but refining, process with the final code smaller and easier to understand.

  In between NGINX and Bottle is uWSGI, which runs in emperor mode and manages application logging and spinning up new processes seamlessly as the code changes.

• BitBucket Repos I was previously running my own Git server (Gitosis) instance on my old server (which is highly negligent), so migrating that over to BitBucket's code repositories seemed like a good option. They are fast, have private repos, and a decent web interface. As mentioned above, all code is pushed to Git before being pushed to the server(s).

• Logging and Stats Another thing that I wanted out of the technical changes was to start logging and capturing stats on the servers, just in case anything interesting ever happened (nothing so far :). As such, I started using Collectd and Monit. Collectd in particular is pretty amazing, and supports collecting information from a variety of sources with the help of a number of plugins. Common metrics like CPU, Disk, Network usage work out of the box, but also things like statsd, and more.

  For storing and presenting the stats, I currently use Graphite and Leonardo. Mainly because I can run both of those through uWSGI instead of having a separate server. It's not pretty, but it works.

Graphite-powered graphs supplied by Collectd

All in all, it has been a good learning experience to recreate the foundation from the ground up. In particular, I no longer worry about what happens if the server goes down (or worse, the instance disappears), and I also appreciate the "staging" process by which changes are made. It's slower and more deliberate, but there is significantly less confusion about the state of the all the moving pieces, which is invaluable.