A lot has changed under the hood since my last post about the servers that power my sites. In particular, the goal of reducing the mental load of managing the servers (and the code that runs on them) has been important to me because of the fragmented time I have to work on them, and thanks to a variety of great open source tools, I've slowly settled into a process that seems sane and solid.

Here are a few changes that happened over that time, in no particular order:

• Migration from Amazon (AWS) to Digital Ocean DO's servers are fast (they all have SSD), affordable, and generally reliable. Their web management interface is simple and elegant, and they even have a great API for managing all your cloud instances. Since what I do doesn't really require the scalability of EC2, and since the cost of a medium server on DO was still less than the reserved pricing of an EC2 small instance, I decided to move everything over.

• Adopting Ansible Ansible is awesome and has been the cornerstone of all my processes and allows me to spin up new cloud instance and push code to the servers in a sane, reproducible and idempotent way. The YAML scripting takes some getting used to, but writing the scripts forces you to break down your actions in a self-describing sort of way. I can write individual scripts to spin up webservers or db servers (though I don't have one anymore) and be explicit in what I want on each machine.

  Changes are never made directly on the servers, but only as a part of a push from Git using Ansible. Another interesting side effect of breaking down the actions is that you really distinguish what user each action runs under, and in practices, you should do as little as technically possible as root. One thing that I should track better though is the exact versions of my dependencies, especially if they are just built from source.

• Python 100% I had started out with PHP, which with the right frameworks (ie. Kohana) can be pretty solid and well architected. But I found myself context switching too much between C++/Java at work, Python for other projects, and PHP for web projects, and decided to consolidate to a Python web code base for all personal projects (where appropriate).

  In addition to the simplicity of the language, Python has a plethora of libraries to do all sorts of things, including a beautifully simple web framework called bottle.py, which powers all my sites. Migrating from Kohana to Bottle was a slow, but refining, process with the final code smaller and easier to understand.

  In between NGINX and Bottle is uWSGI, which runs in emperor mode and manages application logging and spinning up new processes seamlessly as the code changes.

• BitBucket Repos I was previously running my own Git server (Gitosis) instance on my old server (which is highly negligent), so migrating that over to BitBucket's code repositories seemed like a good option. They are fast, have private repos, and a decent web interface. As mentioned above, all code is pushed to Git before being pushed to the server(s).

• Logging and Stats Another thing that I wanted out of the technical changes was to start logging and capturing stats on the servers, just in case anything interesting ever happened (nothing so far :). As such, I started using Collectd and Monit. Collectd in particular is pretty amazing, and supports collecting information from a variety of sources with the help of a number of plugins. Common metrics like CPU, Disk, Network usage work out of the box, but also things like statsd, and more.

  For storing and presenting the stats, I currently use Graphite and Leonardo. Mainly because I can run both of those through uWSGI instead of having a separate server. It's not pretty, but it works.

Graphite-powered graphs supplied by Collectd

All in all, it has been a good learning experience to recreate the foundation from the ground up. In particular, I no longer worry about what happens if the server goes down (or worse, the instance disappears), and I also appreciate the "staging" process by which changes are made. It's slower and more deliberate, but there is significantly less confusion about the state of the all the moving pieces, which is invaluable.

As mentioned in the bike post from last year, I ended up replacing several more components on my bike, though some parts turned out to be much more difficult than I had anticipated.

• Freewheel Ended up getting a 13T-28T (T=tooth, gear ratios are 13-15-17-19-22-25-28) Shimano freewheel to replace the old 14T-34T "Mega-range" freewheel that came with the bike. The new freewheel is smooth, has less of a silly jump from the lowest to second-lowest gear (28 to 25 vs. 34 to 24), and in my opinion the second highest gear (15T) is a better general gear. The highest gear (13T) also gives a nice little boost when going downhill too. After changing the Freewheel, I had to adjust the rear derailleur as well.

• Chain It's always recommend that you buy a new chain when replacing a freewheel since an old chain can cause more accelerated wear, but in my case, the old chain had already been severely worn and stretched (by several percent) on its own. A trick I learned to test if a chain is worn (without a tool) is to take the chain off the bike, and use a ruler to measure from the centroid of the first pin-hole in a link to the pin-hole on the next link. On a new chain, this should be about 1-inch, while on a really old chain, it'll be quite a bit longer. The new chain I got (KMC-72) also happened to include a master-link which is convenient for taking the chain off without chain riveting tool.

• Crank set This is where things went off the rails. I noticed one day that the bottom bracket was making a noise when pedaling hard, so I thought I would take the opportunity to just remove the old crank set (the chain rings and pedals) and replace the bottom bracket. Easy, right? I bought a crank set remover, screwed it into the drive-side crank arm, and screwed in the pusher, which then promptly proceeded to stripped all the threads on the crank arm.

  After two weeks of research and trying everything under the sun, including: riding hard several miles to jiggle it loose, pouring boiling water on the crank arm to expand it, using an automotive puller tool, using an automotive pickle fork, hammering (a lot), getting advice from several other bikers, more hammering, and more. I came to the conclusion that I needed some serious help, so with the help of someone at the Google welding workshop and a handy plasma cutter, I watched as 30,000°F plasma sliced through the soft aluminum of the crank arm to finally free it from the bottom bracket.

  
  Ta-da! Crank set, post plasma cutting :)
  It's still a mystery to us exactly why the drive side crank was so stuck, but given how close I was to giving up and getting a new bike (a bike with a jacked up crank set is hardly useful), I am seriously considering hanging this crank arm like a trophy somewhere :) The crank set that replaced it was pretty much the same (28T-38T-48T, 170mm), and went in without a hitch once I got the new bottom bracket.

• Bottom bracket Once the crank set came off, replacing the sealed bottom bracket was pretty straight forward. The only issue was that when I took out the old BB, the two end caps that screw into the bike frame had rusted onto the frame, so I had to use a dremel and a wire brush to take off the rust, and then made sure I put on some more grease before installing the new one.

• Miscellaneous Strangely enough, even though I bought a BB with the same length as before, the new crank set seems to stick out slightly further than the old one, which means that there is a bit of cross chaining when at the highest gear in the front and the lowest gear in the back. Not a big issue, since I'm rarely in that gear configuration though. I also recently moved back into riding in the middle front gear at a higher cadence than at the biggest front gear at a lower cadence. It's less tiring on the legs, and not actually slower than before (which is slightly counter intuitive since the gear inches are the same).

My goal for this year is to bike and log 1000 miles, but I'm already behind, which means I'll have to commit to biking a bit more to work these next few summer months!