The Developer's Toolbox

Posted on | 1122 words | ~6mins

Simple, Secure, Repeatable

Most of us live a connected life by default. I have six computers and three mobile devices that hold bits and pieces of my work and personal lives. As a software engineer that may be on the high end, but surely it’s not uncommon to have a computer or two, a shell account or VPS, a phone, and perhaps a tablet too.

On each device, I need different parts of my digital life:

  1. accounts and passwords
  2. configuration files and settings
  3. applications
  4. documents, source code, and so on.

Upgrading or replacing devices and restoring my usual environment has always been painful, so I resolved to automate a large part of this process once and for all. In devising a system to manage this data, I came up with the following goals:

  1. Changes to a file or configuration should be reflected on all my devices.
  2. All my information should be reasonably secure. Any cloud storage used should involve encryption that I control. Two-factor authentication should be used everywhere it possibly and sensibly can.
  3. Setting up a new computer or virtual machine with my comfortable development environment should take less than a minute or two, and involve minimal manual effort.
  4. There should be a thought out and tested disaster recovery plan, one in which I could bootstrap my environment from an encrypted cloud-based backup after losing all devices.
  5. Not every device needs or should have information; setup needs to be selective.

Over the years I have evolved a workflow and set of tools to achieve all of these goals. Here’s how I do it.

Accounts and Passwords

As of the time of writing this post I have 383 online accounts. This number is both ridiculous and common. The most popular approach to handling this many accounts in a sane way is either to use a password manager or reuse passwords. Reusing passwords is a terrible, horrible idea. Are you sure that every web application you use is correctly using bcrypt, scrypt or PBKDF2 to store one-way encrypted passwords? If one of the sites using your shared password is compromised, how many other passwords would you need to change? Do you even know if a site is compromised? Does the site owner know?

If you can’t re-use passwords, and it is not feasible to try to remember 383 passwords, the only sane option is to use a password manager. I happen to use 1Password, though LastPass and others are good options too. Since I don’t memorize passwords, in every case I create long and complex passwords and store them in my database. Here’s one that I just asked 1Password to create:

6!fuwq8Wx!hS1LWbzBR5ryxPDRKh1Ig`cKdxtCnEPLjHudEkoeF

These passwords, as well as other account information like user name, URL, notes, etc., are all stored in my passwords database file, which is encrypted locally and then synced to all of my devices via 1Password service. I have access to all of my accounts and passwords from every device with only a very slight loss of convenience and huge boost in security.

My password database is the key to my entire digital life – it contains my GPG private keys, scans of my driver’s license, passport, credit card numbers, and other important documents and notes. In addition to hosting my encrypted data in the 1Password cloud, I also use their command line application and the 1pass wrapper application I wrote to keep an encrypted archive in an offsite, 3rd-party email account. In the event that I lose all devices, I can recover everything by remembering only two passwords.

Configuration Files and Settings

I do non-trivial amounts of software development on at least three systems – my laptop, home desktop, and a VPS shell account. Often I am working on work software, but sometimes I play with my own projects or help out friends with their projects and websites. Where possible, I want to keep these various lives separate; files like SSH keys need to be handled thoughtfully, and debug and configuration settings for one system should not interfere with another. Only the needed components should be installed into the accounts that need them.

Installing these bits of configuration should be easy too (I’m lucky in that I upgrade computers fairly often) but VPSs also are designed to be easily blown away and re-provisioned, and I don’t want a carefully-but-manually setup configuration to prevent me from doing that.

To enable consistent, partitioned, quick and easy config file management, I built a little tool called wsup to take care of all the grunt work. I could have used Puppet or Ansible or similar tools, but I wanted something very simple that I wouldn’t forget how to use over time. In addition, my configurations are typically managed in Git repositories. I make changes to local clones of these repositories in various machines, and I don’t want to have to remember to manually copy those changes back to a master Puppet repository.

wsup is built around the concept of named layers – each of which is backed by a Git repository. Using wsup, I can go from a bare-OS to a full work development environment like this:

$ bash <(curl -fsSL https://raw.github.com/dcreemer/wsup/master/bin/wsup)
$ wsup add dotfiles dotemacs <url-to-private-work-config-repository>

This first command bootstraps wsup into my ~/bin directory (and installs any requirements like Git if needed). The second command downloads my personal configuration files, my Emacs configuration, and my private work configs and installs them into my home directory with symbolic links. wsup makes it easy to add and remove layers. For example, I can add my (hypothetical) Vim config and unlink my dotemacs configs like this:

$ wsup add vim
$ wsup unlink dotemacs

or switch back just as easily:

$ wsup unlink vim
$ wsup link dotemacs

For more on wsup, see wsup’s GitHub repository.

Applications

In additional to configuration files, I need applications to be pre-installed on each environment. Operating systems provide package management for this task (more or less; Mac OS X is somewhat deficient here). wsup provides a “postinstall” script so I can install applications as needed. For example adding this:

if [[ "$OS" == "Darwin" ]]; then
    brew update
    brew install aspell
fi

will install aspell on Mac OS X. My personal wsup configuration script installs Emacs, Markdown, Leiningen, and many other tools. With brew cask I can even add:

brew cask install google-chrome

and have my Mac OS X applications automatically installed.

Documents

Once configuration and tools are installed with wsup, I can synchronize my data. I have a separate post on how that all works.

Summary

With a password manager, a configuration manager, and a workflow encoded in some support scripts, it’s possible to have a secure, backed up, simple-to-deploy, and relatively easy to manage work environment.