Announcing terraform-provider-linode

Iʼve been interested in HashiCorpʼs terraform ever since it was announced. I loved the idea of being able to tie multiple providers together. You could use Route 53 for DNS and Digital Ocean for your servers or you could use DNS Simple with ec2. I havenʼt been able to try it because I use Linode for both my personal servers and at work. Unfortunately, Linode isnʼt one of the included plugins in Terraform and no one had created a plugin for it. So I did what any developer with a bit of free time and a possibly unhealthy obsession with automation tools and HashiCorp products: I built one.

In its current state, itʼs fairly light on features. It only supports managing Linodes (what Linode calls their servers) and not any of the other things that Linode provides. Iʼm hoping to get Linodeʼs DNS and NodeBalancer services added in the future. terraform-provider-linode also doesn’t support all of the options that Linode provides. In particular, I intentionally omitted all of the various alerting settings that Linode provides. While adding all of them would have been easy, the sheer number of options seemed too complex for the initial release. The currently implemented features are exactly the features that I needed in order to start using terraform. If you would like to have a particular feature, it would be great if you were able to contribute it to the project. If not, please create an issue on GitHub and Iʼll see when I can get it implemented.

Terraformʼs helper/schema made developing this plugin a breeze. I expected that it would take considerably longer and be more difficult to build terraform-provider-linode. As they mention on the provider plugin page, its really just a matter of setting up CRUD functions for a particular resource. That is all that there is to it. Terraform also includes some extremely handy functions for building acceptance tests. Terraform feels incredibly polished, I find this particularly remarkable as Terraform was only released a little under a year ago. I also found Toahʼs Linode api client library to be extremely helpful for developing this plugin. Linode, unfortunately, hasnʼt created an official Go api client but, Toah really stepped up to make one.

Please check out terraform-provider-linode if it is something that would be useful to you. If youʼd like to contribute, then I would really appriciate your help.

Ansible in Production

For quite a while, if you wanted any sort of consistency in your servers, you needed a configuration management tool. This has really started to change over about the last year and a half. Weʼve seen a proliferation of tools aimed at making running a datacenter easier than managing a single machine. These are things like Apache Mesos, Kubernetes, or CoreOs. All of these tools are based on a similar idea, you tell these systems that you want to run x number of this service with these constraints and they figure out how to run them. Of course, they differ quite a bit on the details but, at a broad level, this is what they do. While I find that idea to be hugely compelling, Iʼve decided to forgo one of these systems for now. This is mostly due to running a deployment of one of them and having it fail miserably at unpredictable times. We are also not at a scale in which handling distinct servers is difficult.

With that roundabout introduction out of the way, my configuration management tool of choice is Ansible. I find describing Ansible to be a bit difficult as it is a tool with a few different use cases. In some ways, it operates like a distributed scripting language. You can define a script that will change the current master database, point all of your applications at the new database, restart the old master and then point all of your applications back at the original database instance. It is also equally useful in a more traditional view of configuration management where Ansible installs and configure individual servers in their specific roles.

Ansible has two rather broad modes of operation. One of which is ansible-pull. In this mode of operation, each server pulls down your configuration scripts and then applies them to themselves. This is somewhat similar to traditional configuration management tools like Chef or Puppet. This mode doesnʼt appear to be used very often and this is probably a good thing. Both Chef and Puppet are far superior in this mode of operation. The typical mode of operation for Ansible is push. There is a control server, this could be your machine or a server somewhere, that initiates the Ansible run. The control server then connects to all of the servers that are part of the current playbook over ssh. Each step in the playbook is then applied sequentially with all of the servers specified for that step receiving the commands in parallel. There are knobs that you are able to turn in order to control how many servers receive the commands in parallel. This mode of operation leads to a number of really cool bits of functionality. For instance, when youʼre provisioning a new web server youʼre able to immediately add it to the load balancer which, is something that chef and puppet are unable to do 1.

The terminology that is used to describe Ansibleʼs configuration is a bit strange. There are inventories, playbooks, plays, roles, tasks, host vars, group vars and modules. I think that inventories, host vars, and group vars are self-explanatory. So that just leaves all of the others. I feel like the name playbooks was inspired a little too much from sports but, it actually happens to work quite well for describing their function. A playbook is a script that you run. This can be many different things, it could be as broad as a single script that creates your entire infrastructure, it might also be like the example of provisioning a new web server or even a multistep deployment script. Playbooks can include other playbooks or plays in them and they contain how information on which servers the plays should be applied to. Plays are blocks in the playbook. Each play must have a user and a list of hosts that play should run on. Plays can include custom variables, roles, and applications of modules. Modules are the basic commands of Ansible. These are things like installing a package on the server and copying a configuration file to the server. Tasks are applications of these modules.

Roles are a set of related set of module applications. You might have a role for MySql or Postgres but given that they are the only way to include functionality between multiple playbooks (you can include other playbooks as well but that is limiting in some crucial ways), you end up using them that the name roles doesnʼt apply to. I have a database migrations role. We have a few different infrastructure configurations for our app but the migrations are always applied in the same way, so those steps were pulled out into a self-contained role. While semantically this doesnʼt make much sense, itʼs the only way to pull this common code out and being able to reuse it between different playbooks is extremely valuable. While the power of Ansible comes from being able to group roles and modules in playbooks to automate important processes, youʼre also able to execute the modules directly. You could have patched all of your servers for shellshock by simply running ansible -i inventory all -m apt -a 'update_cache=yes name=bash state=latest'.

Unlike other configuration management tools, Ansible is extremely easy to get started with. Itʼs especially easy by not having a server to set up, unlike Chef2 and Puppet. You can simply install Ansible on your machine and get started automating servers. Itʼs also easy to get started writing playbooks and roles because they are written in YAML. YAML is very easy to read and write and there isnʼt very much syntax to pick up. That being said playbooks have some expected keys and each of modules takes a set of arguments. Other than a few core modules, like apt, copy, template and service, I still have to look up the arguments every time that I use them. I really recommend Dash for referring to the documentation, itʼs great and it will greatly speed up your development time with Ansible. Roles have a rather complex (compared to the rest of ansible) directory structure but once you use it a couple of times, it will really click. That is enough to get up and running with Ansible but there are a few more advanced options that you won’t have seen and to get everything you should probably read the entirety of the Ansible docs except for the modules documentation. Itʼs really worth the effort, the documentation is quite succinct and will greatly assist you in writing your Ansible scripts.

Once you have that basic knowledge, you can start writing scripts for everything. If you do anything over ssh, you can write a playbook for it. In fact, you probably should, it will be much faster and much less error-prone than doing it by hand (if you do it more than once). The separation between what should be a role and what should be in the playbook becomes fairly clear early on, if the task that you are doing requires applying a template, using handlers, or copying a file from the Ansible control server, then it should be a role. This is due to how roles bundle these things together, it makes much more sense after youʼve been using Ansible for a while. As I pointed out previously, you should also put shared scripts steps into roles whether or not the name “role” is semantically correct. In my case, I have the steps necessary to run database migrations in an Ansible role.

In my experience, the best way to build up your Ansible scripts is by simply doing everything with them. If you need to deploy a database server, write a playbook and role for doing just that. When you inevitably find something that could be done better, add it to the role or playbook and then reapply it to all of those nodes. This way, you don&rquo;t need to recall every step that you took when setting up the next one. This is the way that most of our playbooks and roles have been developed. For example, at the time that I was deploying logstash, I had no idea that the disk scheduler should be disabled (set to noop) on ssd based machines. Iʼve since added that step to the relevant roles. This is much the same as any other configuration management tool, youʼre able to distill

Iʼve found that Ansible is extremely good at automating complex multiple server tasks. This includes things like deployments of multiple application services. Our deployments, at Signal Vine, are run using Ansible. Our deployments are quite complex, they involve reconfiguring 4 types of servers and applying migrations to 3 different data stores. Ansible has been handling this all of this beautifully. Iʼve also written playbooks for all of the complicated operation processes. There’s one for changing database settings and restarting the database with its dependent services. With another one, Iʼm able to change the postgres master and then set up replication on the demoted master. After writing a couple of these, it becomes clear how valuable they are.

Now, what would an automation tool be without having some prebuilt patterns available? Ansible Galaxy fulfills this need. Unfortunately, I donʼt have a lot of experience with it as there wasnʼt that much there a year ago 3 and Iʼm not eager to rip out working code to replace them with untested (on our servers) roles. I really like the idea of Ansible Galaxy and I really appreciate that all of the roles have a rating associated with them. It really helps you narrow down which roles you should audit. I feel that the usefulness of this roles is slightly hampered by not being able to run a role multiple times in a play by passing it different variables. This is a feature that is due to arrive in Ansible 2 but, Ansible 2 hasnʼt shipped yet. In some cases, this can be mitigated by the roleʼs author. If they make a variable that someone might want to call multiple times into an array then, they can properly handle this situation. In other cases, this isnʼt possible.

This gets into what I think is Ansibleʼs biggest sore spot, reuse and composability. While Ansible Galaxy is nice, it has no where near the utility of Chefʼs Supermarket or Puppetʼs Forge. I donʼt think that Ansible, as it stands today, will ever have anything like that. Ansible is intentionally not a programming language. While I see some advantages to that for onboarding new userʼs, I really feel like it hampers your ability to abstract things. Certainly users can go too far in abstractions but limiting them so much is also painful. One of the things that I wish that I could do is loop over a set of tasks but, that’s not possible. In a similar vein, Ansible has role dependencies. This is very helpful but, it doesn’t help you in all cases. If you have multiple roles that depend on a single role with different variables set, in a single play, Ansible will not run all of them.

In the past, Iʼve used Ansible to build Docker images. While this is entirely possible4, it is not a pleasant experience. At first, this seems like a good idea, you can use the same dependable scripts to deploy a particular service into any environment, whether that is a single monolithic server or into a container. The reality is that these are very different environments and you probably donʼt want to install your app in the same way on both. You will end up filling your roles with various conditionals to handle being able to run in a container or being directly installed on a server. This ends up being extremely unwieldy. This also doesnʼt work with Dockerʼs expected mode of operation. Each step you specify in the Dockerfile builds up a cached image layer. Then, when you change one of the steps, Docker will use these cached layers up to the point where modified it and then run the remaining steps directly. When you are using Ansible to run the provisioning, all of it happens in a single step. So to change a single thing, youʼll need to completely rebuild the container and Ansible isnʼt well optimized for working in this way, so your container builds will take a significant period of time. I would guess it will be somewhere between 1 – 10 minutes to build a container. This isnʼt horrible but, it is enough to be annoying. Instead, you should use Docker and Ansible as they were intended. Use Dockerʼs toolchain to make container build artifacts and then use Ansible to deploy those to the required servers.

Iʼve been disappointed with the testing situation in Ansible. As far as I can tell, this is the only coverage that testing has gotten. Itʼs really not enough for me. I need a little more hand-holding to really understand the picture that they are trying to paint. I havenʼt yet tried testing the way that theyʼve suggested, I just canʼt see that working well. Iʼve defaulted to mostly manual testing which is a drag. I do have a staging environment that I can apply changes to before running changes against our production environment. in a similar vein, itʼs entirely possible to write a playbook that fails when running it in check mode (--check) that works without a hitch when applying it. I do understand how this could happen but, itʼs very annoying that Ansible doesnʼt notice it and issue a warning for it. I do like that Ansible includes --syntax for checking whether the specified playbookʼs syntax is valid. Unfortunately, it doesnʼt check whether the variables are defined before they are used.

Another area that Iʼm not satisfied with, is how you expand the users of your Ansible scripts. I think is fairly reasonable to expect anyone handling operations in your company to be able to install and run Ansible from the command line. I donʼt think that works to expand it to everyone in your company. It isnʼt clear on how this can be done easily. It also becomes difficult to see when things are happening or when they have happened in the past. Ansible has a commercial product for this, Tower. Tower is an option for doing this but, itʼs both pricey and it may not be exactly what youʼre looking for. You also could set up Ansible tasks in your CI server but, then your CI server needs ssh access to all of your servers with sufficient access rights to do those tasks. That would mean an attacker could change the software running in your production environment if they were able to compromise your CI system. That isnʼt something that I would feel comfortable with.

All that being said, I think Ansible is a great product. If you arenʼt currently using a Configuration Management tool, I highly recommend that you check out Ansible. If you already have one and youʼre satisfied with it, you should probably keep using it but, you may still find Ansible useful. Itʼs very useful for doing multi-server scripting. Of the type that you might do during a deploy. Ansible is a good, dependable and efficient tool, I ʼm happy to use it.


  1. Yes, itʼs possible to make that happen with both chef and puppet but, it involves extra steps. First, you provision the new server, which registers it with the configuration management server. Then, on their next configuration run, the load balancers add the new server into the rotation. With Ansible, it can be available immediately. 

  2. Chef does have chef-solo but, itʼs not what you should use if you want to use Chef. Chef with a server is much better. 

  3. That’s when I started writing our Ansible scripts. Iʼve take another look at it and it seems fairly decent now. 

  4. And easy if you know what you are doing. Basically, you need to setup your Ansible playbook as it was being used for ansible-pull and then you need to install Ansible in the container, add your playbooks/roles, and finally invoke Ansible within the container. 

Tools in Production

About a year ago, I started a new job with SignalVine as the person in charge of Operations.While I strive to give the engineering team the tools that they need to properly manage our production systems, its my responsibility to keep all of the components of our application up and running. In the last year, Iʼve used quite a few different tools to make running our application a pleasant experience. Iʼll be covering a wide variaty of things, from databases to DevOps tools. Some of these things have worked well for me, others have not.

  • Ansible – My Configuration management tool of choice.

Writing a Riemann metric to a file

At work, I recently setup Riemann for monitoring. Riemann is a monitoring tool that works on streams of events. It includes many powerful tools to work on streams of events. As an example, it has a ddt function that will differentiate the eventʼs metrics over time. This allows you get to a rate of change for a counter. While Riemann includes many powerful tools, your Riemann config file is a Clojure program so youʼre able to extend Riemann by simply modifying your config file.

I had such an occasion to do such a thing this week. We send all of our error level log messages from Logstash to Riemann in order to alert us when we need to check the logs. Doing this is a fairly simple process, we use slack and there is a built-in function to send alerts to Slack. While we could send the whole log message to Slack, this isnʼt ideal for us. Our log messages can be quite long, many thousands of characters and sending that to Slack makes for a fairly bad experience. What we decided to do instead was write the full metric to a file and link to that file in the Slack message. Unfortunately, there isnʼt really a built-in way to do this in Riemann. You could write the messages to Riemannʼs log file but that isnʼt what we are looking for here as that results in a single large log file rather than individual log files.

What I decided to do was create a function that would write out the message to a file with the name set to the messages sha-256 hash. Generating the has was the most complicated part of this. The complication arose from my lack of knowledge on the various libraries that can generate a hash. The way that I figured this out was by Googling variations on Clojure/Java sha-256 hashes and then trying them at the Clojure REPL on a checkout of the Riemann source. Unfortunately, neither of the Clojure hashing libraries are included in Riemann but, I was able to find a java package that Riemann includes that is able to generate hashes, Apache Commons. I likely would have known that if I had more experience with the Java ecosystem but I donʼt. So here is what I came up with.

(ns riemann.config
  (:require [clojure.java.io :as io]))
  (:import (org.apache.commons.codec.binary.Base64)
           (java.security.MessageDigest)))

; A couple of helper functions to shuffle data between strings and byte arrays
(defn base64-encode [^String v] (Base64/encodeBase64 (.getBytes v)))
(defn base64-decode [^bytes b] (Base64/decodeBase64 b))

; The hashing function
(defn digest [^bytes b]
  (.digest (doto (MessageDigest/getInstance "SHA-256")
             (.reset)
             (.update b))))

(defn as-hex [v]
  (-> (Integer/toString (+ 0x100 (bit-and 0xff v)) 16) (.substring 1)))

; Gets the sha-256 of a passed in string
(defn get-sha256 [v]
  (apply str (map as-hex (-> (base64-encode v) (base64-decode) (digest)))))

; Returns a function that writes an event to the specified directory as <sha-256>.txt and
; modifies the event to include {:url <url>/<sha-256>.txt}
(defn write [directory url]
  (fn [& children]
    (fn stream [event]
      (let [write-event (fn [individual-event]
                          (let [contents (reduce (fn [r x] (str r "\n" (x 0) ": " (x 1)))
                                                 (conj (seq individual-event) "Message Contents"))]
                            (let [sha (get-sha256 contents)]
                              (with-open [wrtr (io/writer (str directory "/" sha ".txt"))]
                                (.write wrtr contents)
                                (conj individual-event {:url (str url "/" sha ".txt")})))))]
        (if (vector? event)
          (call-rescue (into [] (map write-event event)) children)
          (call-rescue (write-event event) children))))))

Then all you need to do is define the function that you will use on your steams. Something like (def write-log (write "/var/www/alerts" "https://alerts.example.com")) would work where /var/www/alerts is the content directory for alerts.example.com. To include the link in your Slack alert, youʼll need to provide a custom formatter that includes a link. Here is what we use:

; Truncates the String (s) at the specified number of characters (n)
(defn trunc [s n] (subs s 0 (min (count s) n)))

; The formatter that slack will use when there isn't a link
(defn default-formatter [e] {:text (str "*Host*: " (:host e) "\n*Service*: " (:service e) "\n*Metric*: " (:metric e) "\n*Description*: " (trunc (:description e) 100))})
; The formatter that slack will use when there is a link
(defn url-formatter [e] {:text (str "*Host*: " (:host e) "\n*Service*: " (:service e ) "\n*Metric*: " (:metric e) "\n*Link*: " (:url e) "\n*Description*: " (trunc (:description e) 100))})

; Choses which formatter slack will use
(defn slack-formatter [e]
  (if (nil? (:url e))
    (default-formatter e)
    (url-formatter e)))

I know that’s a lot to piece together, so here is a minimal Riemann config that should work, to show you how to use everything.

(ns riemann.config
  (:require [clojure.java.io :as io]))
  (:import (org.apache.commons.codec.binary.Base64)
           (java.security.MessageDigest)))

; A couple of helper functions to shuffle data between strings and byte arrays
(defn base64-encode [^String v] (Base64/encodeBase64 (.getBytes v)))
(defn base64-decode [^bytes b] (Base64/decodeBase64 b))

; The hashing function
(defn digest [^bytes b]
  (.digest (doto (MessageDigest/getInstance "SHA-256")
             (.reset)
             (.update b))))

(defn as-hex [v]
  (-> (Integer/toString (+ 0x100 (bit-and 0xff v)) 16) (.substring 1)))

; Gets the sha-256 of a passed in string
(defn get-sha256 [v]
  (apply str (map as-hex (-> (base64-encode v) (base64-decode) (digest)))))

; Returns a function that writes an event to the specified directory as <sha-256>.txt and modifies the event to include {:url <url>/<sha-256>.txt}
(defn write [directory url]
  (fn [& children]
    (fn stream [event]
      (let [write-event (fn [individual-event]
                          (let [contents (reduce (fn [r x] (str r "\n" (x 0) ": " (x 1)))
                                                 (conj (seq individual-event) "Message Contents"))]
                            (let [sha (get-sha256 contents)]
                              (with-open [wrtr (io/writer (str directory "/" sha ".txt"))]
                                (.write wrtr contents)
                                (conj individual-event {:url (str url "/" sha ".txt")})))))]
        (if (vector? event)
          (call-rescue (into [] (map write-event event)) children)
          (call-rescue (write-event event) children))))))

;The function that you will call to write the event
(def write-alert (write "/var/www/alerts" "http://alerts.example.com"))

; Truncates the String (s) at the specified number of characters (n)
(defn trunc [s n] (subs s 0 (min (count s) n)))

; The formatter that slack will use when there isn't a link
(defn default-formatter [e] {:text (str "*Host*: " (:host e) "\n*Service*: " (:service e) "\n*Metric*: " (:metric e) "\n*Description*: " (trunc (:description e) 100))})
; The formatter that slack will use when there is a link
(defn url-formatter [e] {:text (str "*Host*: " (:host e) "\n*Service*: " (:service e ) "\n*Metric*: " (:metric e) "\n*Link*: " (:url e) "\n*Description*: " (trunc (:description e) 100))})

; Chooses which formatter slack will use
(defn slack-formatter [e]
  (if (nil? (:url e))
    (default-formatter e)
    (url-formatter e)))

(def notify-slack (slack {:account "{% raw %}{{Your Account}}{% endraw %}"
                          :token "{% raw %}{{Your Token}}{% endraw %}"}
                         {:username "Riemann"
                          :channel "#Alerts"
                          :icon ":rotating_light:"
                          :formatter slack-formatter}))

; Expire old events from the index every 5 seconds.
(periodically-expire 5)

(let [index (index)]
  ; Inbound events will be passed to these streams:
  (streams
    index

  (default {:severity 4}
    (where (and (service #".* logs$")(= (:environment event) "production"))
      (by [:service]
        (where (< (:severity event) 4)
          (throttle 2 3600
            (write-log notify-slack))))))))

What that config will do is send alerts to the Alerts channel of your Slack when any events are placed into Riemann that end with logs. They are limited to no more than 2 messages per hour per service.

2 ways I’m using Docker

I recently got to migrate my server from running Ubuntu 12.04 to Ubuntu 14.04. As I was unable to upgrade the kernel while on 12.04 (due to some hardware issues), I was stuck running into the bug mentioned on the installation page. It wasn’t too big of an issue but, basically, sometimes containers would just hang for no real reason. In this state, I couldn’t stop them or delete them. The only way to get rid of them was to either restart docker or restart the server.

Anyways, I’m happy that is no longer an issue for me. So I’ve been looking into moving as much as I can into Docker. I basically have two ways that I’m using it. The first of which is the one that I mention in Docker Appliances, I’m running my Discourse server in Docker using the excellent discourse_docker project. It works extremely well, especially now that it doesn’t hang randomly.

The other way I’m using it is as the execution environment for a couple of other things. What I mean by this is that both the code for the site and the database files are mounted from the host. Here is an example, this site runs in a Docker container. On my server, its code and database reside at /var/node/ruin. Since I last wrote about Ghost, I have switched over to using SQLite as my database. SQLite performance is adequate for my needs and at the time, I hadn’t figured out how to get MySQL up and running in a Docker container. So the docker container simply mounts the code/content directory and runs only the node process. I have it set to expose the default port that Ghost runs at, 2368, to the docker host. On the docker host, I run nginx and reverse proxy the traffic to port 2368. When I want to update the code, I have an ansible playbook that pulls the fresh code down, stops the current container and launches a new one.

I also have a similar set up for my unmark instance, unmark.tobolaski.com (I wouldn’t recommend going there as it uses a custom certificate authority). It mounts the code from /var/unmark/app and the mysql data on /var/unmark/mysql into a lamp container. In this case, I have nginx setup to reverse proxy to 8000 which is mapped to 80 on the Docker container.

If I was going to be putting either of these containers on multiple hosts, I’d move the database into something else, possibly a new container or maybe their own host. Then I add in the application code into the container and distribute it as a single unit, which is more like the intended usage than what I’m currently doing. Basically, I’m using Docker as lightweight virtualization, which it does extremely well.

I’ve open sourced both of the dockerfiles that I discussed here.

The ErgoDox

A photo of the ergodox

A number of months ago, I picked up a kit for an unusual keyboard from Massdrop, the ErgoDox. Unusual is a fitting word for it as it is unusual in nearly every way. The only way that I can describe the layout is unusual, the shape is unusual, the materials are unusual and you probably get the point. It’s also completely open source. You can download the PCB and case design and build the keyboard yourself. Given all of that, you also need to assemble it yourself. In spite (or perhaps because) of all of that, the ErgoDox is the best keyboard that Iʼve ever used.

Layout

As I mentioned before, the layout is unusual. It’s a split layout where you can independently move the halves. This is, by far, the most important improvement over a standard keyboard layout. This allows you to position the two halves to be shoulder-width apart. This vastly decreases the stress on your wrists while you type.

The keys are also arranged in columnar layout which removes any of the staggering that you would typically see between each row of keys. What this means is that your middle and ring fingers only move up and down. Its a little hard to imagine the benefits of this but, after a few days of use, you won’t want to go back.

Feel

Unlike most ergonomic keyboards, the ErgoDox uses mechanical switches. Many people enjoy keyboards with mechanical switches as they have a much better feel while typing. Iʼm one of those people. I donʼt mind typing on scissor key keyboards but typing on keyboards with mechanical switches is a real treat. That’s not something that I would normally say of typing. You can use whatever switches you want for it. If you already have a favorite switch variety, you can use those on your ErgoDox. If not, I really like Cherry MX Clears. They have a tactile bump about just after the actuation point. That makes it very easy to type quickly without bottoming out the keys.

Customizability

With the ErgoDox, everything is customizable. In fact, you are forced to customize it before you can use it. Luckily MassDrop has a web-based configuration tool that’s really easy to use. Its what I used to create my layout. The web configuration tool is great. It exposes almost all of the options that you might want. If that doesnʼt quite do all of the customizations that you want, youʼre able to write C to fully customize your keyboard. The base firmware is available on GitHub. The one thing that Iʼve found is that you arenʼt able to change what character is sent when you use shift with the key. This is important for doing some more exotic layouts. There is an alternate firmware called tmk that supports some more advanced features but I havenʼt tried it yet and I donʼt know whether it supports custom shift modifiers.

There are also a variety of external customization that you can do. Iʼve equipped mine with an aluminum top plate. I also chose to have clear DCS keycaps. I strongly recommend that you get DCS keycaps. It is much nicer to reach the bottom key rows as they are angled to meet your fingers. I also chose the standard case. If I had to make the choice again, Iʼm not sure what I would choose. While Iʼm working at my nextdesk, I really like the standard case. However, I now work in an office and a full hand case would work much better on a more conventional desk.

Although you can put any keycaps on your ErgoDox, there are some practical limitations. You have to be particularly careful when picking out DCS keycaps. Basically, you need to be able to buy a keycap set that was made for the ErgoDox. Youʼll also be locked into using qwerty like layout. If you go with DCS keycaps, you have quite a few more options but, youʼll still have considerable trouble finding keycap sets with all of the extra modifier keys. Youʼll likely need to buy at least the modifier key set from a set designed for the ErgoDox but then youʼll be able to use any keycap set you want for your base keys.

Assembly

My views on the assembly process have changed since the time that I built my ErgoDox. If you would have asked me about it while I was in the process of assembling it, I would have told you that it was the most tedious process that Iʼd ever done. I would have also stated that it really isnʼt worthwhile. While I still feel that the first part is true, I definitely think that the end result is worth the pain. I feel a special connection with my keyboard since I needed to assemble it. I really needed to work to reap the benefits of the ErgoDox. It has made me really appreciate the end result. Iʼm also rather proud that I constructed my primary input device using my own hands.

On to some more practical advice for assembling the ErgoDox. Youʼll need a soldering iron, solder and a tweezers. The last part isnʼt optional. The surface mounted diodes are incredibly tiny and you are not going to want to put your fingers anywhere near the tip of the soldering iron. I used this soldering iron from RadioShack with this solder. Neither of which was ideal. Youʼll likely want a slightly smaller solder. As for the soldering iron, youʼll want something a bit nicer with an adjustable temperature. The one I used frequently was hotter than I was comfortable holding.

The diodes on the ergodox circuit board

The Massdrop ErgoDox kit has a couple of choices that make assembly more difficult. Of course, its unclear whether Massdrop will be doing anymore ErgoDox kits due to their introduction of an Infinity ErgoDox. Due to their case design, you canʼt use standard diodes, you have to use surface mount diodes. They are, of course, included in the ErgoDox kit but, they are tiny, youʼll need to use a tweezers to be able to pick them up and attach them. You also need to be careful while attaching them. I managed to break one of them while I was attaching it. My kit was also short a single diode so I ended up needing to purchase more. Luckily the diodes are fairly easy to find, they are these ones from Digikey. I really wish that MassDrop would have included a few extra diodes in the kit, at the volume that they ordered them at, they are 3¢ a piece. I found the easiest way to attach the diodes was to put down a dot of solder on one side of each diode for an entire row before attaching the diode.

Overall

The ErgoDox is the best keyboard that Iʼve ever used. Its by far the most comfortable keyboard Iʼve ever typed on, I actually enjoy using it everyday. However, you need to be a tinkerer in order to use this keyboard. The assembly is quite tedious and program the keyboard is a little bit involved. Iʼm sure that anyone could make it through the web based configuration but it is another step to complete before you get to experience the ErgoDox. If youʼre a tinkerer too, you should check out the ErgoDox.

Updated ErgoDox

As some of you have noticed, I use an ErgoDox keyboard. Iʼm currently in the process of writing a review of it which, I hope to have completed soon. In the process of writing the review, I discovered that MassDrop has started a drop for a new revision of the design. The new design was not created by Dominic Beauchamp, as the original one was but, is designed by Jacob Alexander and the team at Input Club. Iʼm deeply divided on whether or not I should purchase one.

This revision has fixed my two biggest issues with the Ergodox: the weak connector between the two halves and an awful job of shouldering all of the resistors to the board. One of the TTRS connectors on my current board is flaky due to me putting it in my bag without disconnecting the two halves. When I pulled the ErgoDox out, I grabbed it by the connectors and that has caused the left half of the keyboard to have issues. Basically, it requires me to unplug and then plug the keyboard back in several times per day. I have the replacement parts sitting on my desk but desoldering is quite a pain, so I havenʼt done it yet. This revision of the ErgoDox removes the week point by utilizing a standard USB connection between the two halves. There are a couple of posts on DeskAuthority with people saying that theyʼve replaced their ttrs connectors with USB connectors and it works much better but, I havenʼt yet took that plunge. The another issue I had with the Ergodox was shouldering all of the resistors. It was easily the longest part of the assembly. The resistor was so tiny that it was extremely hard to attach them. This was probably due to my relative lack of experience with soldering (it was the first time that I had assembled something in 4 years) but I found it to be extremely tedious. Iʼm glad to see that that step is gone since the resistors are now integrated into the board.

Theyʼve also added a couple of niceties to the keyboard. First off is that each key can now have an led. They can also be independently controlled. This means that you will be able to setup your Ergodox to be backlit if you desire. They have also added an lcd. Iʼm not really sure what the point of having an lcd on your keyboard is but, no doubt some people will come up with awesome uses for it. Iʼm a little bit concerned that it wonʼt be very visible due to the glare from the acrylic case.

I also have quite a list of concerns over the new design. Much of it boils down to this being a 1.0 product. They have come up with a custom protocol to communicate between the two halves of the keyboard. I really donʼt think that this is a good plan, they could have gone with what the original ErgoDox used and just switch out the connectors for USB. We have no idea how reliable this connection will be and if it turns out to be unreliable, then this new iteration is pretty useless. Iʼm also concerned that neither the PCB design nor the firmware has been open sourced yet. While we have their assurances that they will be open sourced when the keyboard starts shipping but, this would hardly be the first time that a company has promised to open source something and then simply never do it.

I also liked the ErgoDox because it built up a decent community of enthusiasts. This new revision leaves all of that behind. It’s possible that many of the fans will migrate to the new design but, that’s hardly a sure thing. The new design is not compatible in any way with the previous ErgoDox, its more of a spiritual successor rather than an actual one. To add all of the things that they were able to do no doubt necessitated these changes, I question whether those changes are worthwhile or not. In making the assembly easier, I worry that they have lost part of the charm of the original. I think it’s great that this will let people with little to no experience soldering use an Ergodox but, the repairability of this new keyboard is a significant regression from the original ErgoDox. If a component on my ErgoDox fails, I can simply desolder the failed component from the board and replace it with a new one. Sure, that is a pain and you would still have to get a new pcb if the pcb is the part that fails but, I feel like that was an important part of the original ErgoDox. With the new one, if a switch, led or lcd fails, you can swap them out. If anything else fails it will necessitate a whole new pcb. That seems to be a bad trade-off to me.

As I said before, Iʼm deeply divided on whether or not I should pick up one of the new revision. Iʼve wanted to get a second ErgoDox for a while so that I no longer need to transport one between home and the office but, this wouldnʼt be picking up the second one. It would be picking up a whole new keyboard with a similar layout to my ErgoDox. Still, that might be worthwhile. Perhaps this keyboard will gain a larger following than I fear it will. If that’s the case then I really want to get in on the ground floor and figure out what is possible.

Why I use Go

Note Iʼm only writing this because it is my current feelings on my programing language of choice. Iʼm not trying to convince anyone that they should use Go, rather I feel like these things will probably change as I gain more experience in writing software in Go and as I learn other languages and Iʼm interested in recording how my opinion changes over time.

Criticisms of Go

Iʼll start with the things that I donʼt like. Youʼve probably heard many of these before but, I might as well record them. Note that these are in order of pain level.

Dependency management

I really like the way that Go manages libraries. All you need to do is go get <import path> and then import that same import path into your code. Its really simple and I find it to be quite powerful. Of course, there is a relatively large problem here. There’s no way to state what version of that library you are using. This works fine as long as none of your dependencies make breaking changes to their APIs but, once they do, youʼll need to vendor your dependencies or fix the compatibility. There are a number of tools to help with this problem but, all of them are additional tools. There’s no built-in way to handle this, so it’s mostly the wild west out there.

Mutable Values

Go does pass by value meaning that when you pass a variable to a function, that function gets a new variable with the same value as the one that you passed. If you want to mutate the variable in the function, you can pass a pointer to the variable instead. Unfortunately, sometimes the function that you call can still mutate your variable even if you didnʼt pass a pointer to it. In the case of arrays and structs, any modifications made in the function will be present in the original variable.

Lack of Generics

Of course, there is this one, everyone brings up Goʼs lack of generic functions. For the most part, I donʼt miss generics. I find that generics arenʼt needed for most problems. However, I know that there are certain classes of problems that generics could be considered required but, those arenʼt the sort of things that Iʼm doing. Sometimes I do wish for abstractions that could enable a library for.

Benefits

Static Binaries

It seems relatively simple but static binaries make a huge difference. They are so simple to work with, all you need to do is download the binary and run it. There is no need to install a runtime or install libraries. I love it when I run across a go project since, in general, Iʼll be able to just run it. This does have its downsides but I find that the benefit far outweighs any of the downsides. One annoyance that I do have is that Go won’t include your templates in the compiled package. I wish that you could have Go add your templates and other static assets into your binary. Luckily, there is a tool that does this (go-bindata) but, it’s a bit cumbersome to work with.

Simple Cross Compilation

I debated the ordering of this first two quite a bit as I view them to be very closely rated in terms of their importance. With Go, building for a different OS or architecture is as easy as setting GOOS=<target os> and GOARCH=<target architecture> if youʼve installed the cross compilers. For the vast majority of cases, that is all that it takes. It makes it very easy for me to write Go apps on my Mac and then compile them for uses on our Linux based servers. Unfortunately, there are some things that you need to consider. First is that you need to have the cross compiler support. Luckily this is fairly easy with brew as its a simple command line switch. Some of the runtime is built in C and therefore if you want good performance, youʼll need to build your application on the target OS and architecture. Luckily, both of these things are resolved in Go 1.5. Go 1.5 is completely written in Go and so youʼll no longer need to worry about having the C bits properly built for performance. That does bring up the last thing you need to consider and that is cgo. If your Go app depends on a C library, then you wont be able to utilize Goʼs great cross compilers.

Simple concurrency

Goʼs Channels and goroutines combine to make building effective concurrency easy. Goroutines are great. They quite efficient and youʼre able to spawn as many as make sense for the situation. Go takes care of scheduling them fairly. Now that you have things happening concurrently, youʼll need a way to communicate between them. Luckily, Go includes a a great way to do just that, goroutines.

Readability

I find Go to be extremely readable. Go has a very succinct syntax and it will likely be familiar to most programmers. Iʼve found that I can easily make my way through any third party library.

Great standard library

Goʼs standard library is really great. it includes a great many of the things that you might want to do. The standard library is very effective.

Go interfaces

The way that Go handles interfaces is outstanding. You donʼt have to declare what interfaces your type implements instead, you simply implement the interfaceʼs methods and then you can use your type anywhere that accepts that interface. There are many useful interfaces built right into the standard library. Things like io.Reader and io.Writer for reading and writing data. Even for things like sorting collections. Go interfaces tend to be extremely simple which makes implementing them simple as well.

Fast

While speed isnʼt at the top of my list of priorities, Goʼs speed is really nice. For the most part, I just write straightforward code and the performance is excellent. This doesnʼt mean that I donʼt that I donʼt think about performance. I do think about the performance impact of various choices but, I usually choose the simplest solution. I have yet to run into any performance issues.

Closing thoughts

As I said before, Iʼm not trying to convince anyone of anything. Iʼm merely recording my current thoughts so that Iʼll be able to look back on them in the future and see how my opinions have changed. If what Iʼve said resonates with you then, I encourage you to try out Go. I find Go to be a great language to work with and there are a great number of awesome projects being built with Go right now.

Using Multiple Elasticsearch Indices in Logstash

Logstash is a great way to make the wealth of information available in logs available. Specifically logstash, elasticsearch, and kibana combine to make searching and making sense of the data in logs. Due to the ease of collection and the uncertainty of what you may need in the future, it’s likely that you are collecting everything. I know that we are but, this has its drawbacks.

The main one being that there is a limited amount of data that we can store due to the size of the drives attached to the elasticsearch servers. For us, we can only hold the last 3 months of logs. For most uses this is sufficient but, what if there are some logs that need to be retained for longer? Unfortunately, elasticsearch-curator is very coarse-grained, you can only drop whole indices, not the result of queries. Of course, you could always make use of another one of Logstashʼs output options but there is an easy way to handle this situation, by sending important logs to a different index.

While this is relatively easy to do, it does take some configuration. For the sake of simplicity, Iʼm going to assume that elasticsearch is running on the same node as the logstash server. If not, fill in the values that you need.

output {
  if ([program] == "logstash" or [program] == "elasticsearch" or [program] == "nginx") and [environment] == "production" {
    elasticsearch {
      host => "127.0.0.1"
      index => "elk-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      host => "127.0.0.1"
    }
  }
}

So from that, you probably gathered the basic form. In this specific case, Iʼve chosen to send the logs from the ELK stack to the elk index. Probably not that useful but if you change out the program name conditions with something identifying more important logs for your app, this is all you need to get it setup.

There are a couple of issues though. First off, this doesnʼt actually solve the problem that we set out to solve. Sure, all of the logs are going to a new index but, elasticsearch-curator is still going to be removing the logs after the configured size or age. To remedy this, youʼll need to change your curator options.

# Change the settings for the default indices
/usr/local/bin/curator delete --disk-space 110 --prefix logstash
# Change the settings for the new indices
/usr/local/bin/curator delete --disk-space 30 --prefix elk

Now that solves the original problem but, it made a new problem. How exactly can you search both of the indexes? Kibana has the ability built in. At least 3.1.0 does, I havenʼt gotten a chance to use kibana 4 yet. Just go to the settings cog and modify this setting to point to both of the indices.

As you can see from the instructions, all you have to do is add all of the indices in a comma-separated list like this [logstash-]YYYY.MM.DD,[elk-]YYYY.MM.DD. Now youʼll be searching both indices whenever you run a query. As far as I can tell, youʼll need to modify the setting for each dashboard.

Youʼve now fixed the original problem but, its likely that you have data in the old indices that you donʼt want to lose on the old expiration schedule. There is a relatively easy way to migrate the data that you want on the new index. The easiest way to make this happen is to wait for the first of the logs to get written to the new index. Youʼll also need to have elasticdump installed. If you already have node.js and npm installed, all you need to do is run npm install -g elasticdump. Once you have it installed, youʼll need to dump the data that you wish to move. elasticdump supports moving the data from the old index to the new one directly but, I ran into issues doing that. I suggest that you first dump it to a file and then import it. Something along these lines should work:

elasticdump --input=http://127.0.0.1:9200/logstash* --searchBody '{
  "query":{
    "filtered":{
      "query":{
        "query_string":{
          "query":"(program: \"logstash\" OR program: elasticsearch OR program: nginx) AND environment: production"
        }
      }
    }
  }
}' --output=elk.logs

Youʼll need to heavily customize that for what you are trying to move. To figure out the query, try it out in kibana. you can replace the "query" value with the exact query you use in kibana but youʼll need to escape any quotes as Iʼ done above. Once the export has completed, youʼll need to purge the old logs. This does mean that youʼll lose a couple of logs during the transition but, I think saving the history is far more important. To delete all of the logs marching your query, simply run curl -XDELETE "http://127.0.0.1:9200/_all/_query" -d '{}' where {} is the same query you ran to export the logs. This will generate an error message but, you can ignore it. It will delete all of the logs marching that query but, it may take a little while. After a sufficient amount of time for the delete to complete, its time to import all of the exported data. To do that simply run:

elasticdump --input=elk.logs --bulk --bulk-use-output-index-name \
  --output=http://127.0.0.1:9200/elk-2015.03.15

Where elk.logs is file that you exported to in the previous step and elk-2015.03.15 is the full name of the new index. There are a variety of ways to find this but I usually just check the disk, on ubuntu the indices are at /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ (you may need to change the 0 to whatever node you are connected to). Once that completes, youʼll have moved all of the data from the old indices to the new one. in my experience, the import will take considerably less time than the export.

One Year of Ruin

In early February of last year, I purchased this shnazy domain, ruin.io. A few weeks after that, I launched the site. At first, it had no content at all. This was intentional, initially, I wanted to start a completely new site with none of my old content on it. Eventually, I realized that wasnʼt the way to go. I painstakingly copied all of the worthwhile content from my previous sites into this one and then redirect the old sites here. That was definitely the correct decision, Iʼve enjoyed writing on this site greatly and Iʼve really enjoyed watching the traffic grow.

Iʼm very happy to report that all of my top 5 most popular posts where things that I wrote in the last year specifically for ruin.io. Here they are:

  1. godoc with homebrew installed Go
  2. Parsing Nginx logs with logstash
  3. ownCloud in Docker
  4. Using a Mac Mini as a server
  5. 2 Ways Iʼm using Docker

I have very mixed feelings about these. On the one hand, these all have to do with my chosen profession. Iʼm glad that other people find my professional discoveries useful. On the other hand, Iʼm rather disappointed that none of my reviews show up here. I really like writing reviews of products that I enjoy using. I wish that would translate into something that people wish to read but, that’s clearly not the case. Also Iʼm a bit disappointed in both of the Docker posts. While I find Docker to be very interesting, Iʼm not actually using it that much anymore. It requires quite a bit of by in from other people you work with before it starts to show its value.

Iʼve had few notable surges of traffic over the last year. By far the biggest one came from the Docker weekly newsletter. They linked to 2 ways Iʼm using Docker in their weekly newsletter. This was by far my largest spike of traffic. It lasted about 2 and half days. By far the happiest moment for me was when Ben Brooks said something nice about my site:

That was easily the best part of my year for this site. It didn’t provide nearly the same amount of traffic that the Docker newsletter did but, having a writer that you enjoy reading say something nice about your own writing is quite rewarding. Unfortunately, I made lots of mistakes in the linked post which made me feel a little less happy about it.

During this past year, Iʼve changed blogging software nearly as often as Iʼve published a post. That is a fairly large exaggeration but, it feels that way sometimes. I havenʼt yet found a blogging tool that I really like. In the past year Iʼve switched off between Ghost and Jekyll. Neither one is perfect but, theyʼre better than anything else that Iʼve used. Iʼve settled for Jekyll recent mainly due to its flexibility. Iʼve also got a little bit of a unix nerd in me, so the fact that its a translation from one text format to another holds quite a bit of attraction for me. I like that my writing is simply text files which I could write a new tool to generate my site. At the same time, Jekyll is flexible enough that I think its unlikely that Iʼll ever need to do that. Other than slowness. I addressed the major cause of slow build times but, the building of the site will slow down gradually as I add additional content. Perhaps, by the time that build times become a problem, Iʼll find a better blogging tool. Until then, Jekyll fits my needs quite nicely.

Now for the hard part, the finances. Very few people discuss the business side of their sites but, recently, a few have begun to share their financials. I figure that I might as well join them. This may be easy for me as I havenʼt made a dime off of this site. In fact, it costs me a decent chunk of money to keep running. First off is the domain from Hover, an .io domain is $49.99 a year. I swear that it was $80 for the first year but, I could be mistaken. Then I have the ssl certificate. While it is not required, I value your privacy so, I make every use tls when connecting to my site. I purchase my certificates from Gandi.net mainly because I donʼt think you should use startssl. Then there is the hosting, since I like to be able to control everything about my site, I host it on a vps from Linode. I have a 2 cpu 2gb of ram server for $20/month. This is very much overkill for the amount of traffic that I receive but, I like having all of the available cpu. I also use Cloud.typography for great web fonts. Since my site is very small Iʼm on their smallest plan for $99/year. This means that the total cost of my main site is $404.99/year. Now I also run a server for Discourse although it is not being used. This is another $20 a month for the server and $16 for the ssl certificate. This is an additional $256/year. The total to run this site is $660.99/year. While I could make things cheaper in some ways, I like the way things are set up now. In the coming weeks, I will be announcing my plan to recoup some of the expenses associated with running this site.

Iʼm really surprised that you made it this far. I really doubt that anyone actually reads these things but thanks for doing it! Iʼve really enjoyed writing this site and I plan to continue writing here for many years.