Replicating Jepsen Results

If you arenʼt aware, Kyle Kingsbury has a great series of posts testing whether databases live up to their claims. Its an invaluable resource as many of the databases he has tested donʼt live up to their stated goals. That being said, some of the posts are getting quite old at this point so its possible that the developers may have fixed the issues that caused them to fail their stated goals. Luckily, Kyleʼs Jepsen project is open source and youʼre free to try and replicate his results.

This does take some setup though. Youʼll need 5 database servers. It’s easiest to use Debian Jessie for this as that is what Kyle uses and therefore all of the tests that heʼs written work against it. You do need to replace SystemD with SysV init before the tests will be able to run. You also need a machine to run Jepsen on. You shouldnʼt try to reuse one of the database servers for this as the tests will cut off access to some servers at certain points in the tests. For the easiest testing process, youʼll want the database servers to be called n1-n5. They need to all be resolvable by all the other database servers and the server running the tests. The server running the tests also needs to be able to ssh to all of the database servers using the same username and password/ssh key and have sudo access. These hosts must also exist in the known hosts file in the non-hashed format before Jepsen is able to execute a test. Iʼm unsure of what the default values that Jepsen uses for username and password but, youʼre easily able to change the values that it uses for each test. Finally, the server running the tests will need the Java JDK 8 and leiningen to run.

That is quite a bit, isnʼt it? I thought that it was and given the wonderful tooling we have to replicate these sorts of environments, I thought that, for sure, someone had created a way to spin up a set of servers on AWS to run any of the tests that you would like. I wasnʼt able to locate one which likely just means that my search skills were lacking. Since I couldnʼt locate one, I made one using Terraform. jepsen-lab is relatively simple but, it goes through the process of setting up all of the previously stated requirements. It sets up all of the servers and configures them as required and once that process is complete, it outputs the ip address that youʼre able to ssh into. It does leave a number of steps for you to complete on your own: You need to clone the Jepsen repo and youʼll need to modify the test configuration for the username and password. The former is simply because I donʼt know what revision you may wish to use and the latter is because the step is dependent on which tests you chose to run. For more information on how to use jepsen-lab, see the readme in the repository.

After getting everything setup, it’s just a matter of running lein test from the correct directory and verifying the results. You can also make any modifications you like to see if they change the results of the tests. In future installments, Iʼll discuss the particular tests that Iʼve tried to replicate, modifications that Iʼve made and the results that Iʼve gotten.

fpm

For many developers, the way that they deploy is by checking out a specific revision from version control. While some people consider that to be a bit of an anti-pattern, I think that is a fine way to deploy applications written in dynamic languages. In theory, you could do the same thing for compiled languages, it just doesnʼt work well in practice. This would require you to compile your application on every server during the deploy. While this is possible, it’s very inefficient and time-consuming. A much better way to do this is to build your application and then distribute the resultant artifacts. The way that Iʼve chosen to do this is by building native packages, specifically debs.

Generating these debs arenʼt very difficult. It took me quite a bit of research to figure out what needed to be there (debianʼs packaging and Clemens Leeʼs package building HowTo guides were both hugely helpful). once you figure that out, it’s just a matter of creating the correct directory structure and then run it through dpkg-deb. Alright then, how do you make a similar rpm? Time to do some research, huh?

Why should any of this be required? Surely many other people have figured out what is required. One of them must have documented their knowledge somehow. The answer to both of these things is of course. There’s an awesome tool called fpm that creates packages of many different types from many different sources. Of course, it can package up files and directories into debs and rpms.

Iʼve known about fpm for quite some time. In fact, I knew about it before I started building debs by hand. As I mentioned, its not terribly difficult to use dpkg-deb to produce a deb. I also donʼt really like that fpm is written in ruby. While I think ruby is a fine language, getting it installed with everything that is needed to build native gem extensions is a pain. A pain that I didnʼt want to pay for a simple cli tool. It also requires a bit more setup than that to fully utilize. The rpm output requires the rpmbuild command to be installed and Iʼm sure that some of the other outputs require similar commands to be available. Iʼd love to see a similar tool compiled into a static binary but, Iʼve long given up on ever producing this tool for myself.

As I alluded to earlier, what prompted me to start using fpm was generating rpms. Iʼve since realized that I shouldnʼt have dragged my feet on it for so long. Instead of figuring out everything that is required to generate an rpm, I just used fpm: fpm -s dir -t rpm -v $VERSION -d libgmp10 ~/.local/bin=/usr/local/bin/. Of course, I can simply swap out the rpm with deb to generate a deb instead of an rpm. This ignores many of the fancier things that fpm can do. You can easily make native packages for gems, python modules, and cpan modules (to name a few). It also supports some more “exotic” formats such as self-extracting scripts and OS X packages. Iʼve converted many of my deb building scripts to use fpm and Iʼll be using fpm for all of my packaging needs.

Otto

Two weeks ago, at their first ever HashiConf, HashiCorp announced two new tools, Otto and Nomad. Both of these are great new tools but, Iʼm going to be concentrating on the former as Iʼm more interested in it. For the first time, HashiCorp is rethinking one of their current products, Vagrant.

I use Vagrant every day, its immensely useful. Its a great way to set up isolated and consistent development environments. All of that comes with a cost though. Setting up Vagrant is quite a bit of work. You have to figure out how to provision the environment. Vagrant has all of the standard choices built in, so, you can pick your favorite but, that requires you to have some existing knowledge. You could provision your environment using shell scripts but, that quickly gets painful. As this is a fairly large pain point, a variety of tools have sprung up in an attempt to fix this, such as Puphpet and Rove. For a while, I was really excited by this sort of thing. I almost built a community with Nathan Leclaire for hosting vagrant environments after he came up with the idea. Things didnʼt work as it was a really busy time for both of us. After quite a bit of thinking, Iʼm glad that we didnʼt. It just wasnʼt the right way to move things forward.

The other big problem with Vagrant is moving your app into production. You put in a lot of work to build your development environment but, theres a pretty good chance that youʼll need to put in a bunch more work to prepare your production environment. The quick setup that you do for setting up your development environment will not be sufficient for production. In a lot of ways, Vagrant seems to work better if youʼre working back from your production environment. Being able to replicate your exact production environment has lots of benefits but, if you donʼt already an existing set of scripts, roles, or modules then, using Vagrant is going to take a lot of setup to get going.

Thats where Otto comes in. Otto is HashiCorp rethinking how you should setup development environments. It can automatically detect the type of application that youʼre developing and it will build an appropriate development environment automatically. Of course, this leverages Vagrant under the covers, it just skips the laborious setup process. The other big thing that Otto provides is a way to move your application into production.

I think Otto is the answer to a question that Iʼve been pondering for quite some time: how should a small group of developers create and deploy an application? There arenʼt a whole lot of good options for small teams. Recently there have been an explosion of tools for simplifying managing application but, they seem to all be focussed on much larger teams and applications. Things like Mesos, kubernetes and Docker are all great but, they require quite a bit of knowledge to run. For a small team, theyʼre all are too much of a knowledge investment to be useful. Deploying to plain servers also requires too much knowledge to keep running and to secure. The only good option here is Heroku but, that isnʼt without its downsides. Heroku is extremely expensive for what it provides and it ties you to a proprietary platform.

Otto really fills this need. When it comes time to move your app into production, Otto will create industry standard infrastructure for you. This is very important as it allows many hard-earned lessons to automatically be handled. Iʼve felt that things like Chef roles and Puppet modules presented a similar opportunity but, both have fallen very short of that goal. This allows developers to get back to what they do best, improving your application.

As with most of HashiCorpʼs products, Otto is launching with a small set of functionality. The two main limitations are that the app types that Otto supports are rather limited and there is only one type of deployment that is available. Of course, both of these things will improve over time but, theyʼve kept me from being able to start using Otto. These days, Iʼm spending most of my time working in Clojure and Otto doesnʼt currently support jvm applications. In the current version, Otto only supports Docker, Go, PHP, Node, and Ruby but, those cover a large swath of developers. Otto also will only deploy to aws which I donʼt use due to the relatively high cost. I really want to use Otto but, its features are quite enough for me yet.

Otto is an important evolution of a crucial DevOps tool, Vagrant. It makes huge strides forward for many of the current use cases. It removes the biggest pain point of getting Vagrant up and running. It also fills a crucial need by providing a good way to move applications into production. I’m looking forward to using Otto in the future.

Catching errors posting to influxdb from riemann

At Signal Vine we recently upgraded from InfluxDB v0.8.8 to v0.9. Unfortunately, we experienced a number of issues related to the migration. One of them was that InfluxDB experienced a problem that caused it to stop accepting data. Since this happened late on a Friday night, no one noticed the lack of data until Monday. This means that we lost all of the data from the Weekend. Luckily for us, our app is not heavily utilized over the weekend but, I decided that we needed to be able to detect any of these sorts of issues in the future.

It turns out that an exception was being omitted each time Riemann failed to send to InfluxDB. I decided to go with a simple (try ... (catch ...)) which is probably not the ideal way to handle this. There is *exception-stream* but I not sure how it is used and I was unable to find an example demonstrating its use. This is what I came up with:

(ns riemann.config
  (:require [riemann.time :as time]))

(def influx (batch 100 1/10
                   (let [throttled-alert (throttle 1 900 tell-brendan)
                         throttled-log (throttle 1 60 (fn log [e] (warn "influxdb-send-exception" (str e))))]
                     (async-queue! :agg {:queue-size 1000
                                         :core-pool-size 1
                                         :max-pool-size 4
                                         :keep-alive-time 60000}
                                   (let [send-influx (influxdb {:host "influxdb.example.com"
                                                                :version :0.9
                                                                :scheme "http"
                                                                :port "8086"
                                                                :db "metrics"
                                                                :username "riemann"
                                                                :password "password"
                                                                :tag-fields #{:host :environment}})]
                                     (fn influx-sending [event]
                                       (try
                                         (send-influx event)
                                         (catch Exception e
                                           (throttled-alert {:host "riemann.example.com"
                                                             :service "influxdb send error"
                                                             :state "fatal"
                                                             :metric 1
                                                             :tags []
                                                             :ttl 60
                                                             :description (str e)
                                                             :time (time/unix-time-real)})
                                           (throttled-log e)))))))))

Obviously, that is a bit more complex then just catching the errors, so Iʼll dig into it. First off, Iʼm batching events together before sending them to InfluxDB. This helps to reduce the cpu load of Riemann. Then I define the two alert function that will be used later on. tell-brendan is a function that sends an email to me. I only want to get one of these every 15 minutes as it is likely that I would see the alert and start working on the problem immediately. However, I do want to see if sending metrics to Riemann is still failing so, I have Riemann log a failure notification every minute. These are both defined here so that the throttle applies to all of the branches later on.

Next up is another performance option. Iʼve set up an async queue so that sending metrics to InfluxDB doesnʼt block Riemannʼs incoming event stream. Iʼve had sending to InfluxDB cause Riemann to back up to the point where events were expiring before Riemann was processing them. Sending them to InfluxDB asynchronously fixes this. It doesnʼt matter how long it takes events to be sent to InfluxDB, all of Riemannʼs processing only depends on Riemann. Since moving to InfluxDB v0.9 and implementing the async queue, the 99.9% stream latency for Riemann has dropped from 50-100ms to 2.5ms.

Next up, I define the Riemann client. There isnʼt much to see here. The only mildly interesting thing is the :tag-fields value. At Signal Vine, all of our events are tagged with an environment. The #{:host :environment} sends both the host value from the event and the environment as InfluxDB tags. This makes it easier to query exactly what you want from Riemann.

Now for the main attraction, the index-sending function. While Riemann tends to hide this in its internal function, Riemannʼs streams are simply made up of functions that take a single argument, the event. Its just that Riemannʼs built in functions return the function that satisfies this and then calls all of the children that you have defined. Since we donʼt have any children, we simply need to construct a function that takes the event. Well in this case it actually takes a vector of events. As previously mentioned, we use a (try ... (catch ... )) to handle any InfluxDB errors. So we simply try to send the event to Riemann. If that throws any exception, we catch it and pass the exception to our notification functions. Iʼve chosen to construct a new event as the passed in events have little to do with the exception.

Iʼm quite fond of this approach but, it does have its limitations. One of the big ones is that this will generate an alert for even momentary issues in InfluxDB. If you happen to restart InfluxDB, you will get an alert. I donʼt really mind this but it is something to keep in mind. It also discards any events which fail to send. In that same scenario, when we restart InfluxDB, we will lose any events that Riemann tries to send to InfluxDB. It would be much better if we would pause the sending process for some period of time and then attempt to resend the events. the main reason that Iʼm not currently doing this is that Iʼm not really sure how to make that happen.

Elk in Production

Elk is a software stack for processing and searching logs. Elk consists of Elasticsearch, Logstash, and Kibana. Logstash is a log processing tool. It supports a variety of inputs, log processing functions and outputs. One of its outputs is typically Elasticsearch. Elasticsearch is a full-text search engine based on Lucene. Elasticsearch makes your logs easily searchable and Kibana is a web interface for searching and making dashboards out of the data stored in logstash.

Logstash is written in Ruby and targets the jRuby interpreter. In general, it doesnʼt matter other than the configuration file is written in ruby. As previously mentioned, Logstash supports a large number of inputs such as a file, heroku, Kafka and many more. There is probably an input for however, you are currently logging. At Signal Vine, we are currently using the lumberjack, Rabbitmq and syslog inputs.

The easiest way to get started with collecting logs is the syslog input. Many components of your current infrastructure probably already send logs over syslog. It’s easy to configure your chosen syslog daemon to forward logs over tcp. Its probably just as easy to use udp but that isnʼt a good idea as then logs will be dropped even under normal operating conditions. You should be careful to keep the logstash server up and running when collecting syslog messages over tcp. Iʼve seen some strange behavior when the Logstash server is unavailable. Basically, some processes use a large amount of cpu while not doing anything. One downside to using the syslog input is that the timestamps arenʼt very precise, they are only second precision. This is only a problem because Elasticsearch doesnʼt preserve the insert order. What this means is that when you are viewing log messages from within the same second, they likely wonʼt be in the same order that they occurred. Most of the time this isnʼt an issue but it is something to keep in mind.

Lumberjack is a bit harder to explain. Lumberjack is a custom binary format with encryption built in. It was designed for use with lumberjack, which was a Go service that reads from files on the server and ships them to a Logstash server. It has recently been acquired by Elastic and renamed to Logstash-forwarder. While you could run the Logstash service on every server to read from log files and ship them to a central instance to finish processing them, the Logstash service has a decent amount of overhead. It runs on the jvm and as such uses a decent chunk of memory. Logstash-forwarder is very efficient and you shouldnʼt notice its overhead even on the smallest of servers. It even has encryption built in. Unfortunately, this does make it harder to get setup. It took me a few tries to get it right but it has been incredibly stable since then. In fact, since I set it up, I havenʼt had to touch it at all. It simply works. It even has handled resuming after downtime on the logstash server without any sort of intervention.

At Signal Vine, we use Rabbitmq for the internal communication medium between components of our application. Weʼve also chosen to ship our application level logs over Rabbitmq. Logstash has a built in input for processing Rabbitmq messages. This works out really well if you send json to Logstash. Logstash can take the log messages from Rabbitmq and directly insert them into Elasticsearch. This allows you add any custom fields that you want to query on. This then becomes a very effective way to query your own logs. It does help if you have a few standard fields that logs must contain. This allows you to query across all of your application logs using very similar queries.

Once Logstash has received the log messages, it can process them with various types of filters. These filters include: adding geolocation data to web server logs, parsing the a date field included in the log message, the ability to process custom log formats and even run ruby code on the message. These filters are useful for coercing whatever sort of log message you get into a queryable format for later consumption. The available functions make this process very easy. Given that the configuration file is made in ruby, youʼre able to selectively apply these filters based on any scheme that you desire. It might be useful to apply one filter to all of the logs that come from a specific input. You can also target these filters based on the contents of the log message.

The notable exception to the processing ease is grok. Grok allows you to process custom log formats using a regex which, can be a difficult process. Luckily Logstash ships with a number of complete patterns and partial patterns that you can reuse for your logging format. There is also a great tool for building your grok filter based on an example of a message. It’s important to add a unique tag if the parsing fails and you should leave the default _grokparsefailure as the combination of these two tags allows you to easily query log messages that failed to parse and then locate the specific filter that failed.

Once the messages have been fully processed, Logstash can send these messages to a variety of systems for further processing or storage. You can send the logs to s3 for permanent storage, Send an email with a particularly troubling log message, forward the data to another logstash instance or store them in Elasticsearch. I think Elasticsearch is nearly universally used. It is a very effective way to query logging data. At Signal Vine, we also use the Riemann output to process logs that we may want to be alerted about. I find that this process works really well as Iʼm able to use all of Riemannʼs powerful stream processing tools to filter the notifications that the engineering team receives. I would like to forward the full log message to Riemann but there happens to be a bug with a pending resolution preventing this.

After all of the information about Logstash, the information about Elasticsearch is rather dull. It serves as an easily queryable data store. It supports full-text search and it includes Luceneʼs powerful querying language to make short work of finding relevant log messages. Elasticsearch allows you to scale horizontally based on the number of shards that you choose when configuring it. This is very useful for if the amount of logs that you want to keep exceeds the capacity of a single server or if your write volume is greater than a single server can handle. Elasticsearch also allows you to specify the number of replicas to keep which will allow you to lose some servers without losing any of your data. Just keep in mind that Elasticsearch has some catastrophic failure modes. Even with the ability to shard the data across many servers, most likely it wonʼt be long before you run out of storage for your logs. That is where curator comes in. Curator allows you to specify how much logging data to keep. You can either choose how long to keep logging data or the total amount of disk space to use.

Once your logs are stored in Elasticsearch, you need a way to easily query them. This is where the final piece of the Elk stack comes in, Kibana. Iʼm currently using Kibana 3 as I havenʼt yet put in the time to understand Kibana 4. Iʼve tried it out but, I wasnʼt a fan of it as it seems very unfamiliar. I found the changes to be dizzying. Iʼm sure that there are quite a number of benefits but, as I havenʼt yet researched them, I donʼt know what they are. With that being said, Kibana is a great way to look at your log data. It exposes all of the powerful querying functionality of Elasticsearch for doing ad-hoc querying of your logs as well as creating great dashboards from the data contained in your logs. When you look through the documentation, there are several great examples of this.

The Elk stack is not without its flaws but, it gives you a powerful set of tools for processing log data. Having centralized storage for logs is an invaluable tool for any IT team to have. Having a single location where you can research issues happening on your infrastructure can greatly speed up the time between detection and the fix. It might even help you find errors that you may not have noticed without it. The Elk stack is the best tooling in this area that Iʼve seen and you should think about using it as well.

Announcing terraform-provider-linode

Iʼve been interested in HashiCorpʼs terraform ever since it was announced. I loved the idea of being able to tie multiple providers together. You could use Route 53 for DNS and Digital Ocean for your servers or you could use DNS Simple with ec2. I havenʼt been able to try it because I use Linode for both my personal servers and at work. Unfortunately, Linode isnʼt one of the included plugins in Terraform and no one had created a plugin for it. So I did what any developer with a bit of free time and a possibly unhealthy obsession with automation tools and HashiCorp products: I built one.

In its current state, itʼs fairly light on features. It only supports managing Linodes (what Linode calls their servers) and not any of the other things that Linode provides. Iʼm hoping to get Linodeʼs DNS and NodeBalancer services added in the future. terraform-provider-linode also doesn’t support all of the options that Linode provides. In particular, I intentionally omitted all of the various alerting settings that Linode provides. While adding all of them would have been easy, the sheer number of options seemed too complex for the initial release. The currently implemented features are exactly the features that I needed in order to start using terraform. If you would like to have a particular feature, it would be great if you were able to contribute it to the project. If not, please create an issue on GitHub and Iʼll see when I can get it implemented.

Terraformʼs helper/schema made developing this plugin a breeze. I expected that it would take considerably longer and be more difficult to build terraform-provider-linode. As they mention on the provider plugin page, its really just a matter of setting up CRUD functions for a particular resource. That is all that there is to it. Terraform also includes some extremely handy functions for building acceptance tests. Terraform feels incredibly polished, I find this particularly remarkable as Terraform was only released a little under a year ago. I also found Toahʼs Linode api client library to be extremely helpful for developing this plugin. Linode, unfortunately, hasnʼt created an official Go api client but, Toah really stepped up to make one.

Please check out terraform-provider-linode if it is something that would be useful to you. If youʼd like to contribute, then I would really appriciate your help.

Tools in Production

About a year ago, I started a new job with SignalVine as the person in charge of Operations.While I strive to give the engineering team the tools that they need to properly manage our production systems, its my responsibility to keep all of the components of our application up and running. In the last year, Iʼve used quite a few different tools to make running our application a pleasant experience. Iʼll be covering a wide variaty of things, from databases to DevOps tools. Some of these things have worked well for me, others have not.

  • Ansible – My Configuration management tool of choice.

Writing a Riemann metric to a file

At work, I recently setup Riemann for monitoring. Riemann is a monitoring tool that works on streams of events. It includes many powerful tools to work on streams of events. As an example, it has a ddt function that will differentiate the eventʼs metrics over time. This allows you get to a rate of change for a counter. While Riemann includes many powerful tools, your Riemann config file is a Clojure program so youʼre able to extend Riemann by simply modifying your config file.

I had such an occasion to do such a thing this week. We send all of our error level log messages from Logstash to Riemann in order to alert us when we need to check the logs. Doing this is a fairly simple process, we use slack and there is a built-in function to send alerts to Slack. While we could send the whole log message to Slack, this isnʼt ideal for us. Our log messages can be quite long, many thousands of characters and sending that to Slack makes for a fairly bad experience. What we decided to do instead was write the full metric to a file and link to that file in the Slack message. Unfortunately, there isnʼt really a built-in way to do this in Riemann. You could write the messages to Riemannʼs log file but that isnʼt what we are looking for here as that results in a single large log file rather than individual log files.

What I decided to do was create a function that would write out the message to a file with the name set to the messages sha-256 hash. Generating the has was the most complicated part of this. The complication arose from my lack of knowledge on the various libraries that can generate a hash. The way that I figured this out was by Googling variations on Clojure/Java sha-256 hashes and then trying them at the Clojure REPL on a checkout of the Riemann source. Unfortunately, neither of the Clojure hashing libraries are included in Riemann but, I was able to find a java package that Riemann includes that is able to generate hashes, Apache Commons. I likely would have known that if I had more experience with the Java ecosystem but I donʼt. So here is what I came up with.

(ns riemann.config
  (:require [clojure.java.io :as io]))
  (:import (org.apache.commons.codec.binary.Base64)
           (java.security.MessageDigest)))

; A couple of helper functions to shuffle data between strings and byte arrays
(defn base64-encode [^String v] (Base64/encodeBase64 (.getBytes v)))
(defn base64-decode [^bytes b] (Base64/decodeBase64 b))

; The hashing function
(defn digest [^bytes b]
  (.digest (doto (MessageDigest/getInstance "SHA-256")
             (.reset)
             (.update b))))

(defn as-hex [v]
  (-> (Integer/toString (+ 0x100 (bit-and 0xff v)) 16) (.substring 1)))

; Gets the sha-256 of a passed in string
(defn get-sha256 [v]
  (apply str (map as-hex (-> (base64-encode v) (base64-decode) (digest)))))

; Returns a function that writes an event to the specified directory as <sha-256>.txt and
; modifies the event to include {:url <url>/<sha-256>.txt}
(defn write [directory url]
  (fn [& children]
    (fn stream [event]
      (let [write-event (fn [individual-event]
                          (let [contents (reduce (fn [r x] (str r "\n" (x 0) ": " (x 1)))
                                                 (conj (seq individual-event) "Message Contents"))]
                            (let [sha (get-sha256 contents)]
                              (with-open [wrtr (io/writer (str directory "/" sha ".txt"))]
                                (.write wrtr contents)
                                (conj individual-event {:url (str url "/" sha ".txt")})))))]
        (if (vector? event)
          (call-rescue (into [] (map write-event event)) children)
          (call-rescue (write-event event) children))))))

Then all you need to do is define the function that you will use on your steams. Something like (def write-log (write "/var/www/alerts" "https://alerts.example.com")) would work where /var/www/alerts is the content directory for alerts.example.com. To include the link in your Slack alert, youʼll need to provide a custom formatter that includes a link. Here is what we use:

; Truncates the String (s) at the specified number of characters (n)
(defn trunc [s n] (subs s 0 (min (count s) n)))

; The formatter that slack will use when there isn't a link
(defn default-formatter [e] {:text (str "*Host*: " (:host e) "\n*Service*: " (:service e) "\n*Metric*: " (:metric e) "\n*Description*: " (trunc (:description e) 100))})
; The formatter that slack will use when there is a link
(defn url-formatter [e] {:text (str "*Host*: " (:host e) "\n*Service*: " (:service e ) "\n*Metric*: " (:metric e) "\n*Link*: " (:url e) "\n*Description*: " (trunc (:description e) 100))})

; Choses which formatter slack will use
(defn slack-formatter [e]
  (if (nil? (:url e))
    (default-formatter e)
    (url-formatter e)))

I know that’s a lot to piece together, so here is a minimal Riemann config that should work, to show you how to use everything.

(ns riemann.config
  (:require [clojure.java.io :as io]))
  (:import (org.apache.commons.codec.binary.Base64)
           (java.security.MessageDigest)))

; A couple of helper functions to shuffle data between strings and byte arrays
(defn base64-encode [^String v] (Base64/encodeBase64 (.getBytes v)))
(defn base64-decode [^bytes b] (Base64/decodeBase64 b))

; The hashing function
(defn digest [^bytes b]
  (.digest (doto (MessageDigest/getInstance "SHA-256")
             (.reset)
             (.update b))))

(defn as-hex [v]
  (-> (Integer/toString (+ 0x100 (bit-and 0xff v)) 16) (.substring 1)))

; Gets the sha-256 of a passed in string
(defn get-sha256 [v]
  (apply str (map as-hex (-> (base64-encode v) (base64-decode) (digest)))))

; Returns a function that writes an event to the specified directory as <sha-256>.txt and modifies the event to include {:url <url>/<sha-256>.txt}
(defn write [directory url]
  (fn [& children]
    (fn stream [event]
      (let [write-event (fn [individual-event]
                          (let [contents (reduce (fn [r x] (str r "\n" (x 0) ": " (x 1)))
                                                 (conj (seq individual-event) "Message Contents"))]
                            (let [sha (get-sha256 contents)]
                              (with-open [wrtr (io/writer (str directory "/" sha ".txt"))]
                                (.write wrtr contents)
                                (conj individual-event {:url (str url "/" sha ".txt")})))))]
        (if (vector? event)
          (call-rescue (into [] (map write-event event)) children)
          (call-rescue (write-event event) children))))))

;The function that you will call to write the event
(def write-alert (write "/var/www/alerts" "http://alerts.example.com"))

; Truncates the String (s) at the specified number of characters (n)
(defn trunc [s n] (subs s 0 (min (count s) n)))

; The formatter that slack will use when there isn't a link
(defn default-formatter [e] {:text (str "*Host*: " (:host e) "\n*Service*: " (:service e) "\n*Metric*: " (:metric e) "\n*Description*: " (trunc (:description e) 100))})
; The formatter that slack will use when there is a link
(defn url-formatter [e] {:text (str "*Host*: " (:host e) "\n*Service*: " (:service e ) "\n*Metric*: " (:metric e) "\n*Link*: " (:url e) "\n*Description*: " (trunc (:description e) 100))})

; Chooses which formatter slack will use
(defn slack-formatter [e]
  (if (nil? (:url e))
    (default-formatter e)
    (url-formatter e)))

(def notify-slack (slack {:account "{% raw %}{{Your Account}}{% endraw %}"
                          :token "{% raw %}{{Your Token}}{% endraw %}"}
                         {:username "Riemann"
                          :channel "#Alerts"
                          :icon ":rotating_light:"
                          :formatter slack-formatter}))

; Expire old events from the index every 5 seconds.
(periodically-expire 5)

(let [index (index)]
  ; Inbound events will be passed to these streams:
  (streams
    index

  (default {:severity 4}
    (where (and (service #".* logs$")(= (:environment event) "production"))
      (by [:service]
        (where (< (:severity event) 4)
          (throttle 2 3600
            (write-log notify-slack))))))))

What that config will do is send alerts to the Alerts channel of your Slack when any events are placed into Riemann that end with logs. They are limited to no more than 2 messages per hour per service.

2 ways I’m using Docker

I recently got to migrate my server from running Ubuntu 12.04 to Ubuntu 14.04. As I was unable to upgrade the kernel while on 12.04 (due to some hardware issues), I was stuck running into the bug mentioned on the installation page. It wasn’t too big of an issue but, basically, sometimes containers would just hang for no real reason. In this state, I couldn’t stop them or delete them. The only way to get rid of them was to either restart docker or restart the server.

Anyways, I’m happy that is no longer an issue for me. So I’ve been looking into moving as much as I can into Docker. I basically have two ways that I’m using it. The first of which is the one that I mention in Docker Appliances, I’m running my Discourse server in Docker using the excellent discourse_docker project. It works extremely well, especially now that it doesn’t hang randomly.

The other way I’m using it is as the execution environment for a couple of other things. What I mean by this is that both the code for the site and the database files are mounted from the host. Here is an example, this site runs in a Docker container. On my server, its code and database reside at /var/node/ruin. Since I last wrote about Ghost, I have switched over to using SQLite as my database. SQLite performance is adequate for my needs and at the time, I hadn’t figured out how to get MySQL up and running in a Docker container. So the docker container simply mounts the code/content directory and runs only the node process. I have it set to expose the default port that Ghost runs at, 2368, to the docker host. On the docker host, I run nginx and reverse proxy the traffic to port 2368. When I want to update the code, I have an ansible playbook that pulls the fresh code down, stops the current container and launches a new one.

I also have a similar set up for my unmark instance, unmark.tobolaski.com (I wouldn’t recommend going there as it uses a custom certificate authority). It mounts the code from /var/unmark/app and the mysql data on /var/unmark/mysql into a lamp container. In this case, I have nginx setup to reverse proxy to 8000 which is mapped to 80 on the Docker container.

If I was going to be putting either of these containers on multiple hosts, I’d move the database into something else, possibly a new container or maybe their own host. Then I add in the application code into the container and distribute it as a single unit, which is more like the intended usage than what I’m currently doing. Basically, I’m using Docker as lightweight virtualization, which it does extremely well.

I’ve open sourced both of the dockerfiles that I discussed here.

Using Multiple Elasticsearch Indices in Logstash

Logstash is a great way to make the wealth of information available in logs available. Specifically logstash, elasticsearch, and kibana combine to make searching and making sense of the data in logs. Due to the ease of collection and the uncertainty of what you may need in the future, it’s likely that you are collecting everything. I know that we are but, this has its drawbacks.

The main one being that there is a limited amount of data that we can store due to the size of the drives attached to the elasticsearch servers. For us, we can only hold the last 3 months of logs. For most uses this is sufficient but, what if there are some logs that need to be retained for longer? Unfortunately, elasticsearch-curator is very coarse-grained, you can only drop whole indices, not the result of queries. Of course, you could always make use of another one of Logstashʼs output options but there is an easy way to handle this situation, by sending important logs to a different index.

While this is relatively easy to do, it does take some configuration. For the sake of simplicity, Iʼm going to assume that elasticsearch is running on the same node as the logstash server. If not, fill in the values that you need.

output {
  if ([program] == "logstash" or [program] == "elasticsearch" or [program] == "nginx") and [environment] == "production" {
    elasticsearch {
      host => "127.0.0.1"
      index => "elk-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      host => "127.0.0.1"
    }
  }
}

So from that, you probably gathered the basic form. In this specific case, Iʼve chosen to send the logs from the ELK stack to the elk index. Probably not that useful but if you change out the program name conditions with something identifying more important logs for your app, this is all you need to get it setup.

There are a couple of issues though. First off, this doesnʼt actually solve the problem that we set out to solve. Sure, all of the logs are going to a new index but, elasticsearch-curator is still going to be removing the logs after the configured size or age. To remedy this, youʼll need to change your curator options.

# Change the settings for the default indices
/usr/local/bin/curator delete --disk-space 110 --prefix logstash
# Change the settings for the new indices
/usr/local/bin/curator delete --disk-space 30 --prefix elk

Now that solves the original problem but, it made a new problem. How exactly can you search both of the indexes? Kibana has the ability built in. At least 3.1.0 does, I havenʼt gotten a chance to use kibana 4 yet. Just go to the settings cog and modify this setting to point to both of the indices.

As you can see from the instructions, all you have to do is add all of the indices in a comma-separated list like this [logstash-]YYYY.MM.DD,[elk-]YYYY.MM.DD. Now youʼll be searching both indices whenever you run a query. As far as I can tell, youʼll need to modify the setting for each dashboard.

Youʼve now fixed the original problem but, its likely that you have data in the old indices that you donʼt want to lose on the old expiration schedule. There is a relatively easy way to migrate the data that you want on the new index. The easiest way to make this happen is to wait for the first of the logs to get written to the new index. Youʼll also need to have elasticdump installed. If you already have node.js and npm installed, all you need to do is run npm install -g elasticdump. Once you have it installed, youʼll need to dump the data that you wish to move. elasticdump supports moving the data from the old index to the new one directly but, I ran into issues doing that. I suggest that you first dump it to a file and then import it. Something along these lines should work:

elasticdump --input=http://127.0.0.1:9200/logstash* --searchBody '{
  "query":{
    "filtered":{
      "query":{
        "query_string":{
          "query":"(program: \"logstash\" OR program: elasticsearch OR program: nginx) AND environment: production"
        }
      }
    }
  }
}' --output=elk.logs

Youʼll need to heavily customize that for what you are trying to move. To figure out the query, try it out in kibana. you can replace the "query" value with the exact query you use in kibana but youʼll need to escape any quotes as Iʼ done above. Once the export has completed, youʼll need to purge the old logs. This does mean that youʼll lose a couple of logs during the transition but, I think saving the history is far more important. To delete all of the logs marching your query, simply run curl -XDELETE "http://127.0.0.1:9200/_all/_query" -d '{}' where {} is the same query you ran to export the logs. This will generate an error message but, you can ignore it. It will delete all of the logs marching that query but, it may take a little while. After a sufficient amount of time for the delete to complete, its time to import all of the exported data. To do that simply run:

elasticdump --input=elk.logs --bulk --bulk-use-output-index-name \
  --output=http://127.0.0.1:9200/elk-2015.03.15

Where elk.logs is file that you exported to in the previous step and elk-2015.03.15 is the full name of the new index. There are a variety of ways to find this but I usually just check the disk, on ubuntu the indices are at /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ (you may need to change the 0 to whatever node you are connected to). Once that completes, youʼll have moved all of the data from the old indices to the new one. in my experience, the import will take considerably less time than the export.