Appcanary makes sure you never run vulnerable software on your servers.
Subscribe to our newsletter!

New Year, New Appcanary Features

We’ve been hard at work the past few months on lots of features touching every aspect of our product, and to ring in the new year, we’re going to announce them all at once.

Search our vulnerabilities

You can now browse and search every vulnerability Appcanary knows about! It’s pretty snazzy:

browse our vulnerabilities

Automatically upgrade packages

We’ve had this feature for Ubuntu, and now we’re adding it for CentOS.

If you have the appcanary agent installed, you can run appcanary upgrade, and we’ll automatically upgrade all of your vulnerable packages to the lowest version that fixes all the vulnerabilities we know about.

Resolve vulnerabilities

There’s now a “marked resolved” button that lets us know that you don’t want to be notified about a vulnerability. This is used if a vulnerability doesn’t affect you, or if you are accepting the risk based on some other mitigation’s (i.e. you’re not using the vulnerable feature of the package, the port in question is blocked by a firewall, etc). We give you the opportunity to record your reasoning and provide a full audit trail of every vulnerability you mark as resolved:

audit log

Brand new dashboard

We just pushed a brand new UX for our dashboard. You can sort and search and sort all of your servers and monitors. Check it out.

new dashboard

The Appcanary rubygem

We released the Appcanary gem. This gives us tighter integration with ruby projects, you can either check your ruby project for vulnerabilities as a one-time check, or set up a monitor with notifications. You can see the source here.

Our gem is still very early, so we very much want your feedback. Please let us know what you think at [email protected]

CentOS 6 support

Last but not lease, we fully support CentOS 6 along with CentOS 7.

Everything you need to know about HTTP security headers

Some physicists 28 years ago needed a way to easily share experimental data and thus the web was born. This was generally considered to be a good move. Unfortunately, everything physicists touch — from trigonometry to the strong nuclear force — eventually becomes weaponized and so too has the Hypertext Transfer Protocol.

What can be attacked must be defended, and since tradition requires all security features to be a bolted-on afterthought, things… got a little complicated.

This article explains what secure headers are and how to implement these headers in Rails, Django, Express.js, Go, Nginx, and Apache.

Please note that some headers may be best configured in on your HTTP servers, while others should be set on the application layer. Use your own discretion here. You can test how well you’re doing with Mozilla’s Observatory.

Did we get anything wrong? Contact us at [email protected].

HTTP Security Headers


X-XSS-Protection: 0;
X-XSS-Protection: 1;
X-XSS-Protection: 1; mode=block


Cross Site Scripting, commonly abbreviated XSS, is an attack where the attacker causes a page to load some malicious javascript. X-XSS-Protection is a feature in Chrome and Internet Explorer that is designed to protect against “reflected” XSS attacks — where an attacker is sending the malicious payload as part of the request1.

X-XSS-Protection: 0 turns it off.
X-XSS-Protection: 1 will filter out scripts that came from the request - but will still render the page
X-XSS-Protection: 1; mode=block when triggered, will block the whole page from being rendered.

Should I use it?

Yes. Set X-XSS-Protection: 1; mode=block. The “filter bad scripts” mechanism is problematic; see here for why.


Platform What do I do?
Rails 4 and 5 On by default
Express.js Use helmet
Go Use unrolled/secure
Nginx add_header X-XSS-Protection "1; mode=block";
Apache Header always set X-XSS-Protection "1; mode=block"

I want to know more

X-XSS-Protection - MDN

Content Security Policy

Content-Security-Policy: <policy>


Content Security Policy can be thought of as much more advanced version of the X-XSS-Protection header above. While X-XSS-Protection will block scripts that come from the request, it’s not going to stop an XSS attack that involves storing a malicious script on your server or loading an external resource with a malicious script in it.

CSP gives you a language to define where the browser can load resources from. You can white list origins for scripts, images, fonts, stylesheets, etc in a very granular manner. You can also compare any loaded content against a hash or signature.

Should I use it?

Yes. It won’t prevent all XSS attacks, but it’s a significant mitigation against their impact, and an important aspect of defense-in-depth. That said, it can be hard to implement. If you’re an intrepid reader and went ahead and checked the headers returns2, you’ll see that we don’t have CSP implemented yet. There are some rails development plugins we’re using that are holding us back from a CSP implementation that will have an actually security impact. We’re working on it, and will write about it in the next instalment!


Writing a CSP policy can be challenging. See here for a list of all the directives you can employ. A good place to start is here.

Platform What do I do?
Rails 4 and 5 Use secureheaders
Django Use django-csp
Express.js Use helmet/csp
Go Use unrolled/secure
Nginx add_header Content-Security-Policy "<policy>";
Apache Header always set Content-Security-Policy "<policy>"

I want to know more

HTTP Strict Transport Security (HSTS)

Strict-Transport-Security: max-age=<expire-time>
Strict-Transport-Security: max-age=<expire-time>; includeSubDomains
Strict-Transport-Security: max-age=<expire-time>; preload


When we want to securely communicate with someone, we face two problems. The first problem is privacy; we want to make sure the messages we send can only be read by the recipient, and no one else. The other problem is that of authentication: how do we know the recipient is who they say they are?

HTTPS solves the first problem with encryption, though it has some major issues with authentication (more on this later, see Public Key Pinning). The HSTS header solves the meta-problem: how do you know if the person you’re talking to actually supports encryption?

HSTS mitigates an attack called sslstrip. Suppose you’re using a hostile network, where a malicious attacker controls the wifi router. The attacker can disable encryption between you and the websites you’re browsing. Even if the site you’re accessing is only available over HTTPS, the attacker can man-in-the-middle the HTTP traffic and make it look like the site works over unencrypted HTTP. No need for SSL certs, just disable the encryption.

Enter the HSTS. The Strict-Transport-Security header solves this by letting your browser know that it must always use encryption with your site. As long as your browser has seen an HSTS header — and it hasn’t expired — it will not access the site unencrypted, and will error out if it’s not available over HTTPS.

Should I use it?

Yes. Your app is only available over HTTPS, right? Trying to browse over regular old HTTP will redirect to the secure site, right? (Hint: Use letsencrypt if you want to avoid the racket that are commercial certificate authorities.)

The one downside of the HSTS header is that it allows for a clever technique to create supercookies that can fingerprint your users. As a website operator, you probably already track your users somewhat - but try to only use HSTS for good and not for supercookies.


The two options are

  • includeSubDomains - HSTS applies to subdomains
  • preload - Google maintains a service that hardcodes3 your site as being HTTPS only into browsers. This way, a user doesn’t even have to visit your site: their browser already knows it should reject unencrypted connections. Getting off that list is hard, by the way, so only turn it on if you know you can support HTTPS forever on all your subdomains.
Platform What do I do?
Rails 4 config.force_ssl = true
Does not include subdomains by default. To set it:
config.ssl_options = { hsts: { subdomains: true } }
Rails 5 config.force_ssl = true
Django SECURE_HSTS_SECONDS = 31536000
Express.js Use helmet
Go Use unrolled/secure
Nginx add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; ";
Apache Header always set Strict-Transport-Security "max-age=31536000; includeSubdomains;

I want to know more

HTTP Public Key Pinning (HPKP)

Public-Key-Pins: pin-sha256=<base64==>; max-age=<expireTime>;
Public-Key-Pins: pin-sha256=<base64==>; max-age=<expireTime>; includeSubDomains
Public-Key-Pins: pin-sha256=<base64==>; max-age=<expireTime>; report-uri=<reportURI>


The HSTS header described above was designed to ensure that all connections to your website are encrypted. However, nowhere does it specify what key to use!

Trust on the web is based on the certificate authority (CA) model. Your browser and operating system ship with the public keys of some trusted certificate authorities which are usually specialized companies and/or nation states. When a CA issues you a certificate for a given domain that means anyone who trusts that CA will automatically trust the SSL traffic you encrypt using that certificate. The CAs are responsible for verifying that you actually own a domain (this can be anything from sending an email, to asking you to host a file, to investigating your company).

Two CAs can issue a certificate for the same domain to two different people, and browsers will trust both. This creates a problem, especially since CAs can be and are compromised. This allows attackers to MiTM any domain they want, even if that domain uses SSL & HSTS!

The HPKP header tries to mitigate this. This header lets you to “pin” a certificate. When a browser sees the header for the first time, it will save the certificate. For every request up to max-age, the browser will fail unless at least one certificate in the chain sent from the server has a fingerprint that was pinned.

This means that you can pin to the CA or a intermediate certificate along with the leaf in order to not shoot yourself in the foot (more on this later).

Much like HSTS above, the HPKP header also has some privacy implications. These were laid out in the RFC itself.

Should I use it?

Probably not.

HPKP is a very very sharp knife. Consider this: if you pin to the wrong certificate, or you lose your keys, or something else goes wrong, you’ve locked your users out of your site. All you can do is wait for the pin to expire.

This article lays out the case against it, and includes a fun way for attackers to use HPKP to hold their victims ransom.

One alternative is using the Public-Key-Pins-Report-Only header, which will just report that something went wrong, but not lock anyone out. This allows you to at least know your users are being MiTMed with fake certificates.


The two options are

  • includeSubDomains - HPKP applies to subdomains
  • report-uri - Inavlid attempts will be reported here

You have to generate a base64 encoded fingerprint for the key you pin to, and you have to use a backup key. Check this guide for how to do it.

Platform What do I do?
Rails 4 and 5 Use secureheaders
Django Write custom middleware
Express.js Use helmet
Go Use unrolled/secure
Nginx add_header Public-Key-Pins 'pin-sha256="<primary>"; pin-sha256="<backup>"; max-age=5184000; includeSubDomains';
Apache Header always set Public-Key-Pins 'pin-sha256="<primary>"; pin-sha256="<backup>"; max-age=5184000; includeSubDomains';

I want to know more


X-Frame-Options: DENY
X-Frame-Options: SAMEORIGIN
X-Frame-Options: ALLOW-FROM


Before we started giving dumb names to vulnerabilities, we used to give dumb names to hacking techniques. “Clickjacking” is one of those dumb names.

The idea goes like this: you create an invisible iframe, place it in focus and route user input into it. As an attacker, you can then trick people into playing a browser-based game while their clicks are being registered by a hidden iframe displaying twitter - forcing them to non-consensually retweet all of your tweets.

It sounds dumb, but it’s an effective attack.

Should I use it?

Yes. Your app is a beautiful snowflake. Do you really want some genius shoving it into an iframe so they can vandalize it?


X-Frame-Options has three modes, which are pretty self explanatory.

  • DENY - No one can put this page in an iframe
  • SAMEORIGIN - The page can only be displayed in an iframe by someone on the same origin.
  • ALLOW-FROM - Specify a specific url that can put the page in an iframe

One thing to remember is that you can stack iframes as deep as you want, and in that case, the behavior of SAMEORIGIN and ALLOW-FROM isn’t specified. That is, if we have a triple-decker iframe sandwich and the innermost iframe has SAMEORIGIN, do we care about the origin of the iframe around it, or the topmost one on the page? ¯\_(ツ)_/¯.

Platform What do I do?
Rails 4 and 5 SAMEORIGIN is set by default.

To set DENY:
config.action_dispatch.default_headers['X-Frame-Options'] = "DENY"
Django MIDDLEWARE = [ ... 'django.middleware.clickjacking.XFrameOptionsMiddleware', ... ]
This defaults to SAMORIGIN.

Express.js Use helmet
Go Use unrolled/secure
Nginx add_header X-Frame-Options "deny";
Apache Header always set X-Frame-Options "deny"

I want to know more


X-Content-Type-Options: nosniff;


The problem this header solves is called “MIME sniffing”, which is actually a browser “feature”.

In theory, every time your server responds to a request it is supposed to set a Content-Type header in order to tell the browser if it’s getting some HTML, a cat gif, or a Flash cartoon from 2008. Unfortunately, the web has always been broken and has never really followed a spec for anything; back in the day lots of people didn’t bother to set the content type header properly.

As a result, browser vendors decided they should be really helpful and try to infer the content type by inspecting the content itself while completely ignore the content type header. If it looks like a gif, display a gif!, even though the content type is text/html. Likewise, if it looks like we got some HTML, we should render it as such even if the server said it’s a gif.

This is great, except when you’re running a photo-sharing site, and users can upload photos that look like HTML with javascript included, and suddenly you have a stored XSS attack on your hand.

The X-Content-Type-Options headers exist to tell the browser to shut up and set the damn content type to what I tell you, thank you.

Should I use it?

Yes, just make sure to set your content types correctly.


Platform What do I do?
Rails 4 and 5 On by default
Express.js Use helmet
Go Use unrolled/secure
Nginx add_header X-Content-Type-Options nosniff;
Apache Header always set X-Content-Type-Options nosniff


Referrer-Policy: "no-referrer" 
Referrer-Policy: "no-referrer-when-downgrade" 
Referrer-Policy: "origin" 
Referrer-Policy: "origin-when-cross-origin"
Referrer-Policy: "same-origin" 
Referrer-Policy: "strict-origin" 
Referrer-Policy: "strict-origin-when-cross-origin" 
Referrer-Policy: "unsafe-url"


Ah, the Referer header. Great for analytics, bad for your users’ privacy. At some point the web got woke and decided that maybe it wasn’t a good idea to send it all the time. And while we’re at it, let’s spell “Referrer” correctly4.

The Referrer-Policy header allows you to specify when the browser will set a Referer header.

Should I use it?

It’s up to you, but it’s probably a good idea. If you don’t care about your users’ privacy, think of it as a way to keep your sweet sweet analytics to yourself and out of your competitors’ grubby hands.

Set Referrer-Policy: "no-referrer"


Platform What do I do?
Rails 4 and 5 Use secureheaders
Django Write custom middleware
Express.js Use helmet
Go Write custom middleware
Nginx add_header Referrer-Policy "no-referrer";
Apache Header always set Referrer-Policy "no-referrer"

I want to know more

Set-Cookie: <key>=<value>; Expires=<expiryDate>; Secure; HttpOnly; SameSite=strict


This isn’t a security header per se, but there are three different options for cookies that you should be aware of.

  • Cookies marked as Secure will only be served over HTTPS. This prevents someone from reading the cookies in a MiTM attack where they can force the browser to visit a given page.

  • HttpOnly is a misnomer, and has nothing to do with HTTPS (unlike Secure above). Cookies marked as HttpOnly can not be accessed from within javascript. So if there is an XSS flaw, the attacker can’t immediately steal the cookies.

  • SameSite helps defend against Cross-Origin Resource Sharing (CSRF) attacks. This is an attack where a different website the user may be visiting inadvertently tricks them into making a request against your site, i.e. by including an image to make a GET request, or using javascript to submit a form for a POST request. Generally, people defend against this using CSRF tokens. A cookie marked as SameSite won’t be sent to a different site.

It has two modes, lax and strict. Lax mode allows the cookie to be sent in a top-level context for GET requests (i.e. if you clicked a link). Strict doesn’t send any third-party cookies.

Should I use it?

You should absolutely set Secure and HttpOnly. Unfortunately, as of writing, SameSite cookies are available only in Chrome and Opera, so you may want to ignore them for now.


Platform What do I do?
Rails 4 and 5 Secure and HttpOnly enabled by default. For SameSite, use secureheaders
Django Session cookies are HttpOnly by default. To set secure: SESSION_COOKIE_SECURE = True.

Not sure about SameSite.
Express.js cookie: { secure: true, httpOnly: true, sameSite: true }
Go http.Cookie{Name: "foo", Value: "bar", HttpOnly: true, Secure: true}

For SameSite, see this issue.
Nginx You probably won’t set session cookies in Nginx
Apache You probably won’t set session cookies in Apache

Thanks to @wolever for python advice

  1. This is opposed to “stored” XSS attacks, where the attacker is storing the malicious payload somehow, i.e. in a vulnerable comment field of a message board. 

  2. If you’re going to point out in the Hacker News comments that this blog itself gets an F from the Mozilla observatory, you’re right! On the other hand, it’s serving static content, and we are comfortable avoiding XSS protection and strict SSL enforcement for static content. That, and it’s served by github pages/cloudflare, so it’s hard to get very granular about the headers we want set. 

  3. So if you’re especially paranoid, you might be thinking “what if I had some secret subdomain that I don’t want leaking for some reason?” You have DNS zone transfers disabled, so someone would have to know what they’re looking for to find it, but now that it’s in the preload list… 

  4. The Referer header is the Hampster Dance in that it’s notorious for being misspelled. It would break the web to try to backport the correct spelling, so instead the W3C decided to go for the worst of both worlds and spell it correctly in Referrer-Policy

Good News: Ubuntu Now Ships With unattended-upgrades On By Default!

Last week, we got a strange support request. One of our users had received the following notification:

Hey! Good job.

We’ve detected that you patched some vulnerabilities.

Here’s what changed:


is no longer present in:

[name of server redacted]

This came as a surprise, since they knew for a fact that no one had touched the package in question, and they were certain they had not enabled unattended upgrades.

Somehow, the vulnerability magically got patched and they wanted to know: what’s going on?

The vuln is a pretty serious remote code execution vulnerability in memcached, and as far as we could tell our user was indeed using the most recent version available for their distribution — 1.4.25-2ubuntu2.1. This version was released on November 3rd, and we could see from our logs that memcached got upgraded that same day.

How did it happen without them knowing about it? The only thing unique about their configuration was that they’re running the recently released Ubuntu 16.10 (Yakkety Yak)1.

We dug around, and set up some test Yakkety boxes, and lo and behold: unattended upgrades is automatically enabled by default!

For those of you who are unaware, unattended-upgrades is a debian/ubuntu package that, well, does what it says on the tin: it automatically upgrades your packages. The most common configuration, and the one enabled in 16.10, is to upgrade any packages that have a published security patch. Unattended upgrades does this by checking and installing any updates from the ${distro_codename}-security repository.

Ubuntu/debian has had this for years, but it simply was never turned on by default. After a year of many security fails, this news warmed the cockles of my heart and gave me hope for our future! And what’s even amazing is that they turned it on without any fanfare.

It’s the quiet, simple changes that provide the biggest wins.

Of course, there are reasons why administrators don’t always want software to be upgraded without their input. And if it does get updated, there are good reasons for knowing exactly what vulnerabilities are being patched when. Appcanary exists in order to allow you to be notified about security updates without automatically installing them, and to have insight into what’s going being installed if you are patching automatically.

But if you don’t have the capacity to actively manage the packages on your linux systems (and even if you do!), we implore you: set up unattended-upgrades!

Ubuntu enabling this by default is a great sign for the future.

Not running Ubuntu 16.10?

Here’s how to turn on unattended upgrades

  • Ansible: jnv.unattended-upgrades
  • Puppet: puppet/unattended_upgrades
  • Chef: apt
  • If you’re using the server interactively:

    sudo apt-get install unattended-upgrades && sudo dpkg-reconfigure unattended-upgrades

  • Set up manually: sudo apt-get install unattended-upgrades and

    • In /etc/apt/apt.conf.d/20auto-upgrades:

      APT::Periodic::Update-Package-Lists "1";
      APT::Periodic::Unattended-Upgrade "1";
    • In /etc/apt/apt.conf.d/50unattended-upgrades

      // Automatically upgrade packages from these (origin, archive) pairs
      Unattended-Upgrade::Allowed-Origins {    
      // ${distro_id} and ${distro_codename} will be automatically expanded
          "${distro_id} ${distro_codename}-security";
      // Send email to this address for problems or packages upgrades
      // If empty or unset then no email is sent, make sure that you 
      // have a working mail setup on your system. The package 'mailx'
      // must be installed or anything that provides /usr/bin/mail.
      //Unattended-Upgrade::Mail "[email protected]";
      // Do automatic removal of new unused dependencies after the upgrade
      // (equivalent to apt-get autoremove)
      //Unattended-Upgrade::Remove-Unused-Dependencies "false";
      // Automatically reboot *WITHOUT CONFIRMATION* if a 
      // the file /var/run/reboot-required is found after the upgrade 
      //Unattended-Upgrade::Automatic-Reboot "false";

  1. 16.10 is not a Long Term Support release. Regular Ubuntu releases are supported for 9 months, while April releases on even years (i.e. 14.04, 16.04, etc…) are designated LTS, and are supported for 5 years. It’s thus more common to see 12.04, 14.04, and 16.04 in use on servers over other Ubuntu releases. This particular user has a good reason for running 16.10. 

We Left Clojure. Here's 5 Things I'll Miss.

On October 11th, Appcanary relied on about 8,500 lines of clojure code. On the 12th we were down to zero. We replaced it by adding another 5,700 lines of Ruby to our codebase. Phill will be discussing why we left, and what we learned both here and at this year’s RubyConf. For now, I want to talk about what I’ll miss.

1) The joy of Lisp

XKCD #297

There’s something magical about writing lisp. Alan Kay called it the greatest single programming language ever devised. Paul Graham called it a secret weapon. You can find tens of thousands of words on the elegant, mind-expanding powers of lisp1. I don’t think my version of the Lisp wizardry blog post would be particularly original or unique, so if you want to know more about the agony and ecstasy of wielding parenthesis, read Paul Graham.

What’s great about Clojure is that while Ruby might be an acceptable lisp, and lisp might not be an acceptable lisp, Clojure is a more than acceptable lisp. If we avoid the minefield of type systems, Clojure addresses the other 4 problems Steve Yegge discusses in the previous link2.

2) Immutability

The core data structures in clojure are immutable. If I define car to be "a dirty van", nothing can ever change that. I can name some other thing car later, but anything referencing that first car will always be referencing "a dirty van".

This is great for a host of reasons. For one, you get parallelization for free — since nothing will mutate your collection, mapping or reducing some function over it can be hadooped out to as many clouds as you want without changing your algorithms.

It’s also much easier to can reason about your code. There’s a famous quote by Larry Wall:

[Perl] would prefer that you stayed out of its living room because you weren’t invited, not because it has a shotgun.

He was talking about private methods, but the same is true for mutability in most languages. You call some method and who knows if it mutated a value you were using? You would prefer it not to, but you have no shotgun, and frankly it’s so easy to mutate state without even knowing that you are. Consider Python:

str1 = "My name "
str2 = str1
str1 += "is Max"
print str1
# "My name is Max"
print str2
# "My name"

list1 = [1, 2, 3]
list2 = list1
list1 += [4, 5]
print list1
# [1, 2, 3, 4, 5]
print list2
# [1, 2, 3, 4, 5]

Calling += on a string returned a new one, while calling += on a list mutated it in place! I have to remember which types are mutable, and whether += will give me a new object or mutate the existing one depending on its type. Who knows what might happen when you start passing your variables by reference to somewhere else?

Not having the choice to mutate state is as liberating as getting rid of your Facebook account.

3) Data first programming

Walking away from object-oriented languages is very freeing.

I want to design a model for the game of poker. I start by listing the nouns3: “card”, “deck”, “hand”, “player”, “dealer”, etc. Then I think of the verbs, “deal”, “bet”, “fold”, etc.

Now what? Here’s a typical StackOverflow question demonstrating the confusion that comes with designing like this. Is the dealer a kind of player or a separate class? If players have hands of cards, how does the deck keep track of what cards are left?

At the end of the day, the work of programming a poker game is codifying all of the actual rules of the game, and these will end up in a Game singleton that does most of the work anyway.

If you start by thinking about data and the functions that operate on it, there’s a natural way to solve hard problems from the top-down, which lets you quickly iterate your design (see below). You have some data structure that represents the game state, a structure representing possible actions a player can take, and a function to transform a game state and an action into the next game state. That function encodes the actual rules of poker (defined in lots of other, smaller functions).

I find this style of programming very natural and satisfying. Of course, you can do this in any language; but I find Clojure draws me towards it, while OO languages push me away from it.

4) Unit Testing

The majority of your code is made up of pure functions. A pure function is one which always gives the same output for a given input — doesn’t that sound easy to test? Instead of setting up test harnesses databases and mocks, you just write tests for your functions.

Testing the edges of your code that talk to the outside world requires mocking, of course, and integration testing is never trivial. But the first thing you want to test is the super-complicated piece of business logic deep in your codebase. The business logic your business depends on, like for instance computing whether your version of OpenSSL is vulnerable to HeartBleed.

Clojure pushes you to make that bit of code a pure function that’s testable without setting up complicated state.

5) Refactoring

Here’s a typical clojure function

(defn foo [a b]
  ;; some code here
  (let [c (some-function a b)]
    ;; a ton of 
    ;; complicated code here

In lisp-speak, a parenthesized block is called a “form”. The foo form is the outer form, and it contains the let form, which ostensibly contains other forms that do complicated things.

I know that all the complicated code inside of the let form isn’t going to mutate any state, and that it’s only dependent on the a and b variables. This means that refactoring this code out into its own functions is as trivial as selecting everything between two matching parentheses and cutting and pasting it out. If you have an editor that supports paredit-style navigation of lisp forms, you can rearrange code at lightning speed.

  1. My favourite essay of this ilk is Mark Tarver’s melancholy The Bipolar Lisp Programmer. He describes lisp as a language designed by and for brilliant failures. Back in university, I ate this shit up. My grades were obvious evidence of half the requirement of being a lisp programmer. 

  2. I’m aware that clojure’s gensym does not a hygenic macro system make. But, if you have strong opinions on hygenic macros as they relate to acceptable lisps, this article might not be for you. 

  3. For the record, I know that this isn’t the “right” way to design OO programs, but the fact that I have to acknowledge this proves my point. 

The Mirai Botnet is Proof the Security Industry is Broken

Last Friday, my workday was rudely interrupted because I couldn’t access Github. To add insult to injury I couldn’t even complain about it on Twitter. I tried to drown my sorrows by listening to moody Leonard Cohen songs on Spotify, but alas…

You’ve probably heard of this. Huge tracts of the Internet were down because the DNS provider Dyn faced a massive Denial of Service attack from the Mirai botnet, which takes advantage of Internet of Things devices like cameras and DVRs.

So, what’s new about Mirai?

I’ve written about 1988’s Morris worm, and I wanted to dig into the source of the Mirai botnet (helpfully published by the author) to see how far we’ve come along in the past 28 years.

Can you guess how Mirai spreads?

Was there new zeroday in the devices? Hey, maybe there was an old, unpatched vulnerability hanging — who has time to apply software updates to their toaster? Maybe it was HeartBleed 👻?


Mirai does one, and only one thing in order to break into new devices: it cycles through a bunch of default username/password combinations over telnet, like “admin/admin” and “root/realtek”. For a laugh, “mother/fucker” is in there too.

Default credentials. Over telnet. That’s how you get hundreds of thousands of devices. The Morris worm from 1988 tried a dictionary password attack too, but only after its buffer overflow and sendmail backdoor exploits failed.

Oh, and Morris’ password dictionary was larger, too.

How do we keep getting this wrong?

Around the world, we spend $75 billion a year on information security. And for what, when we keep getting such basic things wrong? Suppose I waved a magic wand and cut the worldwide security budget in half. Would things really be that much worse? The security industry is addicted to selling expensive complicated products instead of doing the basics well.

I was at a security conference the other week, and there was yet another crop of cyberapocalypse talks. The Internet of Things is a garbage fire. Industrial control systems are going to get us all killed. Users are clicking phishing links like sheep. We’re all doomed. And somehow, it’s always the fault of shitty programmers or dumb users. Let’s all laugh at their fails.

It’s all bullshit.

We sell biometric authentication systems to people who need a good password manager. We sell live threat attribution intelligence with colorful maps to people who need to practice configuration management. We sell advanced in-cpu sandbox endpoint protection to people who need to institute a patching program. There’s a reason why security practitioners get such a kick out of ThreatButt.

There are lots of real, important, conceptually difficult problems in security. We don’t really know how to write secure code, and it’s all too easy to get socially engineered. But, right now, the vast majority of threats can be thwarted by the basics:

  1. Keep your systems patched
  2. Keep your systems properly configured.
  3. Make sure you have strong passwords and two factor authentication.

Do the basics first. The basics matter. Then you can focus on the Sisyphean tasks that remain. Instead, here we are selling fancy bullshit and barely making any progress in 28 years. Lots of money in it, though.

Paying the Bills

Surprise, I also sell a security product! But I will say this: Appcanary isn’t going to protect you from shipping millions of internet-accessible cameras with the same password. We won’t even protect you from having your DNS provider DoSed.

The major botnet of 2016 is simpler than the botnet of 1988. There’s something wrong in how we do security, and at Appcanary, we think it’s a complete lack of focus on the basics.

The highest value, easiest thing you can do to improve your security is patch known vulnerabilities. Most breaches come from years-old vulnerabilities.

Our product, Appcanary, monitors your apps and servers, and notifies you whenever a new vulnerability is discovered in a package you rely on.

Sign up today!

A tale of two worms, three vulnerabilities, and one National Security Agency

Paranoia is natural for security practitioners.

Hacking can feel like being initiated into a secret society of wizards. Once you’re in, you get access to an addictive drug that gives you super powers. But there are other wizards out there; some are good but many practice black magic. And the NSA’s school of the dark arts has a seemingly unlimited budget.

It’s natural to get a little paranoid. Experience shows you that with the right incantation you can turn crashes into working exploits. It follows that every time your computer crashes there could be someone in the shadows, chanting the right incantation. The paranoia can be all-consuming; just because you’re paranoid doesn’t mean they’re not out to get you.

In October 2013, a well known computer security expert named Dragos Ruiu came out with a story. He found that his computers had been behaving oddly, and that the symptoms he was seeing were impossible to eradicate. This was some kind of worm, since the behavior would replicate across air gapped computers in his lab. He theorized that he was infected with a super advanced piece of malware that lived in the BIOS and could spread by sending ultrasonic frequencies from speaker to microphone, undetectable to the human ear. It looked like the work of the NSA or someone equally omnipotent. He dubbed it badBIOS.

Everything Dragos claimed badBIOS could do is at least possible, and most security folks know this. Malware in the BIOS is feasible, and beyond being a research topic, it’s something we know the NSA does. In fact, because of the hype, many people developed ultrasound networking libraries just to demonstrate how viable it is.

Dragos Ruiu imaged his computer and made a lot of data available to the community for peer review, but unfortunately no credible researcher1 has publicly confirmed his findings. Maybe there was something going on. Maybe he was seeing patterns in the noise. Either way, it says something about the world today that when you’re a security expert and your computer starts behaving weirdly, the obvious culprit is the NSA.

It made me think of a different worm, from a more innocent time.

Morris Worm

The Morris Worm

It’s November 2nd 1988, almost exactly 25 years before badBIOS became a hashtag. Robert Tappan Morris, a graduate student at Cornell, executes some code he’d been working on and goes to dinner. The aftermath was a self-replicating computer worm that infected 10% of the Internet2 at the time — a whopping 6,000 computers!

Morris claimed that he wrote his program to map the size of the Internet. And indeed, each infection would send a byte to a machine in Berkeley (hiding the trail to Morris, in Cornell, as the author). Unfortunately, there was a bug that caused it to propagate too aggressively: it infected the same computer multiple times, which resulted in a denial of service attack across the whole Internet. Furthermore, the code to report infections had a bug in it. It tried to send a UDP packet over a TCP socket, making it useless for reporting the Internet’s size.

An alternative explanation is that Morris was trying to bring to wider attention some long-standing bugs in the Internet. As Morris’ friend and future co-founder put it, in classic pg3 style:

Mr. Graham, who has known the younger Mr. Morris for several years, compared his exploit to that of Mathias Rust, the young German who flew light plane through Soviet air defenses in May 1987 and landed in Moscow.

“It’s as if Mathias Rust had not just flown into Red Square, but built himself a stealth bomber by hand and then flown into Red Square,” he said.

What did the Morris Worm actually do?

The Morris Worm4 exploited three separate vulnerabilities. It guessed passwords for rsh/rexec, it exploited a debug-mode backdoor in sendmail and it used “one very neat trick”. I’ll go over each of these in detail, and you can find an archive (decompiled and commented) of the code for yourself here.

1. Rsh and Rexec

rsh and rexec are remote shell protocols from the BSD era that are almost unused today (since supplanted by ssh). rsh can allow passwordless authentication if coming from a “trusted” host, which it determines via a list of addresses stored in a global /etc/hosts.equiv or per-user .rhosts file. When an rsh request comes from a user of a trusted machine, access is automatically granted. The worm used this to propagate, searching those two files — as well as the .forward file, which back then was used to forward your mail around the Internet — for trusted hosts.

Even in 1988, people knew that leaving rsh open on an untrusted network like the Internet was a Bad Idea, and so the worm also propagated via rexec. Now, rexec uses password authentication, but Morris made an intelligent assumption: people tend to reuse passwords. Back then, /etc/passwd used to5 store everyone’s encrypted passwords. The worm shipped with an optimized implementation of crypt and a dictionary, and went to town. Once it cracked a password, it tried it against all the likely hosts it could find.

2. Sendmail’s Backdoor

In the absence of any friendly hosts, the Morris Worm would then exploit a backdoor in Sendmail. You see, Sendmail had a “debug” mode that allowed anyone to route an email to any process, including the shell! Ironically, this was apparently deliberate:

Eric Allman, a computer programmer who designed the mail program that Morris exploited, said yesterday that he created the back door to allow him to fine tune the program on a machine that an overzealous administrator would not give him access to. He said he forgot to remove the entry point before the program was widely distributed in 1985.

(This wasn’t even the first Sendmail backdoor. Sendmail used to ship with “wizard mode”, where sending the strings “WIZ” and “SHELL” gave you a root shell. By the time that Morris was writing his worm, wizard mode was disabled almost everywhere.)

If you’re wondering how sendmail could have backdoors like this, it seems that it was somewhat well known. This quote from a mail by Paul Vixie summarizes the situation.

From: [email protected] (Paul Vixie)
Newsgroups: comp.protocols.tcp-ip,comp.unix.wizards
Subject: Re: a holiday gift from Robert "wormer" Morris
Message-ID: <[email protected]>
Date: 6 Nov 88 19:36:10 GMT
References: <[email protected]> <[email protected]>
Distribution: na
Organization: DEC Western Research Lab
Lines: 15

# the hole [in sendmail] was so obvious that i surmise that Morris
# was not the only one to discover it.  perhaps other less
# reproductively minded arpanetters have been having a field
# 'day' ever since this bsd release happened. 

I've known about it for a long time.  I thought it was common knowledge
and that the Internet was just a darned polite place.  (I think it _was_
common knowledge among the people who like to diddle the sendmail source.)

The bug in fingerd was a big surprise, though.  Overwriting a stack frame
on a remote machine with executable code is One Very Neat Trick.
Paul Vixie
Work:    [email protected]    decwrl!vixie    +1 415 853 6600
Play:    [email protected]     vixie!paul      +1 415 864 7013

The Internet was a polite place, indeed.

3. One Very Neat Trick

The Very Neat Trick that Vixie was talking about is the now-standard stack buffer overflow. It’s fascinating to read contemporary accounts that marvel at the cleverness of a class of bugs that are now ubiquitous — although, for me at least, they still haven’t lost their magic6.

Here’s the main routine from the fingerd of that era:

main(argc, argv)
    char *argv[];
    register char *sp;
    char line[512];
    struct sockaddr_in sin;
    int i, p[2], pid, status;
    FILE *fp;
    char *av[4];

    i = sizeof (sin);
    if (getpeername(0, &sin, &i) < 0)
        fatal(argv[0], "getpeername");
    line[0] = '\0';
    sp = line;
    // ... snip ...
    // build sp into arguments for finger 
    // and call /usr/ucb/finger via execv before
    // putchar'ing the result back to stdout

If you have experience with reading C code,7 you may have spotted the vulnerability. gets(line) reads STDIN and puts the contents into a 512 byte buffer. This means that sending more than 512 bytes will overwrite the stack with an attacker-controlled value.

The worm sent 536 bytes of data, which overwrote the stack frame of the main function. This allowed Morris to overwrite the pointer to where main is returning to. He set that pointer to be within the 536 byte buffer he sent over the network. The beginning of the buffer contained shellcode that called /bin/sh. Game over.


Robert Tappan Morris was convicted and sentenced to three years probation, 400 hours of community service and a $10,050 fine (about $20,000 in today’s dollars) plus the cost of his supervision. He then went on to co-found a little startup called Viaweb. You may have heard the rest of that story. Today, Morris is a tenured professor at the Computer Science and Artificial Intelligence Laboratory at MIT and is one of the leaders of the Parallel and Distributed Systems Groups.

Why did the paranoia around badBIOS make me think of the Morris Worm? If you read contemporary articles about the Morris Worm, they’ll sometimes mention, but never emphasize, who Robert Morris’s father was. The elder Robert Morris just happened to be a computer security expert. While the young Robert Morris was writing his worm, Robert Morris Sr. was serving as Chief Scientist at the NSA’s National Computer Security Center!

The Internet grew up a lot since 1988, and not just in size. In 2013, your computer acting strangely is obviously a NSA-written malware that lives in your BIOS and propagates over sound waves imperceptible to the human ear. In 1988, son of an NSA security executive infects 10% of the Internet with a worm that uses an exotic new exploitation technique called a buffer overflow and… nothing.

Just to be clear, I’m not alleging any conspiracy between father and son, besides perhaps father making some calls after son’s arrest. While the Morris worm was likely the first malicious use, buffer overflows were understood as a problem before 1988, if not widely. The way the media narrative handled the NSA connection in 1988 just says a lot about how the world of the Internet changed in 25 years.

As for Dragos Ruiu, he’s been quiet about badBIOS since 2013. I’m not sure what he’s doing these days besides CanSecWest, but in my heart of hearts, I like to picture him playing the saxophone amidst the detritus of his torn up apartment.

Paying the Bills

We’re trying our best, but we’ll only be able to blog about a minuscule percentage of the world’s vulnerabilities. And starting with 1988 means we have a lot of catching up to do. How will you ever find about the ones that actually affect you?

Our product, Appcanary, monitors your apps and servers, and notifies you whenever a new vulnerability is discovered in a package you rely on.

Sign up today!

  1. One of the things I wish that the security industry would do less of is blind appeals to authority, and I hate that I made one here. Unfortunately, I don’t have the skills or time to make my own analysis of Ruiu’s data, so I just have to trust the Thought Leaders on this one.  

  2. The 60,000 computer-strong Internet was of course one of many networks at the time. The Internet was the one that was global and used TCP/IP — the Internet protocols. Therein lies the pedant’s case against the AP’s capitalization of the word “Internet”. 

  3. Disclosure time: years after giving that quote, Paul Graham and Robert Morris went on to found Y Combinator along with Jessica Livingston and Trevor Blackwell. YC in turn is an investor in Appcanary. Robert Morris and I have never met, though we did once meet with Paul Graham.  

  4. My favourite paper on the analysis of the worm is With Microscope and Tweezers from MIT’s Eichin and Rochlis. They spend a page passionately arguing that it’s a virus by using a complicated appeal to the difference between lytic and lysogenic viruses with references to three separate biology textbooks! 

  5. I assumed that /etc/shadow came about as a consequence of the Morris Worm, but it seems that it was originally implemented in SunOS earlier in the 80’s, and then took 2 years after the Morris Worm to make it into BSD. 

  6. Exploits really are magic, and it goes without saying that exploit users have chosen the Left-Hand Path to wizardhood. If the cover of SICP is to be believed, the Right-Hand Path is available through careful study of functional programming and Lisps. Perhaps this is the true reason why Morris and Graham were such effective collaborators. 

  7. On the other hand, this C code is over 30 years old. When I ran it through the gcc on my machine,I was very happy to see that it complained bitterly but still compiled it. One exercise for the reader is finding where the network operation actually happens. main takes input and output from STDIN/STDOUT, but there’s an uninitialized struct sockaddr_in sin that we call getpeername on. How is a network socket piped to standard input/output and who is initializing the sin struct? I actually haven’t been able to figure this part out. If you know, please tell me! The full code listing is here

    Update 08/29/2016 Dave Vandervies emailed me with an explanation!

    fingerd was meant to be run from inetd (see here), which sets up the network connection and invokes the actual server process with its stdin and stdout attached to the network socket.

    As for the getpeername, the address is an out parameter; this call looks up the peer address of stdin (fd 0), and will fail (and fingerd will error out on that) if it isn’t a socket (see here). Since the actual address doesn’t get used, that appears to be the purpose of the call here.