Here in episode 3, Max and Phill discuss, over breakfast, what Shopify should or should not do with regard to Breitbart. Produced by Katie Jensen.
You can follow along on iTunes, or subscribe directly to the feed.
In November 2016, I delivered a talk at RubyConf where I told the story of how we built the first version in Appcanary in Clojure, and then threw it away.
Here is the talk:
Jump to 12:10 to start at the section where I begin to talk about my specific problems with Clojure.
But, that talk is over a half hour long, and who has time for that?
If you’re starting a project where you’re really pressed for time, you should be biased towards tools you know really well. If you have a startup, you’re supposed to be solving a business problem. Don’t make your life harder than it has to be by also having to learn new tools.
Avoid building a distributed system for as long as you can. They introduce a lot of expensive complexity that is hard for small teams to pay. Micro services are useful for larger teams, but the way you code and deploy your apps should be a reflection of your team’s organization and ability to coordinate.
I personally found it really hard to get into Clojure. I got the sense that Clojure as a community doesn’t value “developer happiness”, and I think it should! Tools should try to make your life easier at every opportunity.
We had mentioned the talk before, but I had avoided actually posting it on this blog for two reasons:
A lot of otherwise well-meaning people looked at the talk and decided that I was a) a bit of an asshole and b) probably incompetent. So many people expressed that view that I felt compelled to ask Confreaks to turn off the comments1.
I don’t really blame them.
Developer tools do not exist in isolation; they are often only useful to the extent there is a live community surrounding them. This requires such a degree of investment that many programmers will inevitably come to integrate their favourite programming languages into a part of their identity. When some outsider comes along and criticizes your community and identity, it can be very hard to be charitable to what they have to say.
It was a good learning experience, though. I came to understand that I did not work hard enough to establish that I was coming from a place of kindness, and that my talk was not a big fuck you! to the community.
I came into the language and its environment with an open heart and invested a ton of resources into it, but, largely because of my specific context and a few mistakes we made early on, continuing to invest in Clojure didn’t make sense for us.
So, much to my surprise, my talk became relevant recently when some peeps in the Clojure community looked around and made an observation:
folks, I’m calling it, clojure’s dying more than it is growing
— 🌹dsa zack🌹 (@ZackMaril) May 15, 2017
Enabled by Twitter’s recent decision to not count usernames towards character limits, this thread gained a whole life of its own. I don’t know how to do justice to the whole range of topics that were discussed; Twitter itself hasn’t quite figured out how to display threads properly.
I think some of the ideas brought up deserve a little more investigation, and so I have summarized parts of the discussion in order to address them. Here it goes:
The hype cycle has moved away from Clojure, and the community feels like it’s shrinking - though there are many more jobs coming up.
It’s actually kinda hard to make a good business case for Clojure, since the JVM makes small tools prohibitive and, if you do like the JVM, you now have a lot of options. In theory, you should be way more productive in Clojure, but it’s hard to tell how true that is.
That said, are people actually leaving Clojure? Well, there was a talk recently on that very topic, and though it may have been written by an incompetent and mean jerk2, it makes some good points.
In the end, what is a reasonable expectation for developers new to Clojure? Lisps are a very different paradigm. Things work differently here. Developers get paid to solve or automate problems, and people waste tons of time trying to be “happy”. Clojure encourages people to reflect on their work, and aggressively pare it down. After all, simple isn’t easy!
OKAY, WOW, are you still with me? Incredible.
It’s hard for me to write about Clojureland nowadays since it’s been a whole year since I last typed lein repl, but I’m about to say something slightly hyperbolic and slightly uncharitable because it’s more memorable, so please bear with me. It’s a little over the top, but I promise I’m coming from a place of love.
Here it goes. Ready?
For those of you outside of Clojureland, this is a reference to well-known 2011 Rich Hickey talk titled “Simple Made Easy”.
Rich’s thesis is that the best, most conceptually simple solution to a problem isn’t automatically also going to be simple to use.
Often, when we say something is easy, we actually just mean that it is familiar to us, and that we’ve used something like it before. Just because we don’t have to engage our brain doesn’t mean a concept or framework or artifact isn’t complex.
Often the simplest solution may be very different from what you were expecting, and may therefore require some elbow grease.
I thought that talk was really insightful when I first watched it. Math, for example, is a place where you can dedicate a lifetime of mental energy to simple but hard things. And it’s true that many easy things are needlessly complex. But, having seen how people refer to it over the years, I’ve since soured on the concept.
By separating “simplicity” from “ease of use”, and then establishing the primacy of simplicity, it gives people an ideological out to discount improving the user interface of our tools and artifacts.
If your chief goal is simplicity, and beauty, and elegance, then it’s not really your fault if people don’t “get it”. It blinds us to critiques of our software: if we don’t value the accessibility of our tools then it becomes very difficult to distinguish “there are a lot of arbitrary barriers preventing my understanding” from “I’m lazy and couldn’t be bothered to try hard enough”3.
Establishing this narrow view of “simplicity” as wholly separate from “easy to use” prevents us from being able to optimize for ease-of-use, aka developer experience.
In my talk I called the Clojureland ecosystem “user-hostile”. This contrasts sharply with the community, who are by and large very sweet and personable and well-meaning.
Undoubtedly, I chafed because Clojureland tools were unfamiliar, yes. But past a certain point they were also unfriendly, because in their error messages and affordances and failure conditions they assumed a rather high tolerance for digging through code and high-level knowledge of Clojure’s concepts and idioms and runtime quirks.
So we went back to Ruby, which we knew really well.
But here is the rub: Ruby’s tooling is terrible. The interpreter is slow and finicky and it used to leak memory like a sieve. There is no static introspection, the meta programming features are basic and slow and it is trivially easy to bleed state and complect the hell out of your programs. It was great compared to Java or PHP in 2004. In 2017, the world is different.
But, do you know what Ruby really has got going for it? Ruby is designed to be fun and make programmers happy. I think this is such a dope design philosophy that I’ll go one step further:
I think this is by far the most important contribution Ruby brought to programming, and as Ruby has stabilized and become less exciting and people have moved on, it has been incredible to ex-Rubyists bring these values with them to their new communities.
You can see this attitude in Elixir, in Rust, in Ember. Our tools should help you achieve success. Our tools should get out of the way. Our tools should Just Work.
The truth is, programming can be complex, but it is also often needlessly hard. Programmers are the end-users of programming tools, and to ignore their experience using them is just plain bad design. We should strive to be as maximally welcoming as we can, for every level of experience.
I called it “abuse” elsewhere, which is maybe a bit much. People were mostly just somewhat rude. It’s just not… pleasant to have a bunch of people calling you incompetent over and over again ¯\_(ツ)_/¯. ↩
I am exaggerating and bear no ill will ;). ↩
By far the most common response to my talk was, but are you sure you tried hard enough? If you keep trying, you’ll get it. Just work harder at it. There’s a hidden insinuation: maybe you’re not smart enough to get it. I’d like to think that I am. ↩
Here in episode 2 of the Appcanary podcast, Max and I discuss a Russian casino hacking ring, and pseudorandom number generators. Produced by Katie Jensen.
You can follow along on iTunes, or subscribe directly to the feed.
Max and I are both avid podcast consumers, and we thought it’d be fun and interesting to produce a podcast. So… we did!
You can follow along on iTunes, or subscribe directly to the feed.
In our inaugural episode, we discuss MongoDB ransomware, and the opacity of algorithms. Produced by Katie Jensen.
How the machine ‘thinks’: Understanding opacity in machine learning algorithms, by Jenna Burrell (Big Data & Society, Jan 6th 2016)
Machine Bias - Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner (Pro Publica, May 23rd 2016)
Benjamin Walker’s Theory of Everything
Ransomware crews are no longer satisfied shutting down hospitals and are now also going after hip tech startups. Last week, a professional crew got involved in ransoming MongoDB instances, and some 28,000 servers were compromised and held for Bitcoin.
The “attack” is so simple it barely deserves the name. The crew scans the internet looking for open and unsecured instances of MongoDB. When they find one, they log in and steal the data. The victims must pay Bitcoin to get their data back. To save on storage costs, sometimes the crew just deletes the data without bothering to save it — and the mark ends up paying for nothing. Even honor among thieves can’t survive the harsh realities of 21st century globalized late cyber-capitalism.
When reading about these attacks, there is a strong temptation blame the victim. Hospitals were at fault because of their antiquated IT departments: huge, sclerotic fiefdoms incapable of moving past Windows 98. We all know that MongoDB sucks, and that Real Programmers would never be caught dead using it. Why, they probably deserved to get hacked!
Don’t do it. As I’ve said before, victim-blaming in security is a trap that must be avoided. Attacks are varied and always changing; best practices can be indistinguishable from cargo cults, and default configurations seem almost designed to foil you.
In this case, MongoDB apparently ships without any authentication, listening to the whole wide world. No wonder so many instances got hacked!
Basically, you should have as little as possible accessible to the outside world. Set up a firewall. This tweet sums up the idea best:
If Postgres, MySQL, memcached, redis, etc responds to commands from outside the VPN/firewall/etc, assume you will lose your deployment.
— Patrick McKenzie (@patio11) January 10, 2017
If you’re anything like me, the first thought when confronted with that tweet is: “is my firewall actually working right?” Would you bet money that you know what ports are open on a given server in your fleet?
Nmap as used in The Matrix
Nmap is an incredibly powerful tool that’s synonymous with port scanning. A port scanner is a program that checks to see what network ports are open on a computer.
Here’s what scanning appcanary.com with default settings from my work computer looks like:
$ nmap appcanary.com
Starting Nmap 7.12 ( https://nmap.org ) at 2017-01-16 12:57 EST
Nmap scan report for appcanary.com (45.55.197.253)
Host is up (0.030s latency).
Not shown: 993 closed ports
PORT STATE SERVICE
22/tcp open ssh
25/tcp filtered smtp
80/tcp open http
135/tcp filtered msrpc
139/tcp filtered netbios-ssn
443/tcp open https
445/tcp filtered microsoft-ds
You’ll see that 22, 80, and 443 are open. This is because we listen on SSH, HTTP and HTTPS. There are also a bunch of filtered ports. This is because some router or firewall between me and appcanary is blocking that port so it’s not clear whether it’s open or closed.
Running the same scan from a random1 Digital Ocean box gives me the true result without the intermediate network filtering.
Starting Nmap 5.21 ( http://nmap.org ) at 2017-01-16 13:04 EST
Nmap scan report for appcanary.com (45.55.197.253)
Host is up (0.0012s latency).
Not shown: 997 closed ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
Nmap has a variety of scanning options, but the most important thing to understand for TCP scans is the difference between SYN scanning and connect scanning.
If you’ve ever tried to write a port scanner (which is a great exercise in network programming and concurrency), you’ve probably tried to write a connect scanner. A connect scanner tries to connect to a port, and closes the connection after success. Nmap’s most popular mode is SYN scanning. In SYN scanning, you send a SYN packet initiating the TCP handshake. If you get an SYN-ACK back, the port is open, and you send a RST instead of an ACK as you would if you were actually connecting. This is “stealthier”, which doesn’t matter as much for our purposes, and is more efficient. Unfortunately it requires nmap to be run as root because it needs raw access to the network interface.
You can specify a SYN scan with nmap -sS, and a connect scan with nmap -sT. By default nmap will prefer to do a SYN scan if it is able to.
By default, nmap will scan the top 1000 most common ports. You can specify a specific ports or a range with -p, i.e. nmap -p22 will scan only port 22, and nmap -p400-500 will scan ports 400 through 500. nmap -p- will scan the complete port range (1-65535).
Regularly port scan yourself; it’s the only way to be certain that your databases aren’t listening to the outside world. Run Nmap against your servers, and make sure that only the ports you expect are open.
To make it easier, here’s a script to do it for you. This will run Nmap, compare the output with predefined ports, and ping you on Slack if there’s a mismatch. Run it on a cron job, so you can check on the regular.
Our mission is to help you do security basics right, so you can be free to worry about the hard stuff. We’re building the world’s best patch management product because we had a deep need for one ourselves, and it automated such a vital part of a security team’s job.
When I started writing this article, I didn’t realize how well it fit into what we’re doing with patch management. To do security right, you need to implement systems and continuously verify they are working correctly. Appcanary helps you verify your patch management program, the script above helps you verify your firewall. Both without any bullshit.
Did I convince you to port scan your systems after reading the article? Are you using the script above? Something else? Let me know: max@appcanary.com.
If you’re observant, you may have noticed that this version of nmap is much older than the one on my work computer. If you’re really observant, you may have noticed that it has a vulnerability. ↩