One of the biggest lessons that I've learned in my career is that all software has bugs, and the more complicated your software gets the more complicated your bugs get. A lot of the time those bugs will be fairly obvious and easy to spot, validate, and replicate. Sometimes, the process of fixing it will uncover your core assumptions about how things work in ways that will leave you feeling like you just got trolled.
Today I'm going to talk about a single line fix that prevents people on a large number of devices from having weird irreproducible issues with Anubis rejecting people when it frankly shouldn't. Stick around, it's gonna be a wild ride.
Anubis is a web application firewall that tries to make sure that the client is a browser. It uses a few challenge methods to do this determination, but the main method is the proof of work challenge which makes clients grind away at cryptographic checksums in order to rate limit clients from connecting too eagerly.
note In retrospect implementing the proof of work challenge may have been a mistake and it's likely to be supplanted by things like Proof of React or other methods that have yet to be developed. Your patience and polite behaviour in the bug tracker is appreciated.
In order to make sure the proof of work challenge screen goes away as fast as possible, the worker code is optimized within an inch of its digital life. One of the main ways that this code is optimized is with how it's run. Over the last 10-20 years, the main way that CPUs have gotten fast is via increasing multicore performance. Anubis tries to make sure that it can use as many cores as possible in order to take advantage of your device's CPU as much as it can.
This strategy sometimes has some issues though, for one Firefox seems to get much slower if you have Anubis try to absolutely saturate all of the cores on the system. It also has a fairly high overhead between JavaScript JIT code and WebCrypto. I did some testing and found out that Firefox's point of diminishing returns was about half of the CPU cores.
One of the complaints I've been getting from users and administrators using Anubis is that they've been running into issues where users get randomly rejected with an error message only saying "invalid response". This happens when the challenge validating process fails. This issue has been blocking the release of the next version of Anubis.
In order to demonstrate this better, I've made a little interactive diagram for the proof of work process:
1. Challenge 3e2c67c9ef91d81fff589db473a2f996 2. Nonce 0 3. Combined Data 3e2c67c9ef91d81fff589db473a2f9960 4. Resulting Hash (SHA-256) ... Auto-Mine New Challenge Reset Nonce
I've fixed a lot of the easy bugs in Anubis by this point. A lot of what's left is the hard bugs, but also specifically the kinds of hard bugs that involve weird hardware configurations. In order to try and catch these issues before software hits prod, I test Anubis against a bunch of hardware I have locally. Any issues I find and fix before software ships are issues that you don't hit in production.
... continue reading