GoKawiilThe release of Anthropic’s Fable (the “safety conscious” version of Mythos) has been a roller coaster ride for many reasons. Originally released on June 9th, the US government placed export controls on Fable and Mythos on June 12.
This meant that Anthropic could not serve the model to anyone who is not a US citizen, including their own employees. The impossibility of verifying all users at the risk of severe legal action made it functionally impossible to continue serving the model so Antrhopic pulled it.
After a couple of weeks of negotiations, export controls were lifted and Anthropic restored access to the model(s) (only Fable is generally available, Mythos is still available only to specific, pre-approved partners). All of this, of course, cause much drama online, and resulted in a tremendous amount of speculation and many news stories. Nonetheless, that is not what I wanted to focus my thoughts on here. Rather, I wanted to explain why I think that, at least for research-level tasks in Computer Science, Fable is not a useful model.
The first flop (understandable; but not really)
I wrote and maintain a widely-used tool for transcript quantification from RNA-seq data called salmon. The original salmon code was written in C++11/14, and early this year, I updated and modularized it a bit (with the help of ChatGPT), and brought it up to the C++20 standard. However, my more ambitious goal, and one that I’ve had for quite some time, was to re-write salmon in rust. Note: This did eventually happen, resulting in salmon 2 (which is already beyond parity with the C++ version and is adding new features quickly, but that is for another post).
When the original Fable release dropped on June 9, my first thought was to use it to help orchestrate the salmon rewrite. Now, I’ve succesfully ported or rewritten several other libraries or programs developed in the lab from C++ into rust, mostly with the help of Opus (4.6-4.8), and those went fairly smoothly. So, I thought that this would be a great opportunity to see how Fable compared. Alas, after spending some time crafting my detailed description and porting narrative, complete with implementation instructions and testing details, I submitted the query to Fable (in “plan” mode) and it immediately flagged my query on safety concerns, rejected it, and offered to send it to Opus 4.8 instead.
Naturally, I wanted to know why the prompt was rejected. Anthropic claims that Fable uses a “classifier” to determine what prompts are rejected and which are allowed, though it’s been widely reported that this classifier is stupendously miscalibrated. Nominally, it seems, my prompt was rejected because it involves working on software that deals with RNA-seq data, and the biological terminology in the documentation and source code set off the red flags needed to trigger the prompt rejection. I tried, to no avail, to explain to Fable that I am simply asking to re-write an existing, open source, widely used, publicly available C++ software tool in rust. The rewrite itself does not involve novel “biological” research. The inital ask is purely a software task, which is supposed to be a strong suit of Fable. Yet, not only did Fable refuse the prompt, but it also refused to disclose or explain why it refused the prompt in any detail, or to help me understand how to prompt it in a way so as to avoid this issue.
After perhaps 15 to 30 minutes of failed attempts at rephrasing my prompts, I gave up and simply proceeded with Opus 4.8 (which happily complied and, all said and done, did a great job with the port). However, this was my first inkling that the “classifier” used by Fable is not so much a classifier as, perhaps, a simple rejection list of terms and users, refusing to even begin to engage with subjects (or users) who have any relation to biological research (and likely, also, cybersecurity research).
This attempt was massively disappointing. Despite the clear lack of any security risk or implication, Fable flatly refused to engage with me in work for which it should be well-suited. Taking a peek on social media, I learned that my experiences were shared by many others, including biologists who had Fable refuse to answer prompts as innocuous as “why am I?”, “what is a mitochondrion?” and …. “what should I have for dinner?”. Truly strange.
The second flop (unforgivable)
... continue reading