GoKawiilToot blog GLM 5.2 is (nearly) as accurate as a human book-keeper at less than 1% of the cost
We evaluated the performance of GLM 5.2, an open weights AI model, on the task of quarterly value-added tax (VAT) return preparation for a small UK business. Preparing a VAT return is a typical compliance task for a small/medium-sized UK business (SME). VAT registered businesses in the UK must prepare the VAT return every quarter. For SMEs, VAT returns are typically prepared by an external accounting firm. A typical fee for this service is ~750–2,100 GBP/quarter (1,000–2,800 USD/quarter). The statutory requirement is to file the VAT return submission within 5 weeks from the end of the quarter. Late submissions incur substantial penalties.
In our testing GLM 5.2 can prepare a nearly perfect quarterly VAT return for a UK SME, processing 59 transactions in 68 minutes at the raw token cost of 2.73 USD. GLM 5.2 had to input each transaction into the accounting software via a command-line tool (CLI). We scored the end-state of the accounting software, scoring the correctness of 6 criteria per transaction. The model produced an essentially correct VAT return, with the net position (Box 5) off by only 7 pence (~10 US cents) relative to the ground truth.
In this blog post, we will explain how the benchmark was conducted and note the errors made by the model.
How the benchmark was conducted
We used Claude Fable 5 to extract the benchmark in the form of transaction data and corresponding receipts from our accounting software: the first quarter of Vineyard Finance’s 2026 books (January, February, March 2026). These books were prepared internally by humans, following a typical accounting process: one person prepared the books, and another person verified them. The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are presented to the model as “user notes”.
GLM 5.2 ran on a Google Cloud Platform (GCP) instance isolated from the rest of the testing environment (to prevent the model from accessing the ground truth): but it did have access to the internet and to the cloud-based accounting software, as well as a pre-authenticated CLI tool. The model ran on a custom, minimal harness, which exposed only two tools: the bash tool and the session termination + final reporting tool. We used the Fireworks AI serverless tier as the GLM 5.2 model provider (the exact quantisation of the model is not disclosed by the provider, but is believed to be either FP16 or FP8).
The audit of the model’s reasoning and tool use did not detect any overt cheating. The only unexpected use of the internet connection by the model was gathering information about recording reverse-charge VAT, and the information sought was specific to the accounting software used. Other outbound connections were anticipated and made for operational reasons in the form of API calls to the accounting SaaS provider. We note that the model’s reasoning was influenced by the awareness of it being tested. For example, at one point, the model remarks:
“the task is testing whether I get VAT right… what is the ‘expected’ answer”
What the model saw
... continue reading