GoKawiilSome bugs announce themselves loudly. A null pointer in a hot path, a missing route that returns 404 to every request — the kind of thing that fails immediately and points straight at the cause. Others are quieter. They let most of the stack work correctly and only reveal themselves at the intersection of two independently correct but mutually incompatible assumptions. The azurerm_storage_table_entity failure was the second kind.
This post is an account of a two-day investigation into a persistent 401 Unauthorized response on Terraform table entity operations, the four separate bugs we found along the way, and how pairing with GitHub Copilot shaped the investigation. The fix touched authentication, HTTP routing, upsert semantics, and stream lifecycle — each uncovered only after the previous one was resolved.
Topaz already had table storage support: accounts, tables, entity insert and query. The azurerm_storage_table Terraform resource worked. What did not work was azurerm_storage_table_entity . Any Terraform run that tried to create a table entity failed immediately:
Error: creating Entity (Partition Key "pk1" / Row Key "rk1" / ...):
executing request: unexpected status 401 (401 Unauthorized) with EOF
The 401 was puzzling because the same storage account, created by the same Terraform run, authenticated correctly for every ARM-level operation. Listing keys worked. Creating the table worked. Only the data-plane entity operation failed.
The first hypothesis was key mismatch. Terraform calls listKeys to get a storage account key and then uses it to sign data-plane requests with HMAC-SHA256. If Topaz returned a different key than it used to verify the signature, every request would fail.
To confirm or rule this out, we added diagnostic logging across the authentication path. The TableStorageSecurityProvider was modified to emit the full stored keys (as base64 and as raw bytes in hex), the received signature, the computed signature for both keys, the full Authorization header, every request header with its raw bytes, and the exact string-to-sign in hex. ListStorageAccountKeysEndpoint was modified to log a prefix of both keys on every call, so we could correlate what Terraform received with what Topaz verified against.
The first finding was that the keys were stable. Terraform's listKeys calls (there were four of them per apply — the provider is aggressive about refreshing credentials) consistently received the same key1 prefix. The key logged at verification time matched. The key mismatch hypothesis was eliminated.
... continue reading