Is the doc bot docs, or not?

July 8, 2025

Upgrading my Shopify email noti­fi­ca­tion templates this morning, I asked Shopify’s LLM-powered devel­oper doc­u­men­ta­tion bot this ques­tion:

What’s the syntax, in Liquid, to detect whether an order in an email noti­fi­ca­tion con­tains items that will be ful­filled through Shopify Collective?

I’d done a few tra­di­tional searches, but couldn’t find an answer, so I thought, okay, let’s try the doc bot!

The reply came, quick and clean:

{% if order.tags contains 'Shopify Collective' %}

Some items in your order are being fulfilled by a partner merchant through Shopify Collective.

{% endif %}

Looks great. I added this code, but it didn’t work; the con­di­tion wasn’t being satisfied, even when orders had the nec­es­sary tag, vis­ible on the admin screen.

Shopify doesn’t pro­vide a way to test uncon­ven­tional email for­mats without actu­ally placing real orders, so I did my cus­tomary dance of order-refund, order-refund, order-refund. My credit card is going to get locked one of these days.

Turns out, the Shopify Collective tag isn’t present on the order at the time the con­fir­ma­tion email is generated. It’s added later, pos­sibly just moments later, by some other cryptic Shopify process.

I don’t think this is doc­u­mented anywhere, so in that sense, it’s not sur­prising the doc­u­men­ta­tion bot got it wrong. My ques­tion is: what’s the point of a doc bot that just takes a guess?

... continue reading