Web Bot Auth is an authentication method that leverages cryptographic signatures in HTTP messages to verify that a request comes from an automated bot. Web Bot Auth is used as a verification method for verified bots and signed agents.
It relies on two active IETF drafts: a directory draft ↗ allowing the crawler to share their public keys, and a protocol draft ↗ defining how these keys should be used to attach crawler's identity to HTTP requests.
This documentation goes over specific integration within Cloudflare.
1. Generate a valid signing key
You need to generate a signing key which will be used to authenticate your bot's requests.
Generate a unique Ed25519 ↗ private key to sign your requests. This example uses the OpenSSL ↗ genpkey command: Terminal window openssl genpkey -algorithm ed25519 -out private-key.pem Extract your public key. Terminal window openssl pkey -in private-key.pem -pubout -out public-key.pem Convert the public key to JSON Web Key (JWK) using a tool of your choice. This example uses jwker ↗ command line application. Terminal window go install github.com/jphastings/jwker/cmd/jwker@latest jwker public-key.pem public-key.jwk
By following these steps, you have generated a private key and a public key, then converted the public key to a JWK.
Note You can also generate a JavaScript key using WebCrypto API ↗, which will produce a key in the correct JWK format. Many existing JWK libraries ↗ support WebCrypto API for generating JavaScript key.
2. Host a key directory
You need to host a key directory which creates a way for your bot to authenticate its requests to Cloudflare. This directory should follow the definition from the active IETF draft draft-meunier-http-message-signatures-directory-01 ↗.
... continue reading