TL;DR:
Put that in your nginx config:
location / { # needed to still allow git clone from http/https URLs if ($http_user_agent ~* "git/|git-lfs/") { set $bypass_cookie 1; } # If we see the expected cookie; we could also bypass the blocker page if ($cookie_Yogsototh_opens_the_door = "1") { set $bypass_cookie 1; } # Redirect to 418 if neither condition is met if ($bypass_cookie != 1) { add_header Content-Type text/html always; return 418 ''; } # rest of your nginx config
Preferably run a string replace from Yogsototh_opens_the_door to your own personal Cookie name.
Main advantage, is that it is almost invisible to the users of my website compartively to other solutions like Anubis.
More detail
Not so long ago I started to host my code to forgejo. There is a promise that in the future it will support federation and forgejo is the same project that is used for codeberg.
The only problem I had was one day, I discovered that my entire node was down. At first I didn't investigate and just restarted the node. But soon after a few hours, it was down again. Looking at the reason, clearly thousands of requests that looked at every commit which put too much pressure on the system. Who could be so interested in using the web API to look at every commit instead, of… you know, clone the repository locally and explore it. Quickly, yep, like so many of you, I discovered that tons of crawlers that did not respect the robots.txt are crawling my forgejo instance until death ensues.
So I had no choice, I first used a radical approach and blocked my website entirely except from me. But hey, why having a public forge if not for people to be able to look into it time to time?
I then installed Anubis, but it wasn't really for me. It is way too heavy for my needs, not as easy as I would have hoped to configure and install.
... continue reading