GoKawiilA few evenings ago my partner and I were in the mood to install this particular Linux ISO. The guys from Red Letter Systems had spoken of it well, it had good reviews on boundingboxd, and my partner liked the installation trailer.
I opened Tremotesf on my phone to check whether I already had it on the Pi, but the torrent list wouldn't load. The SMB mount wouldn't mount. I couldn't SSH into it either.
Not eager to ruin the evening by starting a debugging session then and there, I chalked it up to a simple issue with the recent changes I made to the NixOS configuration and resolved to take a look at it the next day.
The next day
I hooked the Pi up to the dedicated debugging display I have in the closet, cracked knuckles, said "it's debugging time" or something to that effect, and rebooted the brick.
The current NixOS generation started booting, loaded the kernel and... hung there. No more output. The previous generation did the same. I took the microSD card out for further inspection.
The first obvious step was to fsck it; the power had mysteriously gone out in the whole building a few days prior, and perhaps it happened right when one of the blocks containing some files important for the boot process was being written? Anyway, it was worth a check. fsck.ext4 reported and fixed lots of errors. Running it again reported the FS as clean, but ejecting and then plugging the card back in turned up more errors...
Long story short, the card was likely dead, and the blackout had simply delivered the coup de grace. Press F to pay respects.
Taking inventory
This Raspberry Pi 4B had been running for years almost 24/7 so this didn't come as a surprise. My understanding is that SD cards can sustain a comparatively small number of write cycles before they start to fail. The surprising part was that mine managed to run for so long with no issues, considering how badly I'd been treating it. The microSD was mounted as the root FS and I had taken next to no precautions to minimize writes.
... continue reading