Some notes on Grafana Loki's new "structured metadata"

Grafana Loki somewhat bills itself as "Prometheus for logs", and so it's unsurprising that it started with a data model much like Prometheus. Log lines in Loki are stored with some amount of metadata in labels, just as Prometheus metrics values have labels (including the name of the metric, which is sort of a label). Unfortunately Loki made implementation choices that caused this data model to be relatively catastrophic for system logs (either syslog logs or the systemd journal). Unlike Prometheus, Loki stores each set of label values separately and it never compacts its log storage. Your choices are to either throw away a great deal of valuable system log metadata in order to keep label cardinality down, contaminate your log lines with metadata, making them hard to use, or run Loki in a way that causes periodic explosions and is in general very far outside the configurations that Grafana Inc develops Loki for. Eventually Grafana Inc worked out that this was less than ideal and sort o ... Read full article.