What to do with this?
I am a network engineer advising small companies, but I have about 5 clients in different countries, so I have very good visibility into these problems. I can assure you that this is an endemic problem that translates into a terrible experience for Meta services. This failure has been present for about 7 months, and it has been 5 months since I first reported it. This has happened on a previous occasion and I faced the same reporting difficulties, something like:
Me: Hi, I see a problem here. Meta: Hi, how about sending me some irrelevant evidence? Me: I’m sending it because I know it’s protocol, but I’m trying to send the correct information too. Meta: We don’t see anything, bro. Me: OK, I understand you don’t see anything, but look, there are a lot of affected networks (sending evidence). Meta: Hmmm we’re not going to fix anything we don’t see.
The same thing happened in 2023. The ticket was resolved after 6 months when I used a “fake” ticket to get someone from Meta technical support to check the failing interface, and it was fixed 1 day later. So I know firsthand that your monitoring systems simply don’t work for these types of problems.
How can Meta replicate the failure?
1: Look for random MNA cluster IPs from your clients. 2: Ping from 157.240.14.15 with a payload larger than 500 bytes (a packet is more likely to get corrupted on a faulty interface if the payload increases). 3: Ping many servers from point 1.
You will see that once you find the affected upstream or downstream route combination, you will have 10-60% packet loss to the destination host.
How to fix it? Isolate the port or discard faulty hardware.
Why didn’t we see it before?
Simply put, your monitoring tools and troubleshooting protocols don’t work for these problems. The protocol is to attach a HAR file that bases its performance on window scaling and TCP RTT; if both are good, even with data loss, there’s “no problem.” Especially because that HAR file is extracted using QUIC, and QUIC is particularly good at mitigating slowness caused by data loss (since packets are retransmitted without the TCP penalty). You know what uses TCP? WhatsApp Statuses, and those are slow.
... continue reading