CloudFlare Throttling Issue (FTTH)

andyxl987 · ‎20-08-2023

I have EE's Full Fibre package and hadn't had any problems in about 10 months of service. However, for the last 24 hours (maybe longer) I am experiencing extremely slow speeds to certain sites. For example, loading a discord channel may take 4 minutes to load. The emoji panel makes dozens of requests in a short frequency and also fails to load correctly.

I do not experience this issue while connected to a VPN.

I experienced something similar on Friday using 5G (also with discord) but put that down to signal issues.

Discord is not the only site that is affected. I host some public services on a remote server (Germany) and use CloudFlare proxy in front of these. Speed issues are also apparent for these, which again, do not reproduce when using a VPN. In fact, if I edit /etc/hosts to bypass CloudFlare's proxy, everything loads instantly. Note, that Discord also uses CloudFlare.

While it's possible the issue is my end, e.g. my router, Friday's issue while on mobile has me wondering if there's some issue between EE and CloudFlare? There appears to be severe throttling where the first several requests are fine and any subsequent requests are exponentially slower to send/receive.

I've also encountered similar while pulling images from docker hub and github container registry (although I do not believe either use CloudFlare).

pryswilliams · ‎25-08-2023

Things appear to have been resolved. I'm now getting comparable speed test results from Cloudflare vs other sites. Docker image pulls are ok, too. I did refresh my router's internet connection and was allocated a different public IP than I had previously, but both within the 2.28.0.0/14 range.

Akiyama8641 · ‎25-08-2023

Not resolved here. Issues still persist. IP within the 2.24.0.0/14 range.

RukaziSintakumo · ‎25-08-2023

When I did a test 15 mins ago, it was working fine (with the Cloudflare Speedtest), however I‘m back to the usual issue that I faced as well. Issue seems intermittent!

IP range is 95.145.x.x

pryswilliams · ‎25-08-2023

Apologies. I was being slightly bombastic - I have no way of telling if the issue is resolved or not, but the symptoms my end do appear to be ameliorated.

Packet captures of a docker image pull are now much cleaner, whereas earlier I was seeing multiple retransmissions, duplicate ACKs, segments not captured etc. I'll update if things change.

bobpullen · ‎26-08-2023

Doesn't seem to be any correlation with originating IP address then.

@andyxl987 wrote:
MTR shows last hop packet loss when the following route is used: 87.237.21.10 -> 172.71.176.4 -> Cloudflare IP.
No last hop packet loss is observed when the route is 87.237.21.10 -> 141.101.71.47 -> Cloudflare IP.

The IP's in bold are also Cloudflare's, but looking at your MTR, I'm assuming you mean the packet loss starts at the 87.237.21.10 address?

andyxl987 · ‎26-08-2023

@bobpullen wrote:
The IP's in bold are also Cloudflare's, but looking at your MTR, I'm assuming you mean the packet loss starts at the 87.237.21.10 address?

That is correct, packet loss was observed for all of the last 3 hops. However, this may not be a meaningful test as it is incredibly rare that I get a route other than via 172.71.176.4 so lack of packet less via 141.101.71.47 could be coincidental. If I've not used my connection in a couple of hours, everything works ok for the first few requests and then the connectivity issues surface.

This morning things aren't too bad. Discord is loading with mostly acceptable performance (not good, but not taking several minutes). I've had to abort (ctrl+c) and restart some docker pulls. However, installing packages from npm is working a lot better today 🤞.

deemzed · ‎26-08-2023

I've been having the same issue as described for the past week. Very slow downloads and broken images to specific sites only. At first, I suspected a dodgy discord update, but moving entirely off the app to web-based made no difference, and it remained fast over mobile connections (non-EE). I suspected wifi issues, but no, eliminated that as a cause also. The cloudflare page is showing _intermittent_ issues - it's fine with occasional dropout/packet loss (normally >60Mbps, here it is during a dropout).

andyxl987 · ‎26-08-2023

Would this be able to be escalated to the next tier of support at EE? When I spoke to customer support I took away the impression that at least 5 reports were required before the issue may be accepted as being the providers issue and elevated. There are more than 5 affected customers in this thread.

Reliable access to Cloudflare is essential, a huge amount of the internet is proxied via their services. In a typical day my Linux machines requires acess to Cloudflare to update system packages. As a software engineer, my personal workflow requires installing dependencies from various package managers and pulling docker images, a large proportion of these are also behind Cloudflare services. I cannot work effectively without reliable access to Cloudflare.

Mustrum · ‎26-08-2023

@andyxl987 not sure you will get much support on here, as already mentioned there is no direct EE technical support on here. Not sure if others have reported or spoken to EE support.

Is there any way to report and get any answers from Cloudflare?

They seem to have tools, but not sure if any can be trusted. Their speed test tool is very variable.

Also not seeing any other ISP's reporting issues.

That said, I have seen Cloudflare issues when trying to get through to Planefinder but that is about it. I am on a 2.28.5.xxx ip at the moment - but happy to try another tomorrow and report.

andyxl987 · ‎26-08-2023

@Mustrum wrote:
Is there any way to report and get any answers from Cloudflare?
They seem to have tools, but not sure if any can be trusted. Their speed test tool is very variable.

Cloudflare's speed tool is one of the more advanced ones in terms of being able to export detailed results. They are also one of the largest global content delivery networks - their services are generally excellent. I find the speed reliable while using a VPN. It is only on EE where I/we are seeing issues.

It could be that raising an issue with Cloudflare may produce a quicker result than trying to go through EE (if the problem is on Cloudflare's end).

My previously attached VPN results were using PIA (Southampton). Below is a speed test using Mullvad (London). As before, there are no packet loss or speed issues when accessing Cloudflare via Mullvad. I am including this result only because Mullvad (London) routes similarly to EE, e.g. via 172.71.176.4.

And mtr results:

$ mtr discord.com -c 50 -r
Start: 2023-08-26T23:28:09+0100
HOST: **redacted**                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway                   0.0%    50    4.5   5.1   3.8  10.0   1.0
  2.|-- lon-ix1-cr1-v11.31173.se   0.0%    50    8.1   9.3   4.4  66.6  13.2
  3.|-- 195.66.227.207             0.0%    50   11.8   9.9   5.3  32.3   6.2
  4.|-- 172.71.176.4               0.0%    50    6.4   8.7   5.6  22.8   4.0
  5.|-- 162.159.135.232            0.0%    50    5.6   6.6   5.4  14.9   1.9

@Mustrum wrote:
@andyxl987 not sure you will get much support on here, as already mentioned there is no direct EE technical support on here. Not sure if others have reported or spoken to EE support.

Those of us affected, if you have not contacted EE CS, it may be worth doing so and linking them to this thread so they can tie the reports to a single case, maybe it will help trying to get them to investigate.