CloudFlare Throttling Issue (FTTH)

andyxl987 · ‎20-08-2023

I have EE's Full Fibre package and hadn't had any problems in about 10 months of service. However, for the last 24 hours (maybe longer) I am experiencing extremely slow speeds to certain sites. For example, loading a discord channel may take 4 minutes to load. The emoji panel makes dozens of requests in a short frequency and also fails to load correctly.

I do not experience this issue while connected to a VPN.

I experienced something similar on Friday using 5G (also with discord) but put that down to signal issues.

Discord is not the only site that is affected. I host some public services on a remote server (Germany) and use CloudFlare proxy in front of these. Speed issues are also apparent for these, which again, do not reproduce when using a VPN. In fact, if I edit /etc/hosts to bypass CloudFlare's proxy, everything loads instantly. Note, that Discord also uses CloudFlare.

While it's possible the issue is my end, e.g. my router, Friday's issue while on mobile has me wondering if there's some issue between EE and CloudFlare? There appears to be severe throttling where the first several requests are fine and any subsequent requests are exponentially slower to send/receive.

I've also encountered similar while pulling images from docker hub and github container registry (although I do not believe either use CloudFlare).

andyxl987 · ‎22-08-2023

Other speed test sites are OK. The following are taken using a WiFi connection (and other devices are in use):

fast.com (wifi):
620 download
94 upload

speedtest.net (wifi):
536.19 download
104.74 upload

speedtest.net (router inbuilt):
916.83 download
111.42 upload

think broadband (wifi):

Under ideal (WiFi) conditions I can get over 700 Mbps from fast.com and Steam if no other devices are in use. But trying to load a few megabytes channel from discord or install packages from npm (node package repository) takes longer than to download a small game 😞

Mustrum · ‎22-08-2023

Thanks @andyxl987 alas though that will make it harder to get an answer.

Unlike some other forum's there is no direct access on here to the Broadband Technical Care team, and as discovered getting beyond first line is not simple. From your and other posts many first line teams would struggle to understand the issues.

Besides it would appear the issues are outside the EE network so would need more detailed internet investigation.

You could try looking on the BT Forum to see if there are any similar issues being reported, or even on more specialist forums such as Kitz, Thinkbroadband or ISPReviews.

HTH,

andyxl987 · ‎22-08-2023

Here's a small example showing the instability with an example Cloudflare protected host. This makes 20 separate requests to fetch a <2MB JavaScript file. This is a static/cached resource and should be expected to download instantly. The script considers any response taking longer than 5 seconds to be a failure. This limitation is for example purposes.

#!/bin/bash
echo "testing 20 connections to discord.com using 5 second timeout .."
passed=0
failed=0
for i in $(seq 1 20); do
  echo -n "connection attempt #$i: " >&2
  started=$(date +%s%3N)
  if curl -fsSL -m 5 --no-keepalive -H 'accept-encoding: gzip' -o /dev/null \
    --resolve 'discord.com:443:162.159.137.232' \
    'https://discord.com/assets/488e72c5a60aaf2af286.js' \
  ; then
    finished=$(date +%s%3N)
    elapsed=$((finished - started))
    echo "ok ($elapsed ms)" >&2
    passed=$((passed + 1))
  else
    failed=$((failed + 1))
  fi
done
echo ""
echo "passed: $passed"
echo "failed: $failed"

The results are unsurprising:

testing 20 connections to discord.com using 5 second timeout ..
connection attempt #1: curl: (28) Operation timed out after 5000 milliseconds with 1331853 bytes received
connection attempt #2: ok (388 ms)
connection attempt #3: ok (412 ms)
connection attempt #4: ok (2240 ms)
connection attempt #5: curl: (28) Operation timed out after 5000 milliseconds with 1495525 bytes received
connection attempt #6: ok (463 ms)
connection attempt #7: ok (3487 ms)
connection attempt #8: curl: (28) Operation timed out after 5000 milliseconds with 1675574 bytes received
connection attempt #9: ok (354 ms)
connection attempt #10: curl: (28) Operation timed out after 5000 milliseconds with 1516878 bytes received
connection attempt #11: curl: (28) Operation timed out after 5001 milliseconds with 1266367 bytes received
connection attempt #12: ok (384 ms)
connection attempt #13: curl: (28) Operation timed out after 5001 milliseconds with 0 bytes received
connection attempt #14: ok (418 ms)
connection attempt #15: ok (2006 ms)
connection attempt #16: ok (444 ms)
connection attempt #17: curl: (28) Operation timed out after 5000 milliseconds with 1393741 bytes received
connection attempt #18: curl: (28) Operation timed out after 5000 milliseconds with 1865060 bytes received
connection attempt #19: ok (529 ms)
connection attempt #20: curl: (28) Operation timed out after 5000 milliseconds with 415370 bytes received

passed: 11
failed: 9

I repeated the same test using a VPN connection and everything works:

testing 20 connections to discord.com using 5 second timeout ..
connection attempt #1: ok (387 ms)
connection attempt #2: ok (436 ms)
connection attempt #3: ok (414 ms)
connection attempt #4: ok (444 ms)
connection attempt #5: ok (463 ms)
connection attempt #6: ok (481 ms)
connection attempt #7: ok (382 ms)
connection attempt #8: ok (389 ms)
connection attempt #9: ok (419 ms)
connection attempt #10: ok (409 ms)
connection attempt #11: ok (478 ms)
connection attempt #12: ok (420 ms)
connection attempt #13: ok (403 ms)
connection attempt #14: ok (412 ms)
connection attempt #15: ok (354 ms)
connection attempt #16: ok (485 ms)
connection attempt #17: ok (366 ms)
connection attempt #18: ok (454 ms)
connection attempt #19: ok (387 ms)
connection attempt #20: ok (413 ms)

passed: 20
failed: 0

The same test was performed using a 5MB file from a non Cloudflare protected site and that works ok without using a VPN.

pryswilliams · ‎24-08-2023

Also seeing the issue here (FTTC, Fibre Plus Broadband) where I'm having issues downloading docker images which are distributed via CloudFlare. e.g. >docker pull postgres starts off fine but slows down to a trickle.

Speed test on https://speedof.me/ is absolutely fine. Cloudflare speed test dramatically worse.

Here are my Cloudflare speed test results comparing vpn with a direct connection, showing up to 300% greater latency, down to 3% throughput and around 3.5k % greater download time when connecting directly, vs a VPN connection...

time	direction	bytes	latency			bps			duration			serverTime			responseSize
			vpn	direct	diff(%)	vpn	direct	diff(%)	vpn	direct	diff(%)	vpn	direct	diff(%)	vpn	direct	diff(%)
1.69286E+12	download	100000	28.40	20.00	70%	18031404.35	12264737.87	68%	44.50	66.00	148%	71.00	65.00	92%	100300	101184	101%
1.69286E+12	download	100000	59.40	43.00	72%	12537481.00	2483091.67	20%	64.00	326.00	509%	77.00	73.00	95%	100300	101186	101%
1.69286E+12	download	100000	49.00	22.00	45%	12817894.45	7782753.37	61%	62.60	104.00	166%	59.00	72.00	122%	100300	101176	101%
1.69286E+12	download	100000	21.80	23.00	106%	24613414.64	7636975.51	31%	32.60	106.00	325%	64.00	57.00	89%	100300	101190	101%
1.69286E+12	download	100000	21.30	17.00	80%	25074947.50	784372.12	3%	32.00	1032.00	3225%	70.00	62.00	89%	100300	101184	101%
1.69286E+12	download	100000	20.00	16.00	80%	22860443.80	2387917.21	10%	35.10	339.00	966%	78.00	53.00	68%	100300	101188	101%
1.69286E+12	download	100000	23.40	39.00	167%	24023977.99	1318358.03	5%	33.40	614.00	1838%	62.00	61.00	98%	100300	101184	101%
1.69286E+12	download	100000	25.60	30.00	117%	21863738.77	7495255.93	34%	36.70	108.00	294%	63.00	50.00	79%	100300	101186	101%
1.69286E+12	download	100000	21.20	18.00	85%	25967485.43	8014431.87	31%	30.90	101.00	327%	65.00	63.00	97%	100300	101182	101%
1.69286E+12	download	100000	21.00	39.00	186%	23057440.79	1036435.29	4%	34.80	781.00	2244%	73.00	63.00	86%	100300	101182	101%
1.69286E+12	download	1000000	19.50	21.00	108%	51795434.05	7009013.89	14%	154.50	1143.00	740%	110.00	68.00	62%	1000300	1001413	100%
1.69286E+12	download	1000000	23.50	22.00	94%	49397504.64	5019568.76	10%	162.00	1596.00	985%	90.00	106.00	118%	1000300	1001404	100%
1.69286E+12	download	1000000	24.70	94.00	381%	49367093.92	3938705.55	8%	162.10	2034.00	1255%	74.00	69.00	93%	1000300	1001416	100%
1.69286E+12	download	1000000	23.40	20.00	85%	50171831.19	2271441.96	5%	159.50	3527.00	2211%	77.00	67.00	87%	1000300	1001422	100%
1.69286E+12	download	1000000	18.60	15.00	81%	51862570.02	1827396.01	4%	154.30	4384.00	2841%	110.00	65.00	59%	1000300	1001413	100%
1.69286E+12	download	1000000	21.10	61.00	289%	51462326.47	1559537.46	3%	155.50	5137.00	3304%	68.00	63.00	93%	1000300	1001418	100%
1.69286E+12	download	1000000	20.30	19.00	94%	50808851.47	1379845.69	3%	157.50	5806.00	3686%	97.00	65.00	67%	1000300	1001423	100%
1.69286E+12	download	1000000	20.80	63.00	303%	51198992.05	1638625.45	3%	156.30	4889.00	3128%	75.00	77.00	103%	1000300	1001405	100%

lms-dev · ‎24-08-2023

I can also confirm that connections to Cloudflare are being throttled.

This is especially obvious when using a browser with the Secure DNS Over HTTPS set to cloudflare, as it effects ALL sites.

Using a VPN instantly fixes the issue.

bobpullen · ‎25-08-2023

Following this thread with interest.

For those experiencing problems, I wonder if there's any commonality in the public IP addresses you're currently assigned. Perhaps people can check and post the first two or three octects of their IP addresses to see if there's a pattern?

On a similar note, if you power your router/modem off for a period of time, e.g. an hour, and then power back up and reconnect, then you may get assigned an IP from a different range. I also wonder if this will lend any clues.

I only have access to a Plusnet connection at the moment, so unable to try replicating. No sign of a problem there: -

~$ mtr discord.com -c 50 -r
Start: 2023-08-25T08:30:47+0100
HOST: penguin                     Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway.lxd               0.0%    50    0.2   0.2   0.1   0.5   0.1
  2.|-- 100.115.92.25              0.0%    50    0.8   0.9   0.3   1.3   0.2
  3.|-- home.gateway               0.0%    50    5.1   6.0   4.1   7.9   0.9
  4.|-- 250.core.plus.net          0.0%    50   13.6  13.7  12.2  23.1   1.5
  5.|-- 84.93.253.83               0.0%    50   14.9  15.4  12.8  27.4   3.0
  6.|-- core1-BE1.colindale.ukcor  0.0%    50   15.1  14.0  11.3  15.8   0.9
  7.|-- peer8-et-3-0-1.telehouse.  0.0%    50   13.5  15.4  12.7  26.0   3.0
  8.|-- 195.99.126.233             0.0%    50   32.4  18.0  13.9  46.1   6.7
  9.|-- 172.71.176.4               0.0%    50   14.9  16.9  13.5  53.7   6.6
 10.|-- 162.159.136.232            0.0%    50   15.2  15.3  13.2  23.8   2.0

Using @andyxl987's script: -

$ sh cloudflare-test.sh
testing 20 connections to discord.com using 5 second timeout ..
connection attempt #1: ok (429 ms)
connection attempt #2: ok (392 ms)
connection attempt #3: ok (406 ms)
connection attempt #4: ok (391 ms)
connection attempt #5: ok (396 ms)
connection attempt #6: ok (408 ms)
connection attempt #7: ok (354 ms)
connection attempt #8: ok (404 ms)
connection attempt #9: ok (392 ms)
connection attempt #10: ok (379 ms)
connection attempt #11: ok (359 ms)
connection attempt #12: ok (303 ms)
connection attempt #13: ok (380 ms)
connection attempt #14: ok (465 ms)
connection attempt #15: ok (359 ms)
connection attempt #16: ok (722 ms)
connection attempt #17: ok (545 ms)
connection attempt #18: ok (436 ms)
connection attempt #19: ok (363 ms)
connection attempt #20: ok (371 ms)

passed: 20
failed: 0

RukaziSintakumo · ‎25-08-2023

Hey all!

Just want to come on here as well to say that I have the same issue happening recently. (roughly started a week ago)
I'm on EE Full Fibre Gigabit and have the exact same issue happening, with Discord having strange slow speeds happening intermittently. I also noticed that my latency towards US servers seems to have increased tremendously (previously around 120ms, now almost 500ms, making some online games for me impossible to play).

I was on the phone with a lovely gentleman from EE Tech, that did the usual things (router restart, line check, etc) though nothing helped.

andyxl987 · ‎25-08-2023

MTR shows last hop packet loss when the following route is used: 87.237.21.10 -> 172.71.176.4 -> Cloudflare IP.

No last hop packet loss is observed when the route is 87.237.21.10 -> 141.101.71.47 -> Cloudflare IP.

Those of you also experiencing this issue, are you based in (North) London?

RukaziSintakumo · ‎25-08-2023

I‘m located in the North-East (Newcastle)

andyxl987 · ‎25-08-2023

@bobpullen wrote:
Following this thread with interest.
For those experiencing problems, I wonder if there's any commonality in the public IP addresses you're currently assigned. Perhaps people can check and post the first two or three octects of their IP addresses to see if there's a pattern?
On a similar note, if you power your router/modem off for a period of time, e.g. an hour, and then power back up and reconnect, then you may get assigned an IP from a different range. I also wonder if this will lend any clues.

I had IP addresses in the 2.28.x.x range between March and July. On or before 14th August I was assigned an address in the 2.24.x.x range. On or before 20th August I have had addresses in the 2.26.x.x range which coincides with noticing the issue.