topic Re: CloudFlare Throttling Issue (FTTH) in Broadband & Landline

CloudFlare Throttling Issue (FTTH)

andyxl987 — Sun, 20 Aug 2023 18:45:18 GMT

I have EE's Full Fibre package and hadn't had any problems in about 10 months of service. However, for the last 24 hours (maybe longer) I am experiencing extremely slow speeds to certain sites. For example, loading a discord channel may take 4 minutes to load. The emoji panel makes dozens of requests in a short frequency and also fails to load correctly.

I do not experience this issue while connected to a VPN.

I experienced something similar on Friday using 5G (also with discord) but put that down to signal issues.

Discord is not the only site that is affected. I host some public services on a remote server (Germany) and use CloudFlare proxy in front of these. Speed issues are also apparent for these, which again, do not reproduce when using a VPN. In fact, if I edit /etc/hosts to bypass CloudFlare's proxy, everything loads instantly. Note, that Discord also uses CloudFlare.

While it's possible the issue is my end, e.g. my router, Friday's issue while on mobile has me wondering if there's some issue between EE and CloudFlare? There appears to be severe throttling where the first several requests are fine and any subsequent requests are exponentially slower to send/receive.

I've also encountered similar while pulling images from docker hub and github container registry (although I do not believe either use CloudFlare).

Re: CloudFlare Throttling Issue (FTTH)

andyxl987 — Sun, 20 Aug 2023 19:17:28 GMT

CloudFlare's speed tool also shows the difference between using a VPN and direct access, although the problems are less severe than real world usage:

Direct Connection

time	direction	bytes	latency	bps	duration	serverTime	responseSize
1692558043768	download	1000000	106.700031999255	609265.672884655	13134.500032	108.999968	1000300
1692558083014	download	1000000	219.199941001118	647380.51630867	12361.1999410011	88.000059	1000300
1692558055147	download	1000000	180.100031999627	710282.694472192	11266.500032	108.999968	1000300
1692558093402	download	1000000	424.000168	778186.190293525	10283.4001680004	100.999832	1000300
1692558070558	download	1000000	186.600094999627	849746.205882069	9417.40009500037	99.999905	1000300
1692558101508	download	1000000	213.199981001118	1000775.37067777	7996.19998100112	105.000019	1000300
1692558109082	download	1000000	14.4000339985099	1069410.65931311	7483.000034	85.999966	1000300
1692558061036	download	1000000	158.900044000373	1382441.33973208	5788.60004399963	95.999956	1000300

VPN Connection

time	direction	bytes	latency	bps	duration	serverTime	responseSize
1692557984211	download	100000000	6.000034	351494898.088389	2276.000034	85.999966	100000300
1692557986804	download	100000000	7.500097	333737586.094689	2397.10009699963	76.999903	100000300
1692557990090	download	100000000	5.70008199925493	264070763.606601	3029.500082	135.999918	100000300

The above results show that a VPN connection was able to download a 100 MB file in at most 3 seconds. The direct connection did not even progress to that part of the test. The quickest it managed to download a 10 MB file was nearly 6 seconds, i.e. it is taking twice as long to download 10x less data.

Results have been truncated to fit 20,000 character limit.

Re: CloudFlare Throttling Issue (FTTH)

Discobarry — Sun, 20 Aug 2023 21:56:42 GMT

I’ve been having speed issues for the last few days and this sounds like it could be the same issues I’m having. Not all websites are slow but many are. There is definitely an issue some where.

Re: CloudFlare Throttling Issue (FTTH)

Akiyama8641 — Mon, 21 Aug 2023 19:50:19 GMT

Having the same issue here as well, sporadically, follows no real pattern, and can last anywhere from a few seconds to several minutes. Time of day doesn't seem to matter. It's been going on for about a week now. Only CloudFlare hosted sites/services are effected. It also happened about 3 weeks ago now and lasted around 5 days before being rectified.

Re: CloudFlare Throttling Issue (FTTH)

Christopher_G — Tue, 22 Aug 2023 15:05:26 GMT

Hi all.

EE does not employ any traffic management or throttling on our broadband network. Please get in touch with our Customer Care team who can look into the issues you are experiencing. They’ll be happy to help.

Chris

Re: CloudFlare Throttling Issue (FTTH)

jackdowsonuk — Tue, 22 Aug 2023 16:57:44 GMT

As per my post which is pretty much the same, I refer to this:

"When I spoke to technical support and explained the 2.5-5% packet loss on the affected websites, I had to explain to the agent what packet loss even was, as she didn't have a clue (in her defence she said that packet loss isn't something they're trained on). She started to guide me down the route of "well do you have wi-fi signal problems and how many devices do you have connected to the wi-fi", even though I explained twice that it was using an ethernet connection. She also explained to me that because the connection between EE and my property doesn't show any faults, and is fast, there is nothing they can do. I asked if this could be escalated to Openreach and she outright refused."

I am concerned that others in this thread will simply receive the same response. Mobile tech support has been brilliant, broadband support not so much unfortunately.

Edit: I'm literally on the phone to EE again right now, let's see what they say

So, still on the call. Being asked questions like "how many devices are connected to your WiFi" and "maybe someone could have turned on an electrical device and affected your speeds", this is immediately after telling them the issue was occurring in the same way when only the PC was connected directly to the ONT terminal model with an ethernet cable and nothing else. This was also after suggesting that this is out of their remit at their level so would need escalating higher.

This isn't looking positive - I believe these may be questions their manager may be asking to relay to me.

Edit 2: The agent will be writing up notes to pass to their manager. They offered to send an Openreach engineer out tomorrow but I explained to her that it would be a waste of their and my time, as the issue isn't between point A to B, but possibly further up the chain like F to G, and that there must be a process in place to have that investigated. Let's see what they say when they get back to me.

Re: CloudFlare Throttling Issue (FTTH)

andyxl987 — Tue, 22 Aug 2023 18:07:50 GMT

I spoke with a customer support agent who while sympathetic was unable to help as this is so far an isolated incident and hopefully may resolve itself given some time.

In the meantime, it may be useful if those of use affected can provide some additional diagnostics. The tables in my previous post were sampled from the CSV output of Cloudflare's Speed Tool. You can generate results using https://speed.cloudflare.com/. Here are a couple of screenshots comparing a direct connection and one using a VPN:

Cloudflare Speed Test Direct Connection:

Cloudflare Speed Test VPN Connection:

Additionally, here are mtr results while using a direct connection:

MTR Direct Connection:

**redacted** (172.23.223.117) -> discord.com (162.159.137.232)                              2023-08-22T18:49:36+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                               Packets               Pings
 Host                                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. **redacted**                                                              0.0%    52    0.2   0.2   0.2   0.7   0.1
 2. **redacted**                                                              0.0%    52    6.2   3.3   1.1  11.9   2.0
 3. 172.16.11.38                                                              0.0%    52    5.8   5.0   2.8  14.5   2.3
 4. (waiting for reply)
 5. 213.121.98.128                                                            0.0%    52    5.3   6.3   3.5  37.6   5.7
 6. 87.237.20.130                                                             0.0%    51    5.6   5.9   3.7  21.1   2.8
 7. 87.237.21.10                                                              5.9%    51    7.1   8.7   4.6  32.1   5.7
 8. 172.71.176.4                                                              3.9%    51   81.9  11.3   4.9  81.9  14.5
 9. 162.159.137.232                                                           2.0%    51    5.7   5.9   4.2  11.1   1.6

Edit: I believe it is not only Cloudflare related services experience connectivity issue. As mentioned in my initial post, pulling docker images from docker hub is also affected.

Re: CloudFlare Throttling Issue (FTTH)

jackdowsonuk — Tue, 22 Aug 2023 18:06:44 GMT

Unfortunately a couple of your images aren't loading!

Jack

Re: CloudFlare Throttling Issue (FTTH)

XRaySpeX — Tue, 22 Aug 2023 18:12:09 GMT

Don't worry, your images were awaiting approval. I have now done so & they should appear soon.

Re: CloudFlare Throttling Issue (FTTH)

Mustrum — Tue, 22 Aug 2023 20:07:02 GMT

@andyxl987 do you get the same speed results when using more familiar (to me) speed tests?

https://www.speedtest.net/ and/or https://www.thinkbroadband.com/speedtest

Re: CloudFlare Throttling Issue (FTTH)

andyxl987 — Tue, 22 Aug 2023 20:53:40 GMT

Other speed test sites are OK. The following are taken using a WiFi connection (and other devices are in use):

fast.com (wifi):
620 download
94 upload

speedtest.net (wifi):
536.19 download
104.74 upload

speedtest.net (router inbuilt):
916.83 download
111.42 upload

think broadband (wifi):

Under ideal (WiFi) conditions I can get over 700 Mbps from fast.com and Steam if no other devices are in use. But trying to load a few megabytes channel from discord or install packages from npm (node package repository) takes longer than to download a small game 😞

Re: CloudFlare Throttling Issue (FTTH)

Mustrum — Tue, 22 Aug 2023 21:19:38 GMT

Thanks @andyxl987 alas though that will make it harder to get an answer.

Unlike some other forum's there is no direct access on here to the Broadband Technical Care team, and as discovered getting beyond first line is not simple. From your and other posts many first line teams would struggle to understand the issues.

Besides it would appear the issues are outside the EE network so would need more detailed internet investigation.

You could try looking on the BT Forum to see if there are any similar issues being reported, or even on more specialist forums such as Kitz, Thinkbroadband or ISPReviews.

HTH,

Re: CloudFlare Throttling Issue (FTTH)

andyxl987 — Tue, 22 Aug 2023 21:22:29 GMT

Here's a small example showing the instability with an example Cloudflare protected host. This makes 20 separate requests to fetch a <2MB JavaScript file. This is a static/cached resource and should be expected to download instantly. The script considers any response taking longer than 5 seconds to be a failure. This limitation is for example purposes.

#!/bin/bash
echo "testing 20 connections to discord.com using 5 second timeout .."
passed=0
failed=0
for i in $(seq 1 20); do
  echo -n "connection attempt #$i: " >&2
  started=$(date +%s%3N)
  if curl -fsSL -m 5 --no-keepalive -H 'accept-encoding: gzip' -o /dev/null \
    --resolve 'discord.com:443:162.159.137.232' \
    'https://discord.com/assets/488e72c5a60aaf2af286.js' \
  ; then
    finished=$(date +%s%3N)
    elapsed=$((finished - started))
    echo "ok ($elapsed ms)" >&2
    passed=$((passed + 1))
  else
    failed=$((failed + 1))
  fi
done
echo ""
echo "passed: $passed"
echo "failed: $failed"

The results are unsurprising:

testing 20 connections to discord.com using 5 second timeout ..
connection attempt #1: curl: (28) Operation timed out after 5000 milliseconds with 1331853 bytes received
connection attempt #2: ok (388 ms)
connection attempt #3: ok (412 ms)
connection attempt #4: ok (2240 ms)
connection attempt #5: curl: (28) Operation timed out after 5000 milliseconds with 1495525 bytes received
connection attempt #6: ok (463 ms)
connection attempt #7: ok (3487 ms)
connection attempt #8: curl: (28) Operation timed out after 5000 milliseconds with 1675574 bytes received
connection attempt #9: ok (354 ms)
connection attempt #10: curl: (28) Operation timed out after 5000 milliseconds with 1516878 bytes received
connection attempt #11: curl: (28) Operation timed out after 5001 milliseconds with 1266367 bytes received
connection attempt #12: ok (384 ms)
connection attempt #13: curl: (28) Operation timed out after 5001 milliseconds with 0 bytes received
connection attempt #14: ok (418 ms)
connection attempt #15: ok (2006 ms)
connection attempt #16: ok (444 ms)
connection attempt #17: curl: (28) Operation timed out after 5000 milliseconds with 1393741 bytes received
connection attempt #18: curl: (28) Operation timed out after 5000 milliseconds with 1865060 bytes received
connection attempt #19: ok (529 ms)
connection attempt #20: curl: (28) Operation timed out after 5000 milliseconds with 415370 bytes received

passed: 11
failed: 9

I repeated the same test using a VPN connection and everything works:

testing 20 connections to discord.com using 5 second timeout ..
connection attempt #1: ok (387 ms)
connection attempt #2: ok (436 ms)
connection attempt #3: ok (414 ms)
connection attempt #4: ok (444 ms)
connection attempt #5: ok (463 ms)
connection attempt #6: ok (481 ms)
connection attempt #7: ok (382 ms)
connection attempt #8: ok (389 ms)
connection attempt #9: ok (419 ms)
connection attempt #10: ok (409 ms)
connection attempt #11: ok (478 ms)
connection attempt #12: ok (420 ms)
connection attempt #13: ok (403 ms)
connection attempt #14: ok (412 ms)
connection attempt #15: ok (354 ms)
connection attempt #16: ok (485 ms)
connection attempt #17: ok (366 ms)
connection attempt #18: ok (454 ms)
connection attempt #19: ok (387 ms)
connection attempt #20: ok (413 ms)

passed: 20
failed: 0

The same test was performed using a 5MB file from a non Cloudflare protected site and that works ok without using a VPN.

Re: CloudFlare Throttling Issue (FTTH)

pryswilliams — Thu, 24 Aug 2023 08:39:16 GMT

Also seeing the issue here (FTTC, Fibre Plus Broadband) where I'm having issues downloading docker images which are distributed via CloudFlare. e.g. >docker pull postgres starts off fine but slows down to a trickle.

Speed test on https://speedof.me/ is absolutely fine. Cloudflare speed test dramatically worse.

Here are my Cloudflare speed test results comparing vpn with a direct connection, showing up to 300% greater latency, down to 3% throughput and around 3.5k % greater download time when connecting directly, vs a VPN connection...

time	direction	bytes	latency			bps			duration			serverTime			responseSize
			vpn	direct	diff(%)	vpn	direct	diff(%)	vpn	direct	diff(%)	vpn	direct	diff(%)	vpn	direct	diff(%)
1.69286E+12	download	100000	28.40	20.00	70%	18031404.35	12264737.87	68%	44.50	66.00	148%	71.00	65.00	92%	100300	101184	101%
1.69286E+12	download	100000	59.40	43.00	72%	12537481.00	2483091.67	20%	64.00	326.00	509%	77.00	73.00	95%	100300	101186	101%
1.69286E+12	download	100000	49.00	22.00	45%	12817894.45	7782753.37	61%	62.60	104.00	166%	59.00	72.00	122%	100300	101176	101%
1.69286E+12	download	100000	21.80	23.00	106%	24613414.64	7636975.51	31%	32.60	106.00	325%	64.00	57.00	89%	100300	101190	101%
1.69286E+12	download	100000	21.30	17.00	80%	25074947.50	784372.12	3%	32.00	1032.00	3225%	70.00	62.00	89%	100300	101184	101%
1.69286E+12	download	100000	20.00	16.00	80%	22860443.80	2387917.21	10%	35.10	339.00	966%	78.00	53.00	68%	100300	101188	101%
1.69286E+12	download	100000	23.40	39.00	167%	24023977.99	1318358.03	5%	33.40	614.00	1838%	62.00	61.00	98%	100300	101184	101%
1.69286E+12	download	100000	25.60	30.00	117%	21863738.77	7495255.93	34%	36.70	108.00	294%	63.00	50.00	79%	100300	101186	101%
1.69286E+12	download	100000	21.20	18.00	85%	25967485.43	8014431.87	31%	30.90	101.00	327%	65.00	63.00	97%	100300	101182	101%
1.69286E+12	download	100000	21.00	39.00	186%	23057440.79	1036435.29	4%	34.80	781.00	2244%	73.00	63.00	86%	100300	101182	101%
1.69286E+12	download	1000000	19.50	21.00	108%	51795434.05	7009013.89	14%	154.50	1143.00	740%	110.00	68.00	62%	1000300	1001413	100%
1.69286E+12	download	1000000	23.50	22.00	94%	49397504.64	5019568.76	10%	162.00	1596.00	985%	90.00	106.00	118%	1000300	1001404	100%
1.69286E+12	download	1000000	24.70	94.00	381%	49367093.92	3938705.55	8%	162.10	2034.00	1255%	74.00	69.00	93%	1000300	1001416	100%
1.69286E+12	download	1000000	23.40	20.00	85%	50171831.19	2271441.96	5%	159.50	3527.00	2211%	77.00	67.00	87%	1000300	1001422	100%
1.69286E+12	download	1000000	18.60	15.00	81%	51862570.02	1827396.01	4%	154.30	4384.00	2841%	110.00	65.00	59%	1000300	1001413	100%
1.69286E+12	download	1000000	21.10	61.00	289%	51462326.47	1559537.46	3%	155.50	5137.00	3304%	68.00	63.00	93%	1000300	1001418	100%
1.69286E+12	download	1000000	20.30	19.00	94%	50808851.47	1379845.69	3%	157.50	5806.00	3686%	97.00	65.00	67%	1000300	1001423	100%
1.69286E+12	download	1000000	20.80	63.00	303%	51198992.05	1638625.45	3%	156.30	4889.00	3128%	75.00	77.00	103%	1000300	1001405	100%

Re: CloudFlare Throttling Issue (FTTH)

lms-dev — Thu, 24 Aug 2023 10:33:42 GMT

I can also confirm that connections to Cloudflare are being throttled.

This is especially obvious when using a browser with the Secure DNS Over HTTPS set to cloudflare, as it effects ALL sites.

Using a VPN instantly fixes the issue.

Re: CloudFlare Throttling Issue (FTTH)

bobpullen — Fri, 25 Aug 2023 07:42:36 GMT

Following this thread with interest.

For those experiencing problems, I wonder if there's any commonality in the public IP addresses you're currently assigned. Perhaps people can check and post the first two or three octects of their IP addresses to see if there's a pattern?

On a similar note, if you power your router/modem off for a period of time, e.g. an hour, and then power back up and reconnect, then you may get assigned an IP from a different range. I also wonder if this will lend any clues.

I only have access to a Plusnet connection at the moment, so unable to try replicating. No sign of a problem there: -

~$ mtr discord.com -c 50 -r Start: 2023-08-25T08:30:47+0100 HOST: penguin Loss% Snt Last Avg Best Wrst StDev 1.|-- _gateway.lxd 0.0% 50 0.2 0.2 0.1 0.5 0.1 2.|-- 100.115.92.25 0.0% 50 0.8 0.9 0.3 1.3 0.2 3.|-- home.gateway 0.0% 50 5.1 6.0 4.1 7.9 0.9 4.|-- 250.core.plus.net 0.0% 50 13.6 13.7 12.2 23.1 1.5 5.|-- 84.93.253.83 0.0% 50 14.9 15.4 12.8 27.4 3.0 6.|-- core1-BE1.colindale.ukcor 0.0% 50 15.1 14.0 11.3 15.8 0.9 7.|-- peer8-et-3-0-1.telehouse. 0.0% 50 13.5 15.4 12.7 26.0 3.0 8.|-- 195.99.126.233 0.0% 50 32.4 18.0 13.9 46.1 6.7 9.|-- 172.71.176.4 0.0% 50 14.9 16.9 13.5 53.7 6.6 10.|-- 162.159.136.232 0.0% 50 15.2 15.3 13.2 23.8 2.0

Using @andyxl987's script: -

$ sh cloudflare-test.sh testing 20 connections to discord.com using 5 second timeout .. connection attempt #1: ok (429 ms) connection attempt #2: ok (392 ms) connection attempt #3: ok (406 ms) connection attempt #4: ok (391 ms) connection attempt #5: ok (396 ms) connection attempt #6: ok (408 ms) connection attempt #7: ok (354 ms) connection attempt #8: ok (404 ms) connection attempt #9: ok (392 ms) connection attempt #10: ok (379 ms) connection attempt #11: ok (359 ms) connection attempt #12: ok (303 ms) connection attempt #13: ok (380 ms) connection attempt #14: ok (465 ms) connection attempt #15: ok (359 ms) connection attempt #16: ok (722 ms) connection attempt #17: ok (545 ms) connection attempt #18: ok (436 ms) connection attempt #19: ok (363 ms) connection attempt #20: ok (371 ms) passed: 20 failed: 0

Re: CloudFlare Throttling Issue (FTTH)

RukaziSintakumo — Fri, 25 Aug 2023 16:47:45 GMT

Hey all!

Just want to come on here as well to say that I have the same issue happening recently. (roughly started a week ago)
I'm on EE Full Fibre Gigabit and have the exact same issue happening, with Discord having strange slow speeds happening intermittently. I also noticed that my latency towards US servers seems to have increased tremendously (previously around 120ms, now almost 500ms, making some online games for me impossible to play).

I was on the phone with a lovely gentleman from EE Tech, that did the usual things (router restart, line check, etc) though nothing helped.

Re: CloudFlare Throttling Issue (FTTH)

andyxl987 — Fri, 25 Aug 2023 18:24:15 GMT

MTR shows last hop packet loss when the following route is used: 87.237.21.10 -> 172.71.176.4 -> Cloudflare IP.

No last hop packet loss is observed when the route is 87.237.21.10 -> 141.101.71.47 -> Cloudflare IP.

Those of you also experiencing this issue, are you based in (North) London?

Re: CloudFlare Throttling Issue (FTTH)

RukaziSintakumo — Fri, 25 Aug 2023 18:32:29 GMT

I‘m located in the North-East (Newcastle)

Re: CloudFlare Throttling Issue (FTTH)

andyxl987 — Fri, 25 Aug 2023 18:48:45 GMT

@bobpullen wrote:
Following this thread with interest.
For those experiencing problems, I wonder if there's any commonality in the public IP addresses you're currently assigned. Perhaps people can check and post the first two or three octects of their IP addresses to see if there's a pattern?
On a similar note, if you power your router/modem off for a period of time, e.g. an hour, and then power back up and reconnect, then you may get assigned an IP from a different range. I also wonder if this will lend any clues.

I had IP addresses in the 2.28.x.x range between March and July. On or before 14th August I was assigned an address in the 2.24.x.x range. On or before 20th August I have had addresses in the 2.26.x.x range which coincides with noticing the issue.