topic Re: Has something gone wrong with EE Mobile Broadband routing? in Mobile Data

Has something gone wrong with EE Mobile Broadband routing?

dave_101 — Mon, 31 Oct 2022 15:02:54 GMT

I'm trying to understand what's happened with EE's Mobile broadband performance, because it's fallen off a cliff in the past few days.

The culprit appears to be in the way the routing is configured, as whenever I run a traceroute, it flounders around at IP address 11.1.6.254.

In the past, traceroute was quick and logical - it flowed straight from my router to a logical IP address in the same space as my router's (DHCP granted) internet facing IP address. A quick search for 11.1.6.254 seems to imply that it is geolocated to Columbus, OH, and is run by the US Department of Defense.

So my question is, why would EE be forwarding all the traffic from the mobile broadband to a US DoD server?

Re: Has something gone wrong with EE Mobile Broadband routing?

XRaySpeX — Mon, 31 Oct 2022 16:38:02 GMT

What is the full traceroute?

Re: Has something gone wrong with EE Mobile Broadband routing?

dave_101 — Mon, 31 Oct 2022 19:58:19 GMT

So the beginning of the trace (the tail is not problematic) is

1  192.168.60.101 (192.168.60.101)  0.987 ms  0.248 ms  0.214 ms
2  10.10.20.101 (10.10.20.101)  1.246 ms  0.847 ms  0.907 ms
3  11.1.6.254 (11.1.6.254)  58.483 ms  31.277 ms  40.039 ms
4  * * *
5  * * *
6  * * *
7  109.249.132.36 (109.249.132.36)  41.972 ms  43.030 ms
   109.249.132.6 (109.249.132.6)  40.138 ms
8  peer2-et4-0-6.slough.ukcore.bt.net (62.172.103.151)  52.624 ms
   peer8-et-0-1-4.telehouse.ukcore.bt.net (109.159.252.106)  26.451 ms
   peer3-et3-1-4.slough.ukcore.bt.net (62.172.103.232)  23.324 ms

It is hop 3 that is definitely the problem child, and somewhat bizarre. The external IP address that my router has is in the private 10.157.x.x address space (I'm somewhat surprised it's not in the CGNAT range), so I'm intrigued that the next hop is to 11.1.6.254, especially as this is a public address apparently hosted in the 'States.

BTW, don't be concerned about the two private address at the start of the trace. The first is my internal firewall, and the second is the 4G router. NAT only occurs on the 4G router.

Re: Has something gone wrong with EE Mobile Broadband routing?

XRaySpeX — Mon, 31 Oct 2022 21:17:02 GMT

I would agree with your findings but am puzzled how a US DoD hop can come before an EE/Orange node & other UK sites. As it comes logically (as opposed to timewise) nearer to you than them I wonder if it's to do with your setup. It's like it's leaving you to go to the US DoD before coming back to the EE network & thence onto the BT infrastructure (BT Customer Loopbacks). Are you running some sort of VPN?

Or are you being spied on by the CIA or US Military 😉?

Re: Has something gone wrong with EE Mobile Broadband routing?

Mustrum — Mon, 31 Oct 2022 21:51:46 GMT

Is the test being done from an apple device?

Re: Has something gone wrong with EE Mobile Broadband routing?

dave_101 — Tue, 01 Nov 2022 09:05:40 GMT

I see the same behaviour when running tracert on Windows.

Re: Has something gone wrong with EE Mobile Broadband routing?

dave_101 — Tue, 01 Nov 2022 09:10:27 GMT

No VPN involved.

I noticed another post or two on the forums where people where showing things related to this 11.1.6.254 address, so I don't believe it's just me.

Just to add that I also have Starlink, and when I run a traceroute from the same devices, but through the Starlink route, I don't see any illogical behaviour. That is, the hops are all logical up from my firewall, to the Starlink router, through the Starlink CGNAT and then out to the public internet.

It is only with EE that the problem exists.

I have to say that it feels like a router configuration issue within the EE network, as though someone has set something up incorrectly in a routing table.

Re: Has something gone wrong with EE Mobile Broadband routing?

XRaySpeX — Tue, 01 Nov 2022 09:47:42 GMT

Does your router tell you what the Gateway IP to the Internet is?

Re: Has something gone wrong with EE Mobile Broadband routing?

dave_101 — Tue, 01 Nov 2022 12:44:12 GMT

The routing table shows me that the gateway for my router is in a 10.x.x.x network (it changes, because the router is set to reboot each night). So the next hop should not be 11.1.6.254, unless somewhere in the EE network, something strange is going on.

Re: Has something gone wrong with EE Mobile Broadband routing?

dave_101 — Thu, 08 Dec 2022 14:44:25 GMT

I just want to draw a line under this topic, at least from my perspective.

For the last three days, the EE mast in the Taunton area would appear to have been offline, as there's been no service.

It has just come back online.

I tried the traceroute again, and voila!, it no longer goes via a US DoD server. Throughput and general performance is back where it was prior to the sudden introduction of this rogue 11.1.6.254 address.

I'd love to know what was actually going on. It is easy to imagine some wonderful conspiracy theories - for example, was there a rogue terror cell operating in Somerset, which were being tracked by the CIA and who in turn had hacked into the mast?

Personally, I'm sticking with a simpler answer. Either some dodgy/failing kit, or some misconfiguration, which has now been fixed. But (and I know this will never happen) it would be nice for EE to come clean and actually say what went wrong.

Re: Has something gone wrong with EE Mobile Broadband routing?

dave_101 — Thu, 08 Dec 2022 15:00:38 GMT

Spoke too soon.

After maybe ten minutes of stability, the mast is back offline. Then, when it comes back online, the performance has dropped off a cliff, and we're back to routing via the US DoD server again.

I have absolutely no idea what EE are doing, but clearly they are wrecking their customers' internet connections in the South West of England.

Re: Has something gone wrong with EE Mobile Broadband routing?

dave_101 — Thu, 08 Dec 2022 15:30:18 GMT

There are some extremely odd things happening with the Culm Davy mast, by the looks of it.

Allegedly, it's been being worked on (by 3rd party engineers) since the 29th of November. For the past three days, there has been no coverage. Now, the coverage is random and sporadic. One moment you can connect to 3G, then to 4G, then nothing, then back to 3G, then nothing, and so on and so on.

What's particularly worrying is that, for the past month or so, any internet traffic going through that mast has, apparently, been forward to a server in the US, which (according to whois) is run by the US Department of Defense.

Doing some traceroutes today has been revealing.

Connect via 3G, and the traceroute shows normal flow through the EE system and out to the outside internet. Connect (when possible) via 4G, and lo and behold the US DoD server is back in the routing chain.

Back to 3G, and no US server.

Back to 4G and back to the US again.

EE has admitted that 3rd party engineers are working on the mast. But can they say what they're doing?

Has the EE network been hacked? Or are EE doing something that would deliberately route traffic to the US, before coming back to the EE/BT network?

I'd love a clear explanation of what on earth is going on, and some idea of when EE intends to put the mast back into proper commission, and without forwarding everything to US.

Re: Has something gone wrong with EE Mobile Broadband routing?

James_B — Thu, 08 Dec 2022 15:56:47 GMT

Hi @dave_101,

Our Technical Support Team will be happy to take a detailed look at this issue if you get in touch.

Thanks

James

Re: Has something gone wrong with EE Mobile Broadband routing?

dave_101 — Fri, 09 Dec 2022 16:28:55 GMT

@James_B

Aren't you even vaguely perturbed that EE are routing their customers through a US military server?

Does this not worry you at all?

This must violate just about every tenet of GDPR, and should make every EE customer terrified of why EE would forward their requests to a US MILITARY!! web server.

I'm not sure that this is something for EE technical support (who, by the way, would probably be much more useful if they actually answered their phones). Instead, this feels more like something that should be presented to BBC Panorama, so that they can do an exposé on the perils of what goes on under the covers of the UK's mobile networks.

Re: Has something gone wrong with EE Mobile Broadband routing?

James_B — Fri, 09 Dec 2022 17:03:12 GMT

Hi @dave_101,

Our Technical Support Team are best placed to look into this issue for you, as we can't access your account or the necessary diagnostic tools via the community.

If any routing problems are found, they'll make sure it is escalated to the right people.

James

Re: Has something gone wrong with EE Mobile Broadband routing?

XRaySpeX — Sun, 11 Dec 2022 06:46:20 GMT

@dave_101 wrote:

The 11.1.6.254 address in the route fascinates me. Yes, it's possible that EE are bouncing stuff to the US military, and an ARIN search would certainly confirm that the US DoD owns 11.0.0.0/8. But I also wonder whether EE are simply nicking a chunk of a very large public IP range and are using it as a pseudo-private IP range? After all, most of the US military's systems are not publicly routable, so stealing their IP addresses shouldn't cause any issues! At least, you'd hope not.

I like that theory. The 11.1.6.254 IP is appearing immediately after it leaves your LAN & before the EE/Orange nodes & thence onto the BT infrastructure. It is as if it is acting as the WAN Gateway to the Net. This only need be a private IP addy (as it is with fixed BB using 172.16.0.0/12 range IPs for this purpose) & so needn't clash with the real US DoD IP.

Against this is the 30 -50 ms delay compared with fixed BB's 4 -5 ms hop.

Re: Has something gone wrong with EE Mobile Broadband routing?

chistery — Sun, 11 Dec 2022 10:30:43 GMT

This is all good tin foil hat stuff isn't it!

We know all internet data is logged and given to the UK and US agencies and by giving data to foreign agencies it avoids our pesky laws that limit tracking. BT was rumoured to be one of the two unknown ISPs helping build the system. Still, the IPA act means no one will admit to anything and very few people at the unnamed ISPs will even know of its existence. Maybe a misconfigured router has shown something it shouldn't have!

Re: Has something gone wrong with EE Mobile Broadband routing?

jamesharrison79 — Wed, 15 Mar 2023 21:38:28 GMT

@dave_101

Interested to know if you got a resolution. Over the border in Devon my son is giving me a hard time about high ping rates for his gaming. I started doing some research and came across your post. We are also using EE and when trace routing im finding the same issue, stalling at jump 3 (see below) Did EE reveal why our data is heading Stateside?

traceroute to google.co.uk (216.58.209.67), 64 hops max, 52 byte packets

 1  192.168.1.1 (192.168.1.1)  17.422 ms  12.735 ms  18.456 ms

 2  11.1.6.254 (11.1.6.254)  31.690 ms  39.563 ms  43.776 ms

 3  * * *

 4  * * *

 5  * * *

 6  109.249.132.36 (109.249.132.36)  51.078 ms

    109.249.132.38 (109.249.132.38)  46.374 ms

    109.249.132.36 (109.249.132.36)  80.394 ms

 7  core2-pos14-0.birmingham.ukcore.bt.net (62.6.204.137)  41.895 ms

    62.6.204.175 (62.6.204.175)  49.428 ms  47.772 ms

 8  peer5-te0-0-0-32.telehouse.ukcore.bt.net (195.99.126.77)  38.531 ms  38.970 ms

    109.159.253.75 (109.159.253.75)  40.496 ms

 9  * * *

10  108.170.246.161 (108.170.246.161)  53.122 ms

Re: Has something gone wrong with EE Mobile Broadband routing?

XGSIsOn — Tue, 20 Jun 2023 10:32:11 GMT

Steered here by another site.

Someone working on the mast or network decided to use a usually unroutable IP address in the 11/8 range as an internal address or, more likely, someone typo'd 11 instead of 10 when entering 10.1.6.254 as an IP address for a loopback interface.

If the former the US DoD don't advertise most of their ranges onto the public Internet so they are for practical purposes private addresses. They just freak people out and shouldn't be used.

Just to reassure if anything were actually routing via that 11.x range publicly it would be very, very much noticed by the wider Internet. For that to be used for anything other than an internal network you'd have EE advertising US DoD IP ranges to the wider Internet: that doesn't happen.

If the DoD wanted to do such things they wouldn't do them themselves, GCHQ would do it for them, and you wouldn't see a new hop routing things, they'd silently tap the data off.

Boring as my explanation is relative to the exciting idea that random EE users' data is being unsubtlely bundled off to the USA and back at superluminal speeds hopefully it'll put that part of the topic to rest.

Chances are nothing more than that someone hit '11' instead of '10' when configuring something and that happily redistributed into the private part of the EE network or not given it's probably just a loopback and doesn't even need to be routable