EE network changes - email access failure

[Flyingfemme]

Yesterday I had two text messages to say that EE had updated my network settings and I should reboot my phone. Did it this morning and, instantly, I cannot send or receive emails on phone data. My colleague landed at Heathrow this morning and turned on his phone - same problem. Email access is fine over two different wifi providers at home and work and his US phone can access the email servers. PC and other devices also fine.

EE claim it is nothing to do with anything they have done.....yet the call centre lady said she has had "lots" of calls with the same problem this morning.

Anyone else got the same problem? If so, can you share a fix? If not I shall be moving my company account quick-smart. Can't operate without mobile emails.

[JohnMaz]

From what I can gather the issue is somewhere between Lumen and EE network, and so Giacom don;t have any control over that part of the link. They have packet captures and can show where it times out. I understand that information is with EE, as they are working with EE to resolve it, but EE/Lumen need to fix the issue. 

[Flyingfemme]

Looks like Giacom have done a nice job of identifying the problem, as far as they can, and presenting it. A shame that EE are still not giving out proper information. It takes computing back almost 30 years - every program/system/hardware provider blaming all the others. Users don't care!

[MozzaSec]

I dont agree, from talking to someone who is having the same problem on o2 with Giacom, it looks like they are passing the buck, instead of investigating the issue. 

Giacom need to try routing connections through their backup ISP, whoever that is and test from there as currently all connections route through KCOM, and from the PCAP's and the TCP Handshakes it looks like they are resetting the connecting and are suffering high packet loss coming back from there servers. 

Giacom also need to speak to KCOM and ask them why this failing, as they are their IP partner...

Not all traffic from EE routes via Lumens (L3), as with o2, i see data routed via other peers.

It is either a problem with there CAS server and EXC2013/2016 (both of which are in Extended support and a hackers dream)...

No offence but if you are paying for EXC2013 or 2016 then more foul you, as there are cheaper options in M365 for small businesses, even Exchange 2019 which is fully supported is better...

[MozzaSec]

Just to add, 

These are the results of a ping from Lumens Network and a traceroute which shows ICMP being blocked past KCOM's edge 

And a bit more digging shows that EE dont peer with L3 

The private peering also shows no links to L3 (Lumen)

[JohnMaz]

It's a customer of ours, but they still have their emails with Giacom - in fact we Always sell M365 for hosted exchange, as it happens using Giacom as the aggregator for licensing hence a way in with them.  Moving the customer from their current provider to M365 is of course one option, and has advantages as you say, although it's not totally correct to say they are unsupported as Exchange 2016 is still well inside support, and even 2013 is still supported. The latest vulnerabilities for exchange that were widespread about a year or so ago was the same for all three versions 2013, 2016 and 2019 (from memory) we had to update all servers for customers at the time as I recall.  

Interesting you think it's KCOM though.  Is it possible ICMP packets are not allowed for security?. Not all servers respond to pings.  I don't think I get pings back from webmail.Giacomcp.com even on our normal office network when I can access webmail.giacomcp.com OK.  I don't have knowledge of the peering situation, it's beyond my knowledge of analysis I am afraid, but interested to see what you have found though.  I'll maybe have another chat to the chap at Giacom tomorrow.  If it's OK, can you tell me how you got those results so I can repeat them to present to Giacom?  From what the Giacom engineer said they had packet captures that seemed show where things were going wrong - but I'm only the messenger in that respect.

[MozzaSec]

2013 and 2016 are in Extended Support, so means that MS will support where they have been paid to support them, and provide some security updates. 

ICMP is mostly blocked nowadays from most servers, as people can use it for ping flooding servers. 

So I can only summise, from the information I have and have tested.. 

This is what I have told a peer when they asked for my advice and is related to o2 as well as EE

the traffic gets into KCOM's network at 195.66.224.119 from 195.66.227.169 which is Linx and into Giacom's MPLS (i assume) at 213.249.133.1 and then hops to 213.249.133.2 

The 2 IP's in the 213.249.133.0 range are managed by KCOM

So for the peering information...

This is the direct link

https://www.peeringdb.com/net/4642

And for what is know as the Looking Glass you need to visit https://lookingglass.centurylink.com/

and try the London and Manchester Peers, as they are the one that's are UK based. 

I am surprised that Giacom haven't picked this up, and done some research on peering, as it would be one of the things I would pick up to check with KCOM, as well as routing the data across the backup ISP just to test..

[JohnMaz]

Thanks that's very helpful.

[JohnMaz]

There is another thread on the O2 forums here: 

https://link.edgepilot.com/s/ce692194/U1gz4IrrG0agVCJxsYwDew?u=https://community.o2.co.uk/t5/Tech-Su...

I don't have an O2 account.

Our temporary workaround is to divert traffic from devices on our customer network is to change the routing in their Draytek Router so that all these URL's go via their slower ASDL network

URL's we needed to change are:

outlook.giacomcp.com
autodiscover.giacomcp.com
ews.giacomcp.com
oab.giacomcp.com
webmail.giacomcp.com

This works but won't help anyone that doesn't have a way to implement this.
Our solution is done within a Draytek 2862 Router, as the customer had an ADSL2+ connection using another provider on WAN1, and the EE Router connected via WAN2. We then setup a Load balancing Rule, where any traffic to any of the above URL's goes to WAN1, instead of over WAN2, which would be the default. 

This doesn't help for their users working using EE 4G on their mobiles, but it does work when they use WiFI in the office.

[MozzaSec]

@JohnMaz 

That is my thread  (I am madasaf1sh on o2, long story :)_ 

The more digging I do, when I have finished the day job, the more I think Giacom don't have a clue what the issue is. 

I cant even get my Enterprise or Personal o2 sim to route through Lumens / L3, and the same with my EE sim. 

Just to make it clear I am not a customer of Giacom, just and Enterprise Messaging Admin, but some of the stuff I have seen them say doesn't make sense... 

I will keep digging and seeing if I can find owt out... 

[Flyingfemme]

It would help a lot if EE could just tell everyone what it was they changed when all the problems started........would cut down the places to look!