topic Re: High Packet loss / timeouts in broadband backbone network in Broadband & Landline

High Packet loss / timeouts in broadband backbone network

infinidim — Mon, 20 May 2024 15:22:06 GMT

Folks,

I have been monitoring the EE / BT backbone network for some time now and have noticed either in my monitoring tools or in the performance of streaming service that the backbone broadband network has some very high packet loss / timeouts. The timeout I have set in my monitoring tool is 2.5 seconds which I would think would be enough. 2.5 seconds relates to twice around the world on fibre propagation delay, at least......

The effect of this in the backbone network is to have performance issues with streaming services.

The speed of the broadband is what I was contracted to have but the issue of packet loss / timeouts is effecting the quality of service.

I have tried to raise this with customer services but they seem to only look after the front end of the service speed of the broadband and failures of the EE router / modem.

Because I have both my own tools as well as an Oftcom Samknows box on site I can see exactly what is happening in the BT / EE backbone networkPacket loss in backboneTimeouts in backbone

Most of the issue appear to be around the Slough BT core node area.

I have attached a couple of screenshots / captures of what is going on.

Folks I have done all the normal diagnostics and testing but nothing fixes this issue as it is in the backbone network.

Your thoughts folks would be appreciated.

Regards

John Haynes

Re: High Packet loss / timeouts in broadband backbone network

XRaySpeX — Mon, 20 May 2024 15:39:19 GMT

BT's routing nodes have better things to do than responding to pings, i.e. their jobs.

Re: High Packet loss / timeouts in broadband backbone network

Mustrum — Mon, 20 May 2024 16:05:19 GMT

@infinidim Alas your testing does not really prove anything.

Samknows very little about broadband testing and use some very flawed methods to try and justify the funding they get from Ofcom.

Core routers are designed to route traffic as their first priority, and drop ping requests in the same way a server does when they have other stuff to do.

What issues are you actually having?

Re: High Packet loss / timeouts in broadband backbone network

infinidim — Mon, 20 May 2024 17:31:24 GMT

I know that given that up until a few years ago I was responsible for the quality of BT's data and fibre optics outside the UK.

Re: High Packet loss / timeouts in broadband backbone network

XRaySpeX — Mon, 20 May 2024 17:31:59 GMT

I never imagined BT had fibre outside UK.

Re: High Packet loss / timeouts in broadband backbone network

Mustrum — Mon, 20 May 2024 17:51:50 GMT

I seem to remember BT Global Services had fibre products available in a number of countries as well as transcontinental undersea cables. How much is left these days that has not been sold off is another thing..

Re: High Packet loss / timeouts in broadband backbone network

infinidim — Mon, 20 May 2024 17:52:23 GMT

The issues I see is at certain times streaming is struggling. This correlates with the higher drop rate at the core router that we are routed through 90% of the time. Given these core routers are either Juniper or Cisco I would not expected this issue.

What I have not mentioned is that this dropping / delay does not happen all the time. It also doesn't happen if I manage to be routed via smother core router.

If you look at the screen shot of the path then you will see at the time I took it that more of the drops were where the ethernet overlay from Openreach connects to the core network run by BT Wholesale.

If you see this sort of issue there appears via the current EE / BT processes no way to get it investigated.

EE and BT Retail are only really marketing / sales organisations.

Regards

John Haynes

Re: High Packet loss / timeouts in broadband backbone network

Mustrum — Mon, 20 May 2024 19:21:39 GMT

@infinidim are you a legacy EE customer or a more recent New EE customer?

Online technical support has been limited, but has improved recently, although still could be better. Hopefully one of the new technical team will spot your thread and respond. Otherwise you may need to persist and get through to the second line tech team, rather than the script readers on the front line.

As for BT Divisions, YES I am very well aware!

Re: High Packet loss / timeouts in broadband backbone network

infinidim — Mon, 20 May 2024 22:36:49 GMT

Up until a few years ago BT was 2nd largest fiber networks in Spain, Germany, Italy etc. It also had part ownership of many subsea cables as well.

John

Re: High Packet loss / timeouts in broadband backbone network

infinidim — Mon, 20 May 2024 22:37:50 GMT

I was involved with most of them.

John

Re: High Packet loss / timeouts in broadband backbone network

infinidim — Mon, 20 May 2024 22:45:37 GMT

I am a recent EE customer coming from migrated from BT Broadband.

I was hoping that the technical team would spot this thread. I agree about the first level support but sometimes you get put through to BT technical support rather than EE and it seems to be better.

Regards

John

Re: High Packet loss / timeouts in broadband backbone network

Mustrum — Tue, 21 May 2024 08:24:19 GMT

@infinidim As you may have guessed I am not a fan of Samknows, I used them a few years ago when I stopped getting employee broadband, but found they were creating many times more data usage on my network than I was.

Until lightening took out my very good Netgear router I used DslStats, and Thinkbroadband BQM to keep an eye on things https://www.thinkbroadband.com/broadband/monitoring/quality They also have a ping box which I believe is more consistent than Samknows.

Just to help complete the picture, which router are you using and what are your sync speeds?

Re: High Packet loss / timeouts in broadband backbone network

infinidim — Tue, 21 May 2024 09:42:35 GMT

I do like Thinkbroadband apps. I didn't know about the ping box. I'll message them about it.

John

Re: High Packet loss / timeouts in broadband backbone network

infinidim — Tue, 21 May 2024 14:00:21 GMT

@Mustrum

I thought you might be interested in what has happened today.

I saw on the UK core node in Slough very bad dropping both the ethernet access path into it as well as the node itself. What happened was it looks like the monitoring team in BT Wholesale saw this happening and re-routed all the traffic firstly via Colindale core node then onto the Telehouse core node in Docklands - which is where I am currently. I am still seeing a few drops on the ethernet access but the core node appears to be working well.

Regards

John

Re: High Packet loss / timeouts in broadband backbone network

Mustrum — Sat, 25 May 2024 21:03:39 GMT

@infinidim Have spotted another possibly related issues, care to have a look?

Re: High Packet loss / timeouts in broadband backbone network

infinidim — Sun, 26 May 2024 08:39:07 GMT

@Mustrum

That looks interesting.

Since I posted my route into the BT core network has now changed a few times. Also the end point in the Openreach overlay has moved quite a lot. The core BT node isn't dropping as much but still get quite a few issues on the Openreach overlay. I have raised it with EE front end but I am not confident in getting it fixed - process problems between departments of BT.

Re: High Packet loss / timeouts in broadband backbone network

Rags4983 — Sun, 26 May 2024 10:28:11 GMT

Been forced into swapping BT to EE and this is why I stayed with BT for so many years, backbone issues

Re: High Packet loss / timeouts in broadband backbone network

infinidim — Sun, 26 May 2024 12:59:21 GMT

Folks,

Strange thing happened this morning, my routing is no longer going via the core nodes in either Slough or Telehouse (Docklands). It seem now to be going via an outer ring of nodes. Most of the ethernet drops in the Openreach part of my route have disappeared as well. They come and go randomly.

Given the issue has been going on for a while maybe someone in the depths of BT is maybe looking at this at last.

Still no moderator has come and made a comment on these posts but that was the same on the BT forum I used write things on.

Lets hope it stays stable for a while in the core. All I need to get sorted is the local drops between my in premises EE Hub and the FTTC cabinet - I am not going to hold my breath to get that one fixed as it s Openreach....

Regards

John (infinidim)

Re: High Packet loss / timeouts in broadband backbone network

infinidim — Sun, 02 Jun 2024 14:07:59 GMT

Folks,

I have done a lot of research into the issues that I have seen and now on my EE Router and they seem to be not on the local loop - copper from the FTTC cabinet. They appear to be in the core BT network. What appears to happen is there is a lot of drops at a point in the core which causes a high latency if your IP path has to go through it ( it might even be that there is a VLAN from the FTTC cabinet DSLAM). Once theses errors / performance issues get to a high enough level the network re-routes which then causes the EE router to drop it's connection. This whole process looks like it takes about a minute to recover.

If you have EE Hybrid connect this will not kick in as it takes about 2 or 3 minutes for the Hybrid connect to recognise the failure and bring the mobile data up.

What EE / BT retail can do about this I don't know. This looks like it relates to the BT Wholesale / Openreach parts of the network.

Regards

Infinidim (John)