High Packet loss / timeouts in broadband backbone network

[infinidim]

Folks,

I have been monitoring the EE / BT backbone network for some time now and have noticed either in my monitoring tools or in the performance of streaming service that the backbone broadband network has some very high packet loss / timeouts. The timeout I have set in my monitoring tool is 2.5 seconds which I would think would be enough. 2.5 seconds relates to twice around the world on fibre propagation delay, at least......

The effect of this in the backbone network is to have performance issues with streaming services.

The speed of the broadband is what I was contracted to have but the issue of packet loss / timeouts is effecting the quality of service.

I have tried to raise this with customer services but they seem to only look after the front end of the service speed of the broadband and failures of the EE router / modem.

Because I have both my own tools as well as an Oftcom Samknows box on site I can see exactly what is happening in the BT / EE backbone networkPacket loss in backboneTimeouts in backbone

Most of the issue appear to be around the Slough BT core node area.

I have attached a couple of screenshots / captures of what is going on.

Folks I have done all the normal diagnostics and testing but nothing fixes this issue as it is in the backbone network.

Your thoughts folks would be appreciated.

Regards

John Haynes

[infinidim]

I am a recent EE customer coming from migrated from BT Broadband.

I was hoping that the technical team would spot this thread. I agree about the first level support but sometimes you get put through to BT technical support rather than EE and it seems to be better.

Regards 

John 

[Mustrum]

@infinidim  As you may have guessed I am not a fan of Samknows, I used them a few years ago when I stopped getting employee broadband, but found they were creating many times more data usage on my network than I was.

Until lightening took out my very good Netgear router I used DslStats, and Thinkbroadband BQM to keep an eye on things https://www.thinkbroadband.com/broadband/monitoring/quality   They also have a ping box which I believe is more consistent than Samknows.

Just to help complete the picture, which router are you using and what are your sync speeds?

[infinidim]

I do like Thinkbroadband apps. I didn't know about the ping box. I'll message them about it.

John 

[infinidim]

@Mustrum 

I thought you might be interested in what has happened today.

I saw on the UK core node in Slough very bad dropping both the ethernet access path into it as well as the node itself. What happened was it looks like the monitoring team in BT Wholesale saw this happening and re-routed all the traffic firstly via Colindale core node then onto the Telehouse core node in Docklands - which is where I am currently. I am still seeing a few drops on the ethernet access but the core node appears to be working well.

Regards

John

[Mustrum]

@infinidim  Have spotted another possibly related issues, care to have a look?

[infinidim]

@Mustrum

That looks interesting.

Since I posted my route into the BT core network has now changed a few times. Also the end point in the Openreach overlay has moved quite a lot. The core BT node isn't dropping as much but still get quite a few issues on the Openreach overlay.  I have raised it with EE front end but I am not confident in getting it fixed - process problems between departments of BT. 

[Rags4983]

Been forced into swapping BT to EE and this is why I stayed with BT for so many years,  backbone issues

[infinidim]

Folks,

Strange thing happened this morning, my routing is no longer going via the core nodes in either Slough or Telehouse (Docklands). It seem  now to be going via an outer ring of nodes. Most of the ethernet drops in the Openreach part of my route have disappeared as well.  They come and go randomly.

Given the issue has been going on for a while maybe someone in the depths of BT is maybe looking at this at last.

Still no moderator has come and made a comment on these posts but that was the same on the BT forum I used write things on.

Lets hope it stays stable for a while in the core. All I need to get sorted is the local drops between my in premises EE Hub and the FTTC cabinet - I am not going to hold my breath to get that one fixed as it s Openreach....

Regards

John  (infinidim)

[infinidim]

Folks,

I have done a lot of research into the issues that I have seen and now on my EE Router and they seem to be not on the local loop - copper from the FTTC cabinet. They appear to be in the core BT network. What appears to happen is there is a lot of drops at a point in the core which causes a high latency if your IP path has to go through it ( it might even be that there is a VLAN from the FTTC cabinet DSLAM). Once theses errors / performance issues get to a high enough level the network re-routes which then causes the EE router to drop it's connection. This whole process looks like it takes about a minute to recover.

If you have EE Hybrid connect this will not kick in as it takes about 2 or 3 minutes for the Hybrid connect to recognise the failure and bring the mobile data up.

What EE / BT retail can do about this I don't know. This looks like it relates to the BT Wholesale / Openreach parts of the network.

Regards

Infinidim (John)