onderwerp Re: Frequent Lost MDD Timeout leading to outage/drops in Archief

Frequent Lost MDD Timeout leading to outage/drops

vgarot — Thu, 02 Nov 2023 15:44:37 GMT

Dear team,

Can someone review my modem values for any possible reason behind frequent Lost MDD Timeout and T3 timeout ?

Config
=======
Modem is in bridge mode and connected directly to the AOP with no splitter (picture attached), but via a 10 meters coax cable which I know is unsupported.

Impact
=======
My router (RT-AX92U) apparently doesn't handle those events particularly well and sometimes a manual reboot of the modem or router is required to recover connectivity. I can leave with a few seconds drops even though not ideal as working from home but the downtime requiring reboots of router/modem are more problematic.

Installation history
========
The connection has been rock solid since last technician visit in March 2022 where the damaged cable from the street and the AOP were replaced (https://community.ziggo.nl/t5/Archief/WAN-connection-drops-T3-and-T4-errors/m-p/866227#M375072).
No change to the network installation has been performed since then.
This new issue has been happening for a few weeks already.

Logs

======
==> uptime : 3.5 days
Standard specification compliant : DOCSIS 3.0
Hardware version : 5.01
Software version : CH7465LG-NCIP-6.15.32p3-NOSH
Cable MAC address : 38:43:7D:CD:CE:99
Cable modem serial number : DEAP8118041D
System up time : 3day(s)16h:26m:44s
Network access : Allowed

==> Several Lost MDD Timeout per day
02-11-2023 09:28:39 notice GUI Login Status - Login Sucess from LAN interface; client ip=[83.85.197.147];CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
02-11-2023 05:06:30 error DHCP RENEW WARNING - Field invalid in response v4 option;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
02-11-2023 04:26:25 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
02-11-2023 04:26:24 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
01-11-2023 22:14:43 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
01-11-2023 22:14:41 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
01-11-2023 17:48:43 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
01-11-2023 17:48:42 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
01-11-2023 15:54:11 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
01-11-2023 15:54:10 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
01-11-2023 15:27:13 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
01-11-2023 15:27:12 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
01-11-2023 11:12:29 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
01-11-2023 11:12:28 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
30-10-2023 10:38:29 notice GUI Login Status - Login Sucess from LAN interface; client ip=[83.85.197.147];CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
30-10-2023 08:36:50 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
30-10-2023 08:36:49 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
30-10-2023 08:24:45 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
30-10-2023 08:24:42 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
30-10-2023 08:21:31 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS=1.1;CM-VER=3.0;
29-10-2023 22:47:29 warning [38:43:7D:C1:79:C6][DFS] Radar signal detected, DFS sequence applied, channel changed from 60 to 44

==> 2 x T3 Timeouts on channel 1 (I'm not expert but seems to be a low num for a 3.5 days uptime)
Upstream bonded channels
Channel Channel Type T1 Timeouts T2 Timeouts T3 Timeouts T4 Timeouts
1 2.0 0 0 2 0
2 2.0 0 0 0 0
3 2.0 0 0 0 0
4 2.0 0 0 0 0

===> Post RS errors
Downstream bonded channels
Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors
1 Locked 36.610 10845073192 1077
2 Locked 37.356 10843852130 876
3 Locked 37.636 10843853364 871
4 Locked 37.636 10843854232 967
5 Locked 37.356 10843849554 1006
6 Locked 37.356 10843848107 931
7 Locked 37.356 10843847140 1040
8 Locked 37.356 10843852639 1215
9 Locked 37.636 10843853809 1188
10 Locked 37.356 10843853872 1631
11 Locked 37.636 10843854799 1578
12 Locked 35.595 10843830701 17692
13 Locked 37.636 10843853676 1675
14 Locked 37.356 10843843930 2096
15 Locked 37.636 10843855029 2046
16 Locked 37.356 10843855526 2039
17 Locked 37.356 9463776136 124085
18 Locked 37.356 7620600537 137110
19 Locked 37.636 3257008659 88407
20 Locked 36.387 4416924621 317650
21 Locked 36.387 10843765208 82607
22 Locked 36.387 10843855904 4021
23 Locked 36.610 10843856400 4077
24 Locked 35.780 10843848193 6269

==> Downstream values

Downstream bonded channels
Channel Frequency (Hz) Power (dBmV) SNR (dB) Modulation Channel ID
1 618000000 0 36 256qam 6
2 578000000 0 37 256qam 1
3 586000000 0 37 256qam 2
4 594000000 0 37 256qam 3
5 602000000 0 37 256qam 4
6 610000000 0 37 256qam 5
7 626000000 0 37 256qam 7
8 634000000 0 37 256qam 8
9 642000000 0 37 256qam 9
10 650000000 0 37 256qam 10
11 658000000 0 37 256qam 11
12 666000000 0 35 256qam 12
13 674000000 0 37 256qam 13
14 682000000 -1 37 256qam 14
15 690000000 0 37 256qam 15
16 698000000 0 37 256qam 16
17 706000000 -1 37 256qam 17
18 714000000 -1 37 256qam 18
19 722000000 0 37 256qam 19
20 730000000 -2 36 256qam 20
21 738000000 -2 36 256qam 21
22 746000000 -2 36 256qam 22
23 754000000 -2 36 256qam 23
24 826000000 -2 35 256qam 32

===> Upstream
Upstream bonded channels
Channel Frequency (Hz) Power (dBmV) Symbol Rate (ksps) Modulation Channel ID
1 58800000 43 5.120 64qam 1
2 38400000 43 5.120 64qam 4
3 45200000 43 5.120 64qam 3
4 52000000 43 5.120 64qam 2

Re: Frequent Lost MDD Timeout leading to outage/drops

tt14111 — Thu, 02 Nov 2023 15:46:38 GMT

Downstream Rx power is quite low for a modem being connected directly to the AOP. MER is quite low as well and DVB-T2 ingress appears to be present on channel 12 (666 MHz). MDD timeouts may only occur during scheduled maintenance or a known outage.

Would be it possible for you to connect the modem using an IEC/F cable (green/red, 1.5 m.) and test the stability this way?

Re: Frequent Lost MDD Timeout leading to outage/drops

vgarot — Mon, 06 Nov 2023 08:34:49 GMT

Not easily, will take some time and only for a few hours but at least we'll if the modem reports higher values. I'll test and upload the new values here.

Re: Frequent Lost MDD Timeout leading to outage/drops

Welmoed — Mon, 13 Nov 2023 12:49:30 GMT

Hi @vgarot, were you already able to perform a new test and can you show us the new values?

Re: Frequent Lost MDD Timeout leading to outage/drops

vgarot — Tue, 14 Nov 2023 08:19:16 GMT

Hi @Welmoed, Thanks for the reminder. Not yet, planning to perform the test over the weekend and report here by beginning of next week. Please bear with me a little longer.

Re: Frequent Lost MDD Timeout leading to outage/drops

Jacob — Tue, 21 Nov 2023 11:25:44 GMT

@vgarot And so we are a little further in time. Do you have an update for us?

Re: Frequent Lost MDD Timeout leading to outage/drops

vgarot — Fri, 24 Nov 2023 10:16:49 GMT

I don't have news yet and apologize for it. I'll test with IEC/F cable (green/red, 1.5 m.) this weekend and report back on Monday with the new values.

Below is latest, with current installation (10m coax cable). T3 timeout are somehow not displayed anymore. Lost MDD timeout are still very frequent. I guess this is a modem FW/HW limitation but log rotating this fast doesn't help with long term troubleshooting.

Standard specification compliant : DOCSIS 3.0
Hardware version : 5.01
Software version : CH7465LG-NCIP-6.15.32p3-NOSH
Cable modem serial number : DEAP8118041D
System up time : 25day(s)17h:52m:3s

11/24/2023 07:39:06 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM -QOS=1.1;CM-VER=3.0;
11/24/2023 07:39:04 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS =1.1;CM-VER=3.0;
11/24/2023 07:38:22 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM -QOS=1.1;CM-VER=3.0;
11/24/2023 07:38:17 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS =1.1;CM-VER=3.0;
11/24/2023 07:38:05 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM -QOS=1.1;CM-VER=3.0;
11/24/2023 07:38:04 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS =1.1;CM-VER=3.0;
11/24/2023 07:37:23 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM -QOS=1.1;CM-VER=3.0;
11/24/2023 07:37:22 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS =1.1;CM-VER=3.0;
11/24/2023 07:35:00 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM -QOS=1.1;CM-VER=3.0;
11/24/2023 07:34:55 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS =1.1;CM-VER=3.0;
11/24/2023 07:34:36 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM -QOS=1.1;CM-VER=3.0;
11/24/2023 07:34:30 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS =1.1;CM-VER=3.0;
11/24/2023 07:34:20 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM -QOS=1.1;CM-VER=3.0;
11/24/2023 07:34:19 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS =1.1;CM-VER=3.0;
11/24/2023 07:33:42 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM -QOS=1.1;CM-VER=3.0;
11/24/2023 07:33:41 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS =1.1;CM-VER=3.0;
11/24/2023 07:31:40 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM -QOS=1.1;CM-VER=3.0;
11/24/2023 07:31:39 warning Lost MDD Timeout;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM-QOS =1.1;CM-VER=3.0;
11/24/2023 07:29:18 notice MDD Recovery following MDD Loss;CM-MAC=38:43:7d:cd:ce:99;CMTS-MAC=00:01:5c:88:d8:55;CM -QOS=1.1;CM-VER=3.0;

Re: Frequent Lost MDD Timeout leading to outage/drops

tt14111 — Fri, 24 Nov 2023 10:19:08 GMT

The modem has re-established the connection ~5 days ago, which explains why the T3 timeouts counters are reset to 0. In addition to the continuous MDD timeouts, correctable codeword errors ('Post RS Errors') are increasing too rapidly.

Re: Frequent Lost MDD Timeout leading to outage/drops

vgarot — Fri, 24 Nov 2023 18:32:34 GMT

Lost connection again a few mins ago, 5 x T3 Timeouts. I'll report back on Monday

Channel Channel Type T1 Timeouts T2 Timeouts T3 Timeouts T4 Timeouts
1 2.0 0 0 1 0
2 2.0 0 0 4 0
3 2.0 0 0 0 0
4 2.0 0 0 0 0

Channel Locked Status RxMER (dB) Pre RS Errors Post RS Errors
1 Locked 37.636 13924188089 5676
2 Locked 37.636 13923184909 2256
3 Locked 37.356 13923185661 2490
4 Locked 37.636 13923187227 2204
5 Locked 37.356 13923184340 5259
6 Locked 37.636 13923182603 6533
7 Locked 37.356 13923181914 9476
8 Locked 37.356 13923182699 8742
9 Locked 37.356 13923180365 9165
10 Locked 37.636 13923178739 11749
11 Locked 37.636 13923173901 11839
12 Locked 36.610 13923172300 13727
13 Locked 37.356 13923180100 9576
14 Locked 37.636 13923182635 8475
15 Locked 37.356 13923185380 8033
16 Locked 37.356 13923184603 8520
17 Locked 37.636 13923175867 9855
18 Locked 37.356 8031289643 449018
19 Locked 37.356 2320684030 97720
20 Locked 36.610 3214225723 168248
21 Locked 36.610 13922582640 611018
22 Locked 36.387 13923181697 13992
23 Locked 36.610 13923171911 14775
24 Locked 35.780 13923181120 14254

Re: Frequent Lost MDD Timeout leading to outage/drops

vgarot — Sat, 25 Nov 2023 20:41:24 GMT

Just connected via 1,5m red/red cable, hence had to use a splitter.
15 min uptime below. I'll try to buy a red/green cable tomorrow ....

Standard specification compliant : DOCSIS 3.0
Hardware version : 5.01
Software version : CH7465LG-NCIP-6.15.32p3-NOSH
Cable modem serial number : DEAP8118041D
System up time : 0day(s)0h:15m:17s
Network access : Allowed

Re: Frequent Lost MDD Timeout leading to outage/drops

tt14111 — Sat, 25 Nov 2023 20:51:38 GMT

Modem data does not exhibit any issues so far. An IEC/F cable green/red can be sent to you free of charge, no need to purchase it yourself.

Re: Frequent Lost MDD Timeout leading to outage/drops

MR_CHIP — Sat, 25 Nov 2023 22:00:23 GMT

@tt14111 schreef:
Modem data does not exhibit any issues so far. An IEC/F cable green/red can be sent to you free of charge, no need to purchase it yourself.

you consider a snr of only 36db acceptable ?

Re: Frequent Lost MDD Timeout leading to outage/drops

tt14111 — Sat, 25 Nov 2023 22:05:58 GMT

Yes, it is acceptable (> 34 dB). Furthermore, MER is usually 2~3 dB lower on the TG2492 and CH7465 compared to other modem types.

Re: Frequent Lost MDD Timeout leading to outage/drops

MR_CHIP — Sat, 25 Nov 2023 22:07:09 GMT

i personally consider a snr db so close the minimum snr to be non acceptable but thats my opninion

would directly connecting the modem to the aop do anything with the snr db

Re: Frequent Lost MDD Timeout leading to outage/drops

tt14111 — Sat, 25 Nov 2023 22:12:31 GMT

It appears to be inherent to these types of modems. I have never seen a TG2492 or CH7465 display a MER over 40 dB. MER will not increase by connecting the modem directly to the AOP.

Re: Frequent Lost MDD Timeout leading to outage/drops

vgarot — Sun, 26 Nov 2023 19:23:53 GMT

24 uptime with 1.5m cable :

Re: Frequent Lost MDD Timeout leading to outage/drops

tt14111 — Sun, 26 Nov 2023 19:28:28 GMT

Correctable codeword errors are increasing slightly faster than preferable, otherwise still looking good.

Re: Frequent Lost MDD Timeout leading to outage/drops

vgarot — Wed, 29 Nov 2023 19:15:28 GMT

@tt14111 Below are the value for 4 days uptime with direct connection to the AOP via 1.5m cable and splitter.
I had a new WAN disconnect yesterday and today, and just observed 19 x T3 Timeouts in the modem.

Modem logs doesn't say much, just a single T3 timeout this morning

Every time it's the same symptom, all devices on my network lose internet access.
From the router perspective the internet status is connected and the WAN IP is visible. But I can't ping any IP on Internet from the router itself or I can't reach the modem on port 80 via netcat from any equipment on the LAN until either the modem or the router is rebooted (disconnect/re-connect the RJ45 between modem and router doesn't help).

I'll try an end device direct connection to the modem next time to see if it's reachable on port 80 at least.
I guess ssh or telnet access to the modem for further debugging is impossible ?

Happy to consider buying and testing another router, but considering the T3 timeouts + high post rs errors I would prefer to receive confirmation from Ziggo the issue is on my end first.

I work in IT, often from home, wan disconnects while running a session with customers is less than ideal.

Re: Frequent Lost MDD Timeout leading to outage/drops

tt14111 — Wed, 29 Nov 2023 19:39:48 GMT

Correctable codeword errors counts are not increasing exceptionally quickly and I would not consider these to be a problem at this time. However, 16 T3 timeouts occurring in immediate succession usually indicates an upstream/ranging problem.

I suggest having a multi-day monitoring session to the modem set up by a moderator. This should reveal any signal problems that may be occurring. Your router is currently blocking incoming ICMP traffic from the WAN interface. If you allow this traffic in the firewall, an ICMP monitoring session may be set up also.

Re: Frequent Lost MDD Timeout leading to outage/drops

vgarot — Wed, 29 Nov 2023 20:19:08 GMT

Incoming ICMP from WAN unblocked on the router's firewall. Tested and confirming traffic is now allowed.
@Jacob @Welmoed Can you please setup multi-day monitoring and ICMP monitoring to the modem as requested by @tt14111 ?