Март | 2018

Дано: MX80
Поставили на сайт новый MX80, с пред настройкой только IP/mask для управления. На сайте поднял IGP(а еще была мысль сделать это на тестовом стенде), после этого в логах:

R1> show log messages | last
Nov  3 11:46:19  R1 tfeb0 LU complex 0:   
Nov  3 11:46:19  R1 tfeb0 CMLC: Error seting error, no opcode handle 
Nov  3 11:46:20  R1 tfeb0 CMLC: action for error, no opcode handle 
Nov  3 11:46:20  R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from  
Nov  3 11:47:05  R1 tfeb0 LU complex 0:   
Nov  3 11:47:05  R1 tfeb0 CMLC: Error seting error, no opcode handle 
Nov  3 11:47:06  R1 tfeb0 CMLC: action for error, no opcode handle 
Nov  3 11:47:06  R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from  
Nov  3 11:47:07  R1 tfeb0 LU complex 0:   
Nov  3 11:47:07  R1 tfeb0 CMLC: Error seting error, no opcode handle 
Nov  3 11:47:08  R1 tfeb0 CMLC: action for error, no opcode handle 
Nov  3 11:47:08  R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from

R1> show log messages | last

Nov 3 11:46:19 R1 tfeb0 LU complex 0:

Nov 3 11:46:19 R1 tfeb0 CMLC: Error seting error, no opcode handle

Nov 3 11:46:20 R1 tfeb0 CMLC: action for error, no opcode handle

Nov 3 11:46:20 R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from

Nov 3 11:47:05 R1 tfeb0 LU complex 0:

Nov 3 11:47:05 R1 tfeb0 CMLC: Error seting error, no opcode handle

Nov 3 11:47:06 R1 tfeb0 CMLC: action for error, no opcode handle

Nov 3 11:47:06 R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from

Nov 3 11:47:07 R1 tfeb0 LU complex 0:

Nov 3 11:47:07 R1 tfeb0 CMLC: Error seting error, no opcode handle

Nov 3 11:47:08 R1 tfeb0 CMLC: action for error, no opcode handle

Nov 3 11:47:08 R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from

Логи с TFEB аналогичные, собираем:

R1> request pfe execute target tfeb0 command "show syslog messages"
R1> request pfe execute target tfeb0 command "show nvram"

1 2	R1> request pfe execute target tfeb0 command "show syslog messages" R1> request pfe execute target tfeb0 command "show nvram"

Все плохо, пробуем перезагрузить TFEB, сохранив логи загрузки для JTAC. В одной сессии заходим на консоль TFEB(TCP сессия не разорвется, в консоль ничего не отправлять, ну и keepalive таймер в 10 сек не ставить конечно):

R1> start shell user root
# cty -F tfeb0

1 2	R1> start shell user root # cty -F tfeb0

А в параллельной сессии перезагружаем:

R1> request chassis tfeb restart
TFEB will be restarted NOW.

1 2	R1> request chassis tfeb restart TFEB will be restarted NOW.

Наблюдаем за загрузкой:

R2> ping 10.42.1.203
PING 10.42.1.203 (10.42.1.203): 56 data bytes
64 bytes from 10.42.1.203: icmp_seq=71 ttl=61 time=5.169 ms
...
--- 10.42.1.203 ping statistics ---
111 packets transmitted, 40 packets received, 63% packet loss
round-trip min/avg/max/stddev = 3.155/4.622/6.135/0.941 ms

R2> ping 10.42.1.203

PING 10.42.1.203 (10.42.1.203): 56 data bytes

64 bytes from 10.42.1.203: icmp_seq=71 ttl=61 time=5.169 ms

...

--- 10.42.1.203 ping statistics ---

111 packets transmitted, 40 packets received, 63% packet loss

round-trip min/avg/max/stddev = 3.155/4.622/6.135/0.941 ms

Проверяем логи после загрузки, оформляем RMA:

R1> show log messages | last | match LUCHIP
Nov  3 14:01:31  R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from
Nov  3 14:01:33  R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from
Nov  3 14:02:21  R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from
Nov  3 14:04:25  R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from
Nov  3 14:05:15  R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from
Nov  3 14:23:12  R1 tfeb0 LUCHIP(0) RMC 0  Correctable ECC @ 0x280002, cnt 1, syn 0x0 - EDMEM[0xa00008]
Nov  3 14:23:12  R1 tfeb0 LUCHIP(0):
Nov  3 14:23:14  R1 tfeb0 LUCHIP(0) RMC 0  Correctable ECC @ 0x280002, cnt 3, syn 0x0 - EDMEM[0xa00008]
Nov  3 14:23:14  R1 tfeb0 LUCHIP(0):
Nov  3 14:24:10  R1 tfeb0 LUCHIP(0) RMC 0  Correctable ECC @ 0x280002, cnt 3, syn 0x0 - EDMEM[0xa00008]
Nov  3 14:24:10  R1 tfeb0 LUCHIP(0):
Nov  3 14:24:12  R1 tfeb0 LUCHIP(0) RMC 0  Correctable ECC @ 0x280002, cnt 2, syn 0x0 - EDMEM[0xa00008]
Nov  3 14:24:12  R1 tfeb0 LUCHIP(0):
Nov  3 14:26:14  R1 tfeb0 LUCHIP(0) RMC 0  Correctable ECC @ 0x280002, cnt 6, syn 0x0 - EDMEM[0xa00008]
Nov  3 14:26:14  R1 tfeb0 LUCHIP(0):
Nov  3 14:26:16  R1 tfeb0 LUCHIP(0) RMC 0  Correctable ECC @ 0x280002, cnt 3, syn 0x0 - EDMEM[0xa00008]
Nov  3 14:26:16  R1 tfeb0 LUCHIP(0):

R1> show log messages | last | match LUCHIP

Nov 3 14:01:31 R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from

Nov 3 14:01:33 R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from

Nov 3 14:02:21 R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from

Nov 3 14:04:25 R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from

Nov 3 14:05:15 R1 tfeb0 LUCHIP(0)::FATAL ERROR!! from

Nov 3 14:23:12 R1 tfeb0 LUCHIP(0) RMC 0 Correctable ECC @ 0x280002, cnt 1, syn 0x0 - EDMEM[0xa00008]

Nov 3 14:23:12 R1 tfeb0 LUCHIP(0):

Nov 3 14:23:14 R1 tfeb0 LUCHIP(0) RMC 0 Correctable ECC @ 0x280002, cnt 3, syn 0x0 - EDMEM[0xa00008]

Nov 3 14:23:14 R1 tfeb0 LUCHIP(0):

Nov 3 14:24:10 R1 tfeb0 LUCHIP(0) RMC 0 Correctable ECC @ 0x280002, cnt 3, syn 0x0 - EDMEM[0xa00008]

Nov 3 14:24:10 R1 tfeb0 LUCHIP(0):

Nov 3 14:24:12 R1 tfeb0 LUCHIP(0) RMC 0 Correctable ECC @ 0x280002, cnt 2, syn 0x0 - EDMEM[0xa00008]

Nov 3 14:24:12 R1 tfeb0 LUCHIP(0):

Nov 3 14:26:14 R1 tfeb0 LUCHIP(0) RMC 0 Correctable ECC @ 0x280002, cnt 6, syn 0x0 - EDMEM[0xa00008]

Nov 3 14:26:14 R1 tfeb0 LUCHIP(0):

Nov 3 14:26:16 R1 tfeb0 LUCHIP(0) RMC 0 Correctable ECC @ 0x280002, cnt 3, syn 0x0 - EDMEM[0xa00008]

Nov 3 14:26:16 R1 tfeb0 LUCHIP(0):

Мораль, если есть возможность, нужно «залить» немного в коробку :).

Дано: ACX2000,12.3X54-D10.6.

Перестал работать L2VPN, статус в UP но трафик не ходит, после различных плясок с бубном и решений типа «проблема на вашей стороне», административно выкл/вкл интерфейс к клиенту:

set interfaces ge-0/1/2 disable
commit
rollback 1
commit

set interfaces ge-0/1/2 disable

commit

rollback 1

commit

не помогло, но после этого в логах:

R1# run show log dump | last
Aug 23 14:35:57  R1 rpd[2276]: L2circuit IFF Change: IFL ge-0/1/2.3694 flags 0x0, nbr-id 10.110.0.157 vc-id 3694
Aug 23 14:35:57  R1 rpd[2276]: L2circuit IFL Repository added: flag 0x0 nbr-id 10.110.0.157, vc-id 3694 IFL ge-0/1/2.3694 index 341 state Up Broadcast Multicast
Aug 23 14:35:57  R1 rpd[2276]: L2circuit IFF Change: IFL ge-0/1/2.3694 connection protection, working status
= UP, protect status = UP, active path = Working
Aug 23 14:35:57  R1 rpd[2276]: L2circuit IFF Change: IFL ge-0/1/2.1857 flags 0x0, nbr-id 10.110.0.158 vc-id 1857
Aug 23 14:35:57  R1 rpd[2276]: L2circuit IFL Repository added: flag 0x0 nbr-id 10.110.0.158, vc-id 1857 IFL ge-0/1/2.1857 index 340 state Up Broadcast Multicast
Aug 23 14:35:57  R1 rpd[2276]: L2circuit IFF Change: IFL ge-0/1/2.1857 connection protection, working status
= UP, protect status = UP, active path = Working
Aug 23 14:35:57  R1 feb0 Failed to install the RT entry in acx_rt_ccc_entry_create (status: 1)
Aug 23 14:35:57  R1 /kernel: ether_ifl_ccc_status_change 3414: Change in IFF_CCCDOWN on IFL - GE LOS alarm being reset - sending async notification
Aug 23 14:35:57  R1 /kernel: if_pfe_iflconfig_tlv_helper ge-0/1/2.3694: header->tlv_type 82 received
Aug 23 14:35:57  R1 /kernel: ether_ifl_ccc_status_change 3414: Change in IFF_CCCDOWN on IFL - GE LOS alarm being reset - sending async notification
Aug 23 14:35:57  R1 /kernel: if_pfe_iflconfig_tlv_helper ge-0/1/2.1857: header->tlv_type 82 received
Aug 23 14:35:57  R1 mib2d[2300]: SNMP_TRAP_LINK_DOWN: ifIndex 609, ifAdminStatus up(1), ifOperStatus down(2),
 ifName ge-0/1/2.3694
Aug 23 14:35:57  R1 rpd[2276]: EVENT <Address> ge-0/1/2.3694 index 2684275464 <Up Broadcast Multicast> address #0 1e.6e.f.ff.0.0.0.0.0.0.81.0.81.0.0.1.3.0.0.0.0.0.0.0.1.2.0.0.e.6e.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0
Aug 23 14:35:57  R1 mib2d[2300]: SNMP_TRAP_LINK_DOWN: ifIndex 604, ifAdminStatus up(1), ifOperStatus down(2),
 ifName ge-0/1/2.1857
Aug 23 14:35:57  R1 mib2d[2300]: SNMP_TRAP_LINK_UP: ifIndex 609, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/1/2.3694
Aug 23 14:35:57  R1 mib2d[2300]: SNMP_TRAP_LINK_UP: ifIndex 604, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/1/2.1857
Aug 23 14:35:57  R1 feb0 RT-HAL,rt_entry_add_msg_proc,3002: rt_halp_vectors->rt_create failed
Aug 23 14:35:57  R1 feb0 RT-HAL,rt_entry_add_msg_proc,3062: proto ccc,len 0 prefix default nh 1048688
Aug 23 14:35:58  R1 feb0 RT-HAL,rt_msg_handler,601: route process failed
Aug 23 14:35:58  R1 feb0 acx_bcm_mpls_nni_port_add: NNI VPWS port_add failed (-6:Table full)
Aug 23 14:35:58  R1 feb0 acx_rt_ccc_eth_vpws_vpn_nni_port_add: NNI VPWS port_add failed AC-IFL:340 UC-NH:1537
 in_label 466929  flags 0x1d20 port 0x0 vpn 8193 (-6:Table full)
Aug 23 14:35:58  R1 feb0 acx_rt_ccc_eth_vpws_hw_create: add of NNI port failed for PW-ifl: 340, status: 1
Aug 23 14:35:58  R1 feb0 acx_rt_ccc_entry_install: PW HW setup failed - RT:default NHID:1048801 Err:1
Aug 23 14:35:58  R1 feb0 Failed to install the RT entry in acx_rt_ccc_entry_create (status: 1)
Aug 23 14:35:58  R1 feb0 RT-HAL,rt_entry_add_msg_proc,3002: rt_halp_vectors->rt_create failed
Aug 23 14:35:58  R1 feb0 RT-HAL,rt_entry_add_msg_proc,3062: proto ccc,len 0 prefix default nh 1048801
Aug 23 14:35:59  R1 feb0 RT-HAL,rt_msg_handler,601: route process failed

R1# run show log dump | last

Aug 23 14:35:57 R1 rpd[2276]: L2circuit IFF Change: IFL ge-0/1/2.3694 flags 0x0, nbr-id 10.110.0.157 vc-id 3694

Aug 23 14:35:57 R1 rpd[2276]: L2circuit IFL Repository added: flag 0x0 nbr-id 10.110.0.157, vc-id 3694 IFL ge-0/1/2.3694 index 341 state Up Broadcast Multicast

Aug 23 14:35:57 R1 rpd[2276]: L2circuit IFF Change: IFL ge-0/1/2.3694 connection protection, working status

= UP, protect status = UP, active path = Working

Aug 23 14:35:57 R1 rpd[2276]: L2circuit IFF Change: IFL ge-0/1/2.1857 flags 0x0, nbr-id 10.110.0.158 vc-id 1857

Aug 23 14:35:57 R1 rpd[2276]: L2circuit IFL Repository added: flag 0x0 nbr-id 10.110.0.158, vc-id 1857 IFL ge-0/1/2.1857 index 340 state Up Broadcast Multicast

Aug 23 14:35:57 R1 rpd[2276]: L2circuit IFF Change: IFL ge-0/1/2.1857 connection protection, working status

= UP, protect status = UP, active path = Working

Aug 23 14:35:57 R1 feb0 Failed to install the RT entry in acx_rt_ccc_entry_create (status: 1)

Aug 23 14:35:57 R1 /kernel: ether_ifl_ccc_status_change 3414: Change in IFF_CCCDOWN on IFL - GE LOS alarm being reset - sending async notification

Aug 23 14:35:57 R1 /kernel: if_pfe_iflconfig_tlv_helper ge-0/1/2.3694: header->tlv_type 82 received

Aug 23 14:35:57 R1 /kernel: ether_ifl_ccc_status_change 3414: Change in IFF_CCCDOWN on IFL - GE LOS alarm being reset - sending async notification

Aug 23 14:35:57 R1 /kernel: if_pfe_iflconfig_tlv_helper ge-0/1/2.1857: header->tlv_type 82 received

Aug 23 14:35:57 R1 mib2d[2300]: SNMP_TRAP_LINK_DOWN: ifIndex 609, ifAdminStatus up(1), ifOperStatus down(2),

ifName ge-0/1/2.3694

Aug 23 14:35:57 R1 rpd[2276]: EVENT <Address> ge-0/1/2.3694 index 2684275464 <Up Broadcast Multicast> address #0 1e.6e.f.ff.0.0.0.0.0.0.81.0.81.0.0.1.3.0.0.0.0.0.0.0.1.2.0.0.e.6e.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0

Aug 23 14:35:57 R1 mib2d[2300]: SNMP_TRAP_LINK_DOWN: ifIndex 604, ifAdminStatus up(1), ifOperStatus down(2),

ifName ge-0/1/2.1857

Aug 23 14:35:57 R1 mib2d[2300]: SNMP_TRAP_LINK_UP: ifIndex 609, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/1/2.3694

Aug 23 14:35:57 R1 mib2d[2300]: SNMP_TRAP_LINK_UP: ifIndex 604, ifAdminStatus up(1), ifOperStatus up(1), ifName ge-0/1/2.1857

Aug 23 14:35:57 R1 feb0 RT-HAL,rt_entry_add_msg_proc,3002: rt_halp_vectors->rt_create failed

Aug 23 14:35:57 R1 feb0 RT-HAL,rt_entry_add_msg_proc,3062: proto ccc,len 0 prefix default nh 1048688

Aug 23 14:35:58 R1 feb0 RT-HAL,rt_msg_handler,601: route process failed

Aug 23 14:35:58 R1 feb0 acx_bcm_mpls_nni_port_add: NNI VPWS port_add failed (-6:Table full)

Aug 23 14:35:58 R1 feb0 acx_rt_ccc_eth_vpws_vpn_nni_port_add: NNI VPWS port_add failed AC-IFL:340 UC-NH:1537

in_label 466929 flags 0x1d20 port 0x0 vpn 8193 (-6:Table full)

Aug 23 14:35:58 R1 feb0 acx_rt_ccc_eth_vpws_hw_create: add of NNI port failed for PW-ifl: 340, status: 1

Aug 23 14:35:58 R1 feb0 acx_rt_ccc_entry_install: PW HW setup failed - RT:default NHID:1048801 Err:1

Aug 23 14:35:58 R1 feb0 Failed to install the RT entry in acx_rt_ccc_entry_create (status: 1)

Aug 23 14:35:58 R1 feb0 RT-HAL,rt_entry_add_msg_proc,3002: rt_halp_vectors->rt_create failed

Aug 23 14:35:58 R1 feb0 RT-HAL,rt_entry_add_msg_proc,3062: proto ccc,len 0 prefix default nh 1048801

Aug 23 14:35:59 R1 feb0 RT-HAL,rt_msg_handler,601: route process failed

Находим похожий PR, PR1216198:

On ACX series routers, in a scaled environment with link protection/FRR, frequent LDP route flaps/NH flaps or route churn due to some activity may lead to the «Operation still running» messages . These route flaps are leading to creation of duplicate mpls entries in the hardware ASIC and due to that memory is consumed in the hardware ASIC. This memory full is observed with the pattern «acx_nh_tag_hw_install: BCM L3 Egress create object failed for NH 13423 (-6:Table full)» messages in the logs.
Workaround: Restart the PFE
Triggers: This issue might be seen if the following conditions are met:

On ACX series routers

Interface flapping

LDP/RSVP/MPLS route flaps

Выполняем WA:

R1> request chassis feb restart slot 0

1	R1> request chassis feb restart slot 0

Через сек 20 связь с коробкой должна восстановится. Трафик по CCC начал ходить.

IPv5

Архив за месяц: Март 2018

Juniper: LUCHIP(0)::FATAL ERROR!

Juniper: Трафик не ходит через L2VPN

Networking