Home > Articles > Programming > General Programming/Other Languages

  • Print
  • + Share This
Like this article? We recommend

Addendum - snoop of Failover

The network cable connected to the qfe0 interface was unplugged to cause the failover and then reconnected to cause the failback. A trace was taken of the network traffic during these events.

The following console messages, show the times of failure detection and recovery. The times can be cross-referenced against the network trace so that the traffic caused by the failure and recovery can be identified.

camelot# 
camelot# 
camelot# 
camelot# May 9 16:18:28 camelot qfe: SUNW,qfe0 : 
No response from Ethernet network : Link down -- cable problem?
May 9 16:18:35 camelot in.mpathd[33]: 
NIC failure detected on qfe0 of group production
May 9 16:18:35 camelot in.mpathd[33]: 
Successfully failed over from NIC qfe0 to NIC qfe4
May 9 16:18:40 camelot qfe: SUNW,qfe0 : 
No response from Ethernet network : Link down -- cable problem?
May 9 16:18:51 camelot last message repeated 1 time
May 9 16:19:01 camelot qfe: SUNW,qfe0 : External Transceiver Selected.
May 9 16:19:01 camelot qfe: SUNW,qfe0 : 
Auto-Negotiated 100 Mbps Half-Duplex Link Up
May 9 16:19:17 camelot in.mpathd[33]: 
NIC repair detected on qfe0 of group production
May 9 16:19:17 camelot in.mpathd[33]: Successfully failed back to NIC qfe0


The following is the network trace taken


16:18:24.43283  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10541)
16:18:24.43349 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10541)
16:18:25.57280  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10508)
16:18:25.57361 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10508)
16:18:25.78279  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10542)
16:18:25.78338 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10542)
16:18:27.04279  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10509)
16:18:27.04348 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10509)
16:18:27.07279  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10543)
16:18:27.07340 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10543)
16:18:28.41295  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10510)
16:18:28.41367 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10510)
16:18:29.68282  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10511)
16:18:29.68344 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10511)
16:18:31.08279  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10512)
16:18:31.08379 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10512)
16:18:32.32288  camelot-1 -> 129.152.10.1 
DNS C 57.47.153.129.in-addr.arpa. Internet PTR ?
16:18:32.39289  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10513)
16:18:32.39357 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10513)
16:18:33.93279  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10514)
16:18:33.93361 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10514)
16:18:35.49433    camelot -> (broadcast) ARP C Who is 192.168.49.42, camelot ?
16:18:35.49473  camelot-1 -> (broadcast) ARP C Who is defrtr, defrtr ?
16:18:35.49610 defrtr -> camelot-1  ARP R defrtr, defrtr is 0:50:bd:bb:a4:0
16:18:35.52294  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10515)
16:18:35.52357 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10515)
16:18:36.56277  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10516)
16:18:36.56337 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10516)
16:18:37.49274    camelot -> (broadcast) ARP C Who is 192.168.49.42, camelot ?
16:18:38.46287  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10517)
16:18:38.46363 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10517)
16:18:39.49265    camelot -> (broadcast) ARP C Who is 192.168.49.42, camelot ?
16:18:39.98279  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10518)
16:18:39.98352 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10518)

The repeating ICMP 
Echo requests and replys have been deleted.



16:19:14.24296  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10542)
16:19:14.24359 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10542)
16:19:14.52282  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10577)
16:19:14.52343 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10577)
16:19:15.51279  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10543)
16:19:15.51347 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10543)
16:19:16.13297  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10578)
16:19:16.13360 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10578)
16:19:17.13298  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10544)
16:19:17.13355 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10544)
16:19:17.53401    camelot -> (broadcast) ARP C Who is 192.168.49.42, camelot ?
16:19:17.53464    camelot -> (broadcast) ARP C Who is defrtr, defrtr ?
16:19:17.53543 defrtr -> camelot    ARP R defrtr, defrtr is 0:50:bd:bb:a4:0
16:19:17.53545  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10579)
16:19:17.53601 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10579)
16:19:17.84286  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10580)
16:19:17.84353 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10580)
16:19:18.07316  camelot-1 -> (broadcast) ARP C Who is defrtr, defrtr ?
16:19:18.07376 defrtr -> camelot-1  ARP R defrtr, defrtr is 0:50:bd:bb:a4:0
16:19:18.07380  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10545)
16:19:18.07441 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10545)
16:19:19.10283  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10581)
16:19:19.10341 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10581)
16:19:19.41279  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10546)
16:19:19.41339 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10546)
16:19:19.53279    camelot -> (broadcast) ARP C Who is 192.168.49.42, camelot ?
16:19:20.24284  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10582)
16:19:20.24342 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10582)
16:19:20.73276  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10547)
16:19:20.73331 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10547)
16:19:21.53278    camelot -> (broadcast) ARP C Who is 192.168.49.42, camelot ?
16:19:21.69284  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10583)
16:19:21.69343 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10583)
16:19:22.79281  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10548)
16:19:22.79350 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10548)
16:19:23.04303  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10584)
16:19:23.04356 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10584)
16:19:23.97280  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10549)
16:19:23.97337 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10549)
16:19:24.31281  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10585)
16:19:24.31352 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10585)
16:19:25.69282  camelot-qfe4 -> defrtr ICMP 
Echo request (ID: 8451 Sequence number: 10550)
16:19:25.69340 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID: 8451 Sequence number: 10550)
16:19:25.89281  camelot-qfe0 -> defrtr ICMP 
Echo request (ID: 8450 Sequence number: 10586)
16:19:25.89352 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID: 8450 Sequence number: 10586)
16:19:26.84279  camelot-qfe4 -> defrtr ICMP 
Echo request (ID:8451 Sequence number: 10551)
16:19:26.84354 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID:8451 Sequence number: 10551)
16:19:27.39283  camelot-qfe0 -> defrtr ICMP 
Echo request (ID:8450 Sequence number: 10587)
16:19:27.39342 defrtr -> camelot-qfe0  ICMP 
Echo reply (ID:8450 Sequence number: 10587)
16:19:28.29302  camelot-qfe4 -> defrtr ICMP 
Echo request (ID:8451 Sequence number: 10552)
16:19:28.29370 defrtr -> camelot-qfe4  ICMP 
Echo reply (ID:8451 Sequence number: 10552)
  • + Share This
  • 🔖 Save To Your Account