AKS Windows Server 2022: RSTs to a deleted pod's IP loop between the synthetic NIC and the accelerated networking VF, pinning cores at high DPC #655

Activity

1. Rich Shadman (richshadman)

   added

   bugSomething isn't workingSomething isn't working

   triageNew and needs attentionNew and needs attention

   on Oct 5, 2026
2. github-actions commented on Oct 5, 2026

   

   github-actionsbot

   on Oct 5, 2026 – with GitHub Actions

   More actions

   Thank you for creating an Issue. Please note that GitHub is not an official channel for Microsoft support requests. To create an official support request, please open a ticket here. Microsoft and the GitHub Community strive to provide a best effort in answering questions and supporting Issues on GitHub.
3. richshadman commented on Oct 5, 2026

   

   Rich Shadman (richshadman)

   on Oct 5, 2026

   Author

   More actions

   Update: full component capture on an affected node

   We took a pktmon start --capture --comp all capture (6 s, filtered to the stale pod IP) on a node that was still looping, plus collectlogs.ps1 output.

   The same packets circulate indefinitely; nothing generates new ones.

   • 465,230 packet events for the single dead pod IP in 6 seconds.
   • Every one had TTL 128, and nearly all had IP ID 1. Nothing on the path decrements TTL, so one reset loops forever.

   The reset appears to be generated on the Windows side.

   • TTL 128 is the Windows default.
   • The ingress controller whose VIP is the source address runs on Linux nodes, and Linux would send TTL 64.
   • So the circulating RST looks like it came from the Windows host stack or VFP answering for a connection it still tracked, not from the remote server.

   Per-component counters: no component drops anything, and all drop counters read 0.

   vSwitch ext
     Microsoft NetVsc Nic #2 (upper)   Rx   841,559 pkts   | Tx 1,018,969 pkts
     VMS Protocol (lower)              Rx 1,018,939 pkts   | Tx 1,018,939 pkts
   
   Mellanox ConnectX-4 Lx Virtual Ethernet Adapter
     Mellanox ConnectX-4 Lx (upper)    Rx         0 pkts   | Tx 1,018,969 pkts
     NETVSCVFPP (lower)                Rx         0 pkts   | Tx 1,018,939 pkts
   

   Packets enter on the synthetic NetVsc path and leave through the VF, and none come back in through the VF.

   HNS has no endpoint for the IP, yet it is still one of the node NIC's ipConfigurations.

   The capture ETL, HNS endpoint/network/policy dumps, VFP port list and collectlogs.ps1 output are available on request.
4. richshadman commented on Oct 5, 2026

   

   Rich Shadman (richshadman)

   on Oct 5, 2026

   Author

   More actions

   Update: restarting HNS clears the loop; restarting the VF does not

   We tested mitigations on two looping nodes in a non-customer cluster. The nodes run the same image and versions as the original report: AKSWindows-2022-containerd-20348.5139.260513, containerd 1.7.20, kubelet v1.35.4, accelerated networking with a Mellanox ConnectX-4 Lx VF. Each node was its own baseline, and the other looping node served as the control.

   Action on the node	Before (synthetic NIC rx / VF tx, DPC)	After	Pods
   Restart-NetAdapter on the Mellanox VF	15,797 / 15,817 pps, 3.2%	16,961 / 16,984 pps, 2.4% at +3 min. No effect	not restarted
   Restart-Service hns -Force, node A	16,270 / 16,292 pps, 2.2%	24 / 34 pps, 0% at +20 s; still clear at +3 min	not restarted
   Restart-Service hns -Force, node B	17,222 / 17,240 pps, 3.2%	16 / 15 pps, 0% at +20 s; still clear at +3 min	not restarted

   • The control node kept looping at full rate throughout node A's test, so the drop is not coincidental.
   • HNS endpoint counts were identical before and after the restart (227), and every pod kept its IP.
   • Cost of the restart: about one second of connectivity. We restarted HNS again on node B while probing a pod on it over HTTP from another node, every ~270 ms with a 1 s timeout. 436 of 437 probes succeeded in 12-38 ms. One probe timed out, starting 0.7 s after the restart was issued, and the next probe succeeded 1.3 s later. We did not measure in-flight connections across that window.

   A VF reset does not clear the loop, but an HNS restart clears it within seconds. That suggests the forwarding state lives in the rules HNS programs into the virtual switch (VFP), not in the NIC. Our inference is that on restart HNS reprograms VFP from its live endpoint list, and the torn-down pod's IP is absent from that list, so the stale state is not rebuilt. We would appreciate confirmation, or a correction, from the HNS/VFP side.

   Fresh nodes are not immune

   Our original assumption was that only nodes reused across deployments were affected. That is not the case.

   • Six nodes created the same day they started looping. They hosted pods for a few hours during a deployment and the pods were then moved off. All six were looping the next day, with little or nothing running on them. Every torn-down pod's IP was still among the node NIC's secondary IP configurations: 33-76 per node, all still present.
   • Two nodes created as replacements during our remediation showed a low-rate symmetric loop (about 1,400-3,000 pps, DPC under 1%) within hours of creation.
   • Teardown is necessary but not sufficient. A node with pods torn down on two earlier dates did not loop afterwards. Loops also sometimes decay on their own.

   So rotating onto fresh nodes each release lowers the odds, but does not prevent the loop.

   Additional capture detail

   A full pktmon --comp all capture filtered to one dead pod IP recorded 465,230 events. Every packet carried TTL 128 and IP ID 1, with zero drops reported by any component. The packets are TCP RST,ACK from the in-cluster ingress load balancer VIP on 443 to the dead pod's ephemeral ports. Because the TTL is never decremented, the packets are being switched rather than routed, and TTL 128 suggests the resets are generated on the Windows host rather than by the Linux ingress pods. Both points are inference. The pktmon ETL with counters, HNS endpoint/network/policy JSON, the VFP port list and collectlogs.ps1 output are available on request.

   Mitigation we are deploying

   A HostProcess DaemonSet on every Windows node that:

   1. samples the synthetic NIC's Packets Received/sec and the VF's Packets Sent/sec every 30 s;
   2. treats the node as looping when both are at least 5,000 pps and within 10% of each other. Across 138 production nodes, healthy nodes measured 10-500 pps, and looping nodes measured 15,800-183,000 pps with the two rates within 0.2%;
   3. after 10 minutes sustained, restarts HNS with Restart-Service hns -Force, waits 60 s and re-measures;
   4. backs off: at most one restart per node per hour and three per day. A loop that returns inside the cooldown, or survives a restart, is reported as an error and the node is replaced;
   5. posts Warning events on the Node and logs JSON, so the condition is alertable.

   This remediates the symptom, not the cause.

   Questions

   1. Why does the host forward, rather than drop, packets addressed to a secondary IP the NIC still owns when HNS has no endpoint for it?
   2. Is restarting HNS a supported mitigation on a production AKS Windows node? Is there a less disruptive way to make HNS reprogram VFP?
   3. Is there a fix, or a known-good node image or Windows build, that addresses this?