mirror of
https://github.com/torvalds/linux.git
synced 2026-09-14 16:10:02 +02:00
451c9075d6
A TX timeout on a qede NIC that has VXLAN/GENEVE tunnel ports
configured wedges the rtnetlink control plane of the whole machine:
NETDEV WATCHDOG: ens6f1 (qede): transmit queue 2 timed out 10226 ms
[qede_tx_timeout:586(ens6f1)]TX timeout on queue 2!
[qede_recovery_handler:2665(ens6f0)]Starting a recovery process
The recovery path deadlocks on the driver's own mutex:
qede_sp_task
rtnl_lock()
mutex_lock(&edev->qede_lock) <- taken
qede_recovery_handler
qede_load
udp_tunnel_nic_reset_ntf
__udp_tunnel_nic_device_sync
info->sync_table == qede_udp_tunnel_sync
mutex_lock(&edev->qede_lock) <- same task: deadlock
The mutex is not recursive, so the kworker blocks on itself with
rtnl_lock held, and neither lock is ever released. Every task that
calls rtnl_lock() afterwards (ip, ovs-vswitchd, lldpad, IPv6
addrconf, sshd) blocks forever while the node still answers ping.
In a vmcore from an affected production node rtnl_mutex.owner
decodes to the very kworker blocked at the innermost mutex_lock()
above.
Re-sync the tunnel ports from qede_sp_task() after the internal lock
is dropped, still under rtnl_lock as the udp_tunnel API requires.
This mirrors qede_open(), which calls udp_tunnel_nic_reset_ntf()
under rtnl without the internal lock.
qede_recovery_handler() now returns whether it has successfully
reloaded an open device, and the caller re-syncs the ports only in
that case. This keeps the old gating exactly: a device that was down
or a failed recovery returns false, as those paths never reached the
udp_tunnel_nic_reset_ntf() call before either.
This was the only user of the qede_lock()/qede_unlock() helpers, so
remove them.
Fixes:
|
||
|---|---|---|
| .. | ||
| arcnet | ||
| bonding | ||
| can | ||
| dsa | ||
| ethernet | ||
| fddi | ||
| fjes | ||
| hyperv | ||
| ieee802154 | ||
| ipa | ||
| ipvlan | ||
| mctp | ||
| mdio | ||
| netdevsim | ||
| ovpn | ||
| pcs | ||
| phy | ||
| plip | ||
| ppp | ||
| pse-pd | ||
| slip | ||
| team | ||
| thunderbolt | ||
| usb | ||
| vmxnet3 | ||
| vxlan | ||
| wan | ||
| wireguard | ||
| wireless | ||
| wwan | ||
| xen-netback | ||
| amt.c | ||
| bareudp.c | ||
| dummy.c | ||
| eql.c | ||
| geneve.c | ||
| gtp.c | ||
| ifb.c | ||
| Kconfig | ||
| LICENSE.SRC | ||
| loopback.c | ||
| macsec.c | ||
| macvlan.c | ||
| macvtap.c | ||
| Makefile | ||
| mdio.c | ||
| mhi_net.c | ||
| mii.c | ||
| net_failover.c | ||
| netconsole.c | ||
| netkit.c | ||
| nlmon.c | ||
| ntb_netdev.c | ||
| pfcp.c | ||
| rionet.c | ||
| sungem_phy.c | ||
| tap.c | ||
| tun_vnet.h | ||
| tun.c | ||
| veth.c | ||
| virtio_net.c | ||
| vrf.c | ||
| vsockmon.c | ||
| xen-netfront.c | ||