I've upgraded the machine I am using since my last post, and simplified my network setup... it doesn't happen often, just occasionally.

I'm not running on a rpi. It was previously a VM and now it's a LXC container. Host is a 12th gen nuc with 32gb of ram with a gigabit nic.