Hurray, I found the problem!
After reading http://stackoverflow.com/questions/11812731/first-udp-message-to-a-specific-remote-ip-gets-lost I made some more traces, and the problem is that my box loses the arp entry
of its router. In a trace, I can see ARP requests going out for the address
of the router that are unanswered. Then, about three seconds later, the box
stops sending packets. Seven seconds later we do get an ARP reply, and after
that traffic flows again.
So it’s not in Asterisk or in my box, but in my switch or router.
Thanks for your suggestions everyone, and have a nice weekend!