One potential problem I see is that if your neighbors are into gardening, they're likely to have had the same type of things do well that particular season. So

One potential problem I see is that if your neighbors are into gardening, they're likely to have had the same type of things do well that particular season. So sure, if your neighbor has some apple variety you don't, you can swap. Or your broccoli for their cauliflower or other Brassica. But it might not mean that much for variety in the end. But giving food away and getting nothing in return is preferable to having it spoil, obviously. And every situation is different.