If the alternative is a summer of single or no turnout then I would just risk it. The flies aren't too bad here yet, they will only get worse. If your vet thinks its OK then I would crack on.
My friends stallion was gelded around this time a few summers ago when the weather was very damp but hot, much to the displeasure of the stallions breeder/wider circle of horses known people. Vet is notorious for doing anything for a buck and no other vet would do it. He was found dead two weeks later. TBF it scared me off gelding so much i still have a stallion :/ lol