If you are in control and not a danger to anyone or yourself I can't see why anyone should bother.
If horses are very uncontrollable, barge into others, generally cause mayhem, they can be sent home, but it depends on how tolerant the Master is. I think everyone has suffered an embarassing moment or two at times, so one out of control moment shouldn't get you banned, but if it was constant and annoying others then someone would have a word with you.
You could always say that there is a veterinary reason for no bit if you don't want to get into a debate.
As long as you feel you have sufficient control then there is no reason why you can't go bitless! There are one or two people who hunt in bitless bridles with us, and they seem to have no problems at all.