Interesting thread, I personally believe that we should wear Hi Viz for the following reasons:
It to a degree cuts the legs out from under a driver who argues -" I didn't see you!" if you can prove that yourself and horse are suitably Hi Vizzed every time you ride out. Although I do agree that some people think that because they are wearing a dirty pink tabard it makes it safe to ride out on unlit country lanes in the dark. There are two components to hi viz - the visibility of the material in daylight/poor light and the reflectiveness of the hi viz when an external light source illuminates it (car headlight).
It enables the farm workers to see me when I'm within the yards farm/fields and makes life simpler in that I don't come face to face with a combine harvester on a narrow track or that they can refrain from turning on agricultural machinery until I'm clear.
It allows aircraft to see me.
If the horse and I part company he can be seen.
Should I be injured I can be seen more easily particularly if Air Ambulance etc are involved.
I often ride down a byeway it allows other users - mountain bikers, 4x4 off roaders and Trail bikes to see me and plan accordingly so we can share the space with minimum inconvenience to each other, we often get cheery comments from trail bikers, who have pulled off the track and are waiting for us at a wide part that they could see us from miles away.
With regards to it being a yard rule, you might find that it is part of the yards risk management and mitigation plan that helps reduce their insurance cost and in turn our livery cost. I rode at a yard where the rule was body protectors and safety stirrups at all times because the owners had had to fight a costly legal battle over a personal injury claim brought by a client and the only way that they could get future insurance cover was to make them mandatory.
I would welcome Hi Viz being a requirement for insurance purposes, it shows that you are behaving responsibly and are acting reduce a known risk.
I understand that people from a different generation might not like hi viz because they didn't wear it as children, but the world has changed and we need to take responsibility for our own actions - This is a bit like people arguing about wearing car seat belts being an infringement of personal liberty despite the obvious benefits