The worst thing for a horse is the wind & rain, the wind moves the coat allowing the rain into the skin area. If you rug then you could make the situation worse as the rug flattens the coat preventing it doing its job (as already mentioned) of warming the air. If they are well covered cold is not a problem on its own, cold winds or wind & rain is.
So odd that people think that means the horse is cold and not that the horse's coat is doing its job properly! Rugs flatten the coat so no layer of air can be trapped by the horse so they heat the layer of air in the rug instead. So effectively you are opening a window then turning the heating up to compensate if you rug a horse which was previously fluffing its coat up.
If it is cold and wet then I agree rugs are needed, but when it is cold and crisp as it is at the moment, I see no problem with them being unrugged. Their coats are far better designed than any rug you could use!