I would also move. You need to be able to fetch your horse in at any time, not just when the others are not in their paddock. If you have already been kicked you were lucky not to be seriously injured.
I've been around horses for decades and I would not do what you are expected to do. Along with the unacceptable management problem of hungry horses in a field, your YO is turning a blind eye to the welfare of the horses and liveries. Get out of there quickly, find a better yard and enjoy your horse again. Your horse doesn't deserve this stress either,