Totally get where you're coming from on not liking to wear a body protector, OP - I have a lovely racesafe, which is ten times better than any other BP I've ever had, but I really don't wear it on the flat or for side-saddle because it's just... clunky.
I mean, I wouldn't jump astride without it, and I tend to wear it for fast hacks astride, too, but I have a wonky knee that doesn't get on with cross-saddles well; so I always feel a little more likely than most to part company with my steed.
Either way, it's entirely your choice whether you always/never/sometimes wear a body protector, and I know the BHS centre where I keep my pony only insists on them for XC (and they're strongly advised for small kids, I think, but I don't see much of that side of things). Don't let 'em get you down!
Entirely upto you, I wear mine depending on the situation - unknown or young horse then BP all the time, otherwise only for XC or if on a slightly dodgy jumper.
What I will say though is that the fit on the more modern BPs is completely different to the older styles, especially on the one's that are designed specifically for women. I love my Airowear Outlyne - it even fitted comfortably with super pneumatic breastfeeding boobs, but I know everyone is different and an alternative may fit better.