I assume you event? Do you wear the breastplate during the dressage phase? If so then definitely black! You'll be going too fast during the xc phase for anyone to notice you don't match
I think I'd probably get the darkest brown possible so it sort of merges in with both. Though I can't talk... my dressage saddle and bridle are black, my XC bridle is brown with black grackle noseband, jumping saddle is one of those horrifically coloured (but comfy!) orange Henri de Rivell saddles, and I have a Havana breatplate