It's the done thing to wear tweed jacket with a navy velvet hat, although my friend wears a forest green one and a have seen some goldy beige ones. Never seen anyone wearing a black one except at fun shows.
Black hat is fine. Plenty at county level wear them. Nothing in most society rules to say you can't! My hat with a strap is black and my patey is navy. Wear both at all levels, never had any problems.
As long as you look smart and the overall picture with your horse is good then that is fine!!