No, you most certainly are not! If there are a few different clubs within distance ask which may suit you more as they do vary but you really should give it a go My last riding club had people from 16 (well it had junior members too) to late 60s
I've been in AIRC for years here in the south and have seen all ages/sexes/shapes and sizes. There is a very big over 50s contingent here (me and my mates included) and I have seen people riding and competing at riding club activities well into their 70s. My own club age range is 18 to 65.
The most important thing is to find a club that shares your horsey aims and ideas where you will feel happy and comfortable. Some clubs are very competitive while others just want to be social.
I am 52, there are loads of people in their fifties competing locally to me. I would love to join a riding club but sadly have no transport.
Go for it!!