I've always fenced mine off the 3 horse chesnuts we have at the bottom or our paddock. One of mine loves to try and get at the leaves, but I assume it is the conkers that are toxic as with acorns etc
Right done another search to try and find out how toxic they are. According to here - http://www.ttlntl.co.uk/2/Health/poisonplants.htm they are moderately toxic (Harmful if eaten in quantity), doesn't state what is poisonous i.e. leaves, bark, conkers or all part or what symptons/effects of eating would be.
Don't worry too much. My horses shelter under a huge horse chestnut tree, and I have never seen them eating the leaves or the conkers. I think providing they have something else to eat, they just ignore it.