Never owned a goat YET but we are just looking into it so am a book expert at the moment.
I initially looked at the pygmy goats to keep on my yard with the ponies (seperate housing) but was advised that with horses I would be better off with something like anglo nubians,pygmies are apparently the houdinis of the goat world and all goats are pretty adept at escaping and if you are wanting to keep with horses you are better of with something a bit bigger as a pygmy is more likely to be mistaken for a dog or something and is at risk of getting kicked.
Other than that from what I gather they are the same to keep as any other goat,they are browsers not grazers so will eat the brush round fields in preference to the grass,
They do need warm dry shelter at nights or in cold weather.
You need a DEFRA number to keep or transport them.
They need feeding on hay and special goat mix.
They will escape given half a chance so your fencing needs to be up to standard.
The pygmies I have seen for sale have been around 120 - 150 each so not really cheap.
Hope I have helped a bit,like I say mine is book knowledge at the moment.
You will need very very good high fenceing. they can get under it too! they also hate the wind and rain so need good shelter, Goats are really great but you do need a sense of humor to keep them they get in to all sorts of trouble!
Thanks for everyone's advice so far. Sadly they are not for me as I am asking for a friend. Don't suppose anyone knows anyone who wants a sheep as a pet. Must be DEFRA registered though I think he said? It's a castrated ram, and will be free to a good home. Pm me if anyone is interested. I would have him myself but my horses are on a livery yard now.