At the moment he still likes it when he's actually there, but between that, he says this experience have ruined his love for horses. Don't wait with putting a stop to this until when he no longer likes it also on the days when he is there.
To only take one example, you've said he's been cleaning other horse's tack, not just his supposedly "share" pony's tack. Remind him about that what he currently does, do not equal the actual amount of work of looking after only 1 pony, because then he would only need to keep his pony's tack clean etc.
Even if you don't get his own pony, surely it would be better to travel further to somewhere else, where he might get less time but better quality pony time, than to continue as it is.