I don't see a problem with it. I'd be quite shocked if any of my horses reacted to that. Given that they are riding horses at events where people have umbrellas, newspapers and are clapping and cheering etc you'd be amazed to see a woollen poncho causing a spook surely?
It's not as bad as the flimsy waterproof hoods that tuck into rain coats is it? I regularly lead mine with one of those flapping about on the back of my coat in the wind and would be most surprised to see that upset any of mine too.