Cold horse

His fur will stand up a bit - they tend to get a fluffy appeareance, they look tucked up through their hind quaters and often clamp their tails down. He will probably carry his head lower & will generally look miserable.
 
Everything Panda & AmyMay have already said!

Try feeling his/her ears - though the tips may be cold, if you place your hand around the base, you should be able to feel warmth coming through - if you can't, the horse/pony is certainly too cold.
Similarly, is there is little heat coming through the body when a hand is placed firmly against it & held there.

Something brought into a stable all dejected & miserable from a cold, wet field should start to look comfortable & happier almost immediately. If they do not do so and/or have not started eating their forage etc. virtually straightaway, they are pretty darned cold indeed!
Horses cope quite well with the cold generally but unless native ponies, it's the wind & the rain that really gets to them!
 
Top