I would get your vet in straight away and without any further delay. Lameness indicates pain. With every respect a farrier is trained to shoe a horse not to diagnose lameness.
If there's no obvious heat or swelling then the problem is quite likely to be in the foot somewhere. My boy is currently in his 4th week of box rest. He presented lame in his near fore over 3 weeks ag but with no obvious signs of trauma. We rested him for a week and he wasn't much better so we got the vet out. He was nerve blocked, which identified the foot and then x-rayed a few days later. There were no changes thankfully so it looked like a strain after a hot air balloon crash landed just behind his paddock. He is now sound in a straight line, but still a bit off on a circle.
To be honest, I would get the vet out as well as your farrier - there may be a small strain in one of the foot ligaments and the fact that he's been brought back into work but have just aggravated that problem before it had fully had a chance to fix. On the other hand there may be underlying problems which a knock or pulling a shoe could have aggravated and therefore flared up. Don't write him off yet - I have an ex-racer and have tremendous fun with him and this is the first real problem we've had in 4 years (touch wood).
Thanks everyone - for all the advice. After the farrier visits on Monday I will call the vet. My daughter thinks he is just footsore as he doesn't like to walk without shoes. So we are keeping him in the stable to keep his foot clean and out of the mud.