I agree. Trousers and shirt and make sure you wear flat shoes. If they give you the guided tour the last thing you want it to get a heel stuck somewhere.
Are you likely to be riding as part of the job? If so wear your jodhs. I went for an interview at a yard years ago and my Da told me not to wear my riding gear, so I went in jeans and the woman was annoyed cos she wanted to see me ride and wouldn't let me ride in jeans...