A couple of days to be sure it won't heamorrage. After that, the work (gentle walking) helps the swelling go down. If you can't turn him out, because he is still fertile for 4 weeks or so, my vet told me to work mine. He was 4 when he was cut. Otherwise turn him out for a couple of weeks and work when the swelling is all gone. It doesn't take long.
Work after castration is vital in the healing process. A couple of days off and then walking for long periods followed by enforced trotting and cantering for a couple of weeks will get it to drain well and hopefully prevent infection.
I've just been through a Scirrhous Cord with my 4 yr old and it was hell.