Is your horse thin? Old? Sick? On supplements? Nursing? What is your pasture like?
If your horse (and you know him best) isn't losing weight and/or condition, and is managing the workload then in answer to your question I would say 'No'.
Each horse must be treated as an individual though.