I would strongly recommend you get the help of an instructor, as everything you do now will last the pony for the rest of his life. Can you afford some lessons, and do you have a regular instructor, or one that someone can recommend?
There is no point rushing in and just jumping him, you really need to be doing some pole work, followed by grid work.
It sounds as if he has lost confidence rather quickly, so slow down, and go back to basics.
As an instructor, I would start you over trot poles, to get his rhthym and your approach right, then move on to canter poles, and then cross poles with placing poles - all takinga couple of weeks. And all this needs to be correct, you need to know your distances and not rush him on to the next stage until he has got the basics right.
All this will pay dividends, but you need to be patient and get some help! Even if he has been jumped over single jumps already, you cannot go wrong if you start again from the basics and build him up. I promise you this is the only way
You need to get yourself a good patient instructor who can give you the correct advice after seeing your horse & you ride. Any problems can be assessed & suitable exercises & instruction can be given. To take random advice from a forum from people who haven't seen you or your horse jump, albeit meant well, could cause you more problems. An instructor is the way to go....good luck.