What is your turnout like? Is it large, pasture with hollows/hills - or flat agricultural? Can you try him in a barn/large pasture arrangement? The fact he weaves and stresses when horses move suggests he isn't quite happy. Sometimes unless you have experienced horses living out in truely proper...
Another thought- build a shed for the kids to play in, with heater + kids stuff at current place? Then no car, they've a safe, warm,lit place to be and you have your own place which in 5 years time may feel more appealing! Could you also consider a trial month for kids pony at new yard?
Could you get anyone else in to your current set up for company/help?
Without the children - I'd stay at my own yard.
With the children i see the attraction of being somewhere with other kids - makes it much more fun for them. What age are kids are they anywhere near old enough to be left at...
If you would consider further work up I would ask to be referred to a hospital with an equine medical clinician as they will get you the best chance of a recovery . Low protein is not a great thing to have.
Your latest update also sounds like you are scared of him - this + an excited or unhappy 5 yr old has massive ramifications. I'd suggest see if you can weedle out some local suggestions of a good young rider ( cheaper than pro) who is good at riding youngsters to come ride 2 times to see how he...
Number of possibilities:
Not enough exercise - 16 hours stabled is a fair bit. Does he need more work?
Ulcers - stabling + recent surgery- high risk for ulcers
Other pain
I think you need a competent vet to check him for pain +- scope or trial ulcer treatment + check saddle.
None of which are...
I thnk I would sit down and have a chat - but maybe approach it as you've really enjoyed the loan and wanted to check in plans for the next 6 months/winter
What do you want to do- do you want to ride on your own?
If you said to her you feel too big on the pony - do you think she might stop you...
I think the difficult bit is- if they weren't your in-laws - where would you keep your horses?
I would go into this very carefully. Are you planning children/got children? You need to present them with a reason other than 'I don't want to pay you anymore' as that is hard for anyone to swallow...
I'd suggest taking a step back. This pony has only just landed with you. If there are no wounds or painful sores identified yet, I would slow down. Pause and think - how much past his comfort zone did I go? It's clear from this post that you are aware of building up confidence in tasks. Think...