Welcome to the forums .
The quickest to Drive is through Swaziland but as you said you don't wish to go through then Might I suggest you also consider the N11 ladysmith route .This was full of roadworks until recently I have done all four routes in the past 4 Years and did N11 Volksrus ,Ermelo ,Carolina last in January both ways and the roads were good when we traveled towing the caravan .
If you want to do north roads would suggest the N2 Pongola route but it is slightly longer route .I last traveled this route 2 years ago and it was also fairly good but there were /are some roadworks along the route last time I checked with stop goes and if you get on the wrong side of the stopwatch at those it can add hours to a trip .
We will be going through Swaziland the next trip (depends on a number of factor including wheels bit that's all in another thread ) .Basically need passports ,vehicle papers and list of valuables .No meat products and they fairly quick to process through the border posts .Haven't done this route for about 4 years due to the whole kids needing full birth certificates etc but our friends always travel this way without any issues apart from some short stretch around Siteki with some potholes (also in January they did it both ways) and our youngest rugrat is older than 18 now so that falls away .
Let us know which way you decide to go