In terms of why he is smart, we see this first in terms of his renovations to the shed. He puts basically a foot of soundproof padding in, along with all the necessities for daily life and some more including a tv. He seems to take into account that the mental health of the person he kidnaps has to be sort of stable or none of this would be worth it, but we don't really know his motivations. The garden shed also has the fence so that you couldn't cut your way out of it and other protections including the door lock.
In addition to the physical restrictions that Nick has placed on these people, he also has a strong mental grasp on Ma and many cards to play if they try to deceive him or revolt against their capture. He can starve them, cut off the electricity, and not bring them and food or other necessities. He is also much stronger physically than the undernourished victims as can be seen in how Ma's wrist gets broken after she tries to overpower him. With Jack in the equation, the threat of him getting hurt is also a card that Nick can play and something that Ma is very worried about. He never really has to play this card, but the fact that it is there is such a large factor in how he keeps them in check. Ma is in such a state that she basically needs Jack to stay mentally functioning, and although Nick probably doesn't want her in a suicidal state, shes want Jack to stay alive and with her more than she thinks about Nick objectives and the fact that Nick would probably not hurt Jack to a very severe extent.
All of these above factor coupled with Nick also trying to get favor with Jack in some ways, as with the jeep, paint a very complex picture of the victims relationship with their captor. Although Nick is definitely in the wrong, he seems to be able to think through enough that he doesn't get caught until Ma uses Jack to outsmart him.