I went through this exercise with my Suburban. 4 new tires, rode like poop, turns out 3 were bad. I had to go through a total of 12 tires before I got an acceptable reading. This was via Tirerack and they were very good about supporting me, and paid for the extra labor. They even called Hunter and had them come out and check the calibration on the road force machine (it was fine). The tire company rep called me and asked for numbers off of the tire , and issued a quiet recall and pulled off a few hundred tires from the market. I was never told what the specific problem was, but obviously some manufacturing line defect. I seem to recall that Passenger tires had a limit in the 20's, and LT tires were 80. Above that they were considered unacceptable.
I have to say I was very pleased with Tirerack, because in hindsight I have run into this problem before, and the tire shop would not do any more other than rebalance the tire using a regular machine, even in the case of a major out of round problem. Another time, I had a major argument with a tire shop long ago where new tires were going flat. I kept taking them back, they kept "fixing" the problem. Normal driving was fine, but sometimes when the car was parked a random tire would go flat. The last straw was when my wife was driving the car and the tire went flat while parked at work. She did not notice until she started driving that the tire was flat. She had somebody help her to put the spare on, but the flat tire had damaged sidewalls. They were really fighting me on replacing the tire, and insisted somebody was letting the air out of my tires. I pointed out the 3+ returns I had about the tires going flat, and insisted that they replace the tire on their dime, and fix the flat problems. They kept it for 2 days and monitored it, and called me back and said "we figured it out, replaced your tire and rebalanced all the other tires". The manager gave it to his most experienced tech and told him to either prove I was wrong, or figure out the issue. He finally realized that the initial guy who installed the tires used the wrong wheel weights, and every subsequent tech that looked at it just used the same style weight. The prongs were too long, and cause slow leaks at the tire bead when the tire was at rest with the weight on the bottom.
Another time- tire had lots of high speed vibration. I kept bring it back to be rebalanced. They would do the work, but it would never be right. I finally insisted on watching them balance the tire- it needed about 3 oz of weight in one spot. I insisted that they take the tire off the machine, put it back on and spin it again. It was still out of balance. The results kept changing and he wanted to just do a balance once and send it out, and I refused, so I went and complained to the manager to get somebody else that was experienced to review it. The next guy checked the changing balance results, then proceeded to bounce the tire off of the floor. He heard something and said "something is inside the tire". He must have had really good hearing, better than me. He dismounted the tire from the rim and found a 3 oz truck weight inside the tire. Whoever mounted the tire initially either did not check the tire and resulting balance, or purposely tried to mess with me.