I just posted this on another forum earlier today. It basically addresses all of these "oil shootouts":
I am very skepitcal of these oil marketing "tests" and even most of the tests I see in magazines for the simple fact that proper dyno testing involves more than one or two runs. When my bike was dyno'ed several runs where performed just to establish a baseline, with no changes what-so-ever to the bike or the conditions. HP varied as much as the RP test showed by changing oils. If the HP can vary by 3 or more HP on a 95" Harley motor, with no changes, and if it takes a dozen runs just to establish a baseline that will allow for the before-and-after comparison of timing/jetting changes, then how can anyone say with any confidence that 3 HP (or whatever) was gained from one run to the next by changing the oil. If a correct dyno test was done there would be 10-15 runs made on each oil in similar conditions.
Another problem with the tests is that some of the old oil is left behind in the engine. Did the old oil coat the cylinders, bearing, etc. and cause an increase or decrease in power vs. if the new oil was coating the engine? I believe that Ester-based oils are polar and attracted to metal surfaces and will cling on better than Group I-IV oils will; that could affect oils that are tested after an Ester-based oil (like Redline). These types of tests seem to be very difficult to conduct accurately and would likely involve some sort of matrix wherein all of the oils are tested several times in different order. This would probably take days and hundreds of dyno runs. I doubt if anyone would spend the time or money to do this right.
One thing I have noticed in many of these tests (Joe Gibbs comes to mind) is that a thinner oil is being compared against thicker oils. This seems very deceptive to me. I've learned over the past couple of years that thinner oils are not always what they are cracked up to be. Some engines actually do better with a thicker oil and produce better fuel economy, which normally translates to more power also. My theory is that ring seal is affected by viscosity and I have found that I lost .7MPG in my car (comparing 8,000 miles on each oil viscocity) from switching to a thinner oil. I've also found that my Suburban 5.3L and my Road King are quieter with a thicker oil. On the Suburban, no fuel mileage was lost. On the RK I lost some MPG on short trips but actually gained some on long trips-my theory is that, when cold the oil is causing some pumping losses but when hot the oil is sealing the cylinder/ring better (just a theory, I know).
I think the bottom line is that there may be some slight HP differences between oils, but to find out would be a long and costly process. The fewer oils being compared the better. For example, you could test Amsoil for several runs to establish a baseline one day, then change the oil to Redline and run it for several hundred miles, or more, then dyno test that for several runs to establish a baseline. This would not be an exact science because it will introduce other factors into the equation. Of course the factors would need to be adjusted due to changes in temperature, humidity, barometric pressure, etc.
Any "testing" that involves one dyno run on each oil is complete c**p IMO.