Polaris recommends chaincase fluid. ATF is of similar viscosity. Motor oil will probably work ok, but it's a bit thicker than ATF or chaincase fluid, so it won't get into the layers of the chain as quickly. ATF is used on a lot of 4x4 transfer cases that are gear/chain drive with no issues.
I use chaincase fluid, but have used ATF in the past (on a Sunday when I wanted to ride and had changed the gears on a sled). I used ATF the rest of the season with absolutely no issues. You would probably have the same success with motor oil. Some guys even use synthetic gear oil, but I think that is a bit thick for a sprocket/chain system like a chaincase. As long as you have something that lubricates in there, I don't think it really matters a whole lot. You just don't hear of many issues of chains wearing out because of the wrong fluid in the case. Checking the fluid, making sure it's at the full mark, and changing it once in a while to get rid of contamination go a long ways to make them last a long time...
I think if it works good on a sled should do well in ur ATV.
I actually use Mobil synthetic 5w30 motor oil in my manual gearbox in my car, it works really good too. I think regular gear oil would be too thick. ATF, or Pennzoil synchromesh, or just 5w30 synthetic motor oil, would be fine. Napa has a sale on NAPA synthetic, just 3.29$ a quart.
______________