What year is the Camry? I know the 92-96 Camry with the 4 cylinder has a separate fill for the differential for the automatic transmission. I'm not sure about the V6 but I think they have the same transmission.
It is located right in the center where the front axles joins in. Its easier to see from underneath the car. The drain bolt faces the ground and uses a large hex key to loosen. The fill plug faces the firewall and I think it was a 17mm bolt. It takes the same ATF (Dexron III) as the automatic transmission, ~2 quarts. Make sure you fill at level ground. You just fill it until it drips out of the fill hole. I just use one of those pumps you attach to the bottle to fill it.
I find it easier if you remove the air box assembly to free up space to work on so you have better access to the differential fill plug.
It's possible the differential does not use the same fluid as the transmission housing in the Camry. They both take Dex III or equivalent but depending on year and model, some have a separate drain and refill for each.