I would make it a point to switch drive modes when first getting into the car.
I also think that the oil system is still circulating while in EV. I don't know for sure, and this could be vehicle-specific, but my wife's car, I do hear some sort of pump running.