As long as the RPMs have come down to a reasonable level, you can feel comfortable driving away moderately after just a few seconds. But most cars take longer than a few seconds to complete the cold start cycle. Just don't rev it excessively until the engine is near or at full temp. After a cold start, I wait at least 30 seconds and then drive away normally. By the time I reach the on-ramp, I'm near full temperature so it's safe to rev it.
Tribofilm was mentioned in a previous post, tribofilm is bonded to the metal wear surfaces and this is an excellent point to bring up in the conversation. Also there is always a thin film of oil on all the engine parts, even days after shutoff, unless it has been sitting cold and un-driven for 6 months or longer.