The colder it is, the longer I'd wait. Mostly to get idle rpm down to sane levels, so it doesn't bang into gear (most vehicles being automatics).
on my older diesel I do like to idle a bit, like 5 minutes at 10F or so. It's pretty sluggish until it gets some heat into it. But even then, about three of those minutes are me being inside and getting warm again: if I'm leaving work it's more like 60 seconds, even at -10F.
Just take it easy. It'll warm up faster with gentle driving, as opposed to idling.