A turbocharger is essentially a "free lunch". When you don't need the boost, you don't have to utilize it. Try getting rid of a larger engine's extra displacement (and all that comes with it) when you don't need it. Selective cylinder deactivation doesn't do the job, as you're still carrying around a bunch of dead weight and continue to suffer the parasitic losses. I think you're missing the point that while "X" quantity of air requires "Y" quantity of fuel, the volume of air a turbocharged engine uses isn't directly tied to the engine's RPM, unlike that of an atmospheric engine.
A turbocharged engine of equivalent output is already fulfilling the desire to reduce weight, as compared to the larger atmospheric engine, due to its reduced size. Then, of course, a smaller engine also produces improved brake specific fuel consumption figures versus a larger atmospheric engine. Also, a smaller turbocharged engine, due to its lower weight and reduced size, allows a more space-efficient chassis than can be built lighter.
Frankly, I'm not seeing how a smaller turbocharged engine fails to fulfill the "smaller and lighter" mission...