Vacuuming may get the bigger pieces out of the way, but it seems to me that part of how a paper filter works is the some of the particles get lodged in the holes and gradually block the airflow. You will never get them all back out of the filter, as they have become mechanically lodged and a simple vacuum wont have enough power to pull them back out.
This results in a slow but steady drop in airflow. This will probably be slow enough you wont really notice it, but at some point it will reduce efficiency, power, gas mileage. You probably wont hurt anything, but the system just wont be working as well as it should.
Carbureted engines will probably show the effect first, as modern engine controls should automatically compensate and keep the air/fuel ratio correct. I have seen engines in the 60;s with filters so plugged they were obviously running rich, along with the poor performance that went with it. But those people just never did any maintenance. They would never have bothered with vacuuming, since they couldn't even see their way to do a simple replacement.
Now it must be said. Is that effort worth it for just a few dollars and the best performance?