This mostly happens during break in. That’s why I’m a proponent of dropping the pan early in the first 20k miles or so.
After that, the magnet will be good for a long, long time like 100k miles. No need to drop the pan just to clean the magnets. Regular fluid changes will be more than enough to keep the fluid clean.
As far as the pan goes, the particles are simply too small to stay at the bottom, they will always be picked up by the fluid. And if you have chunks big enough for the gravity to keep them in the pan, cleanliness of the fluid would be the least of your concerns.