If the stuff is coarse enough to feel gritty, it may be material deposited on the bottom of the oil pan, and flushed out with the oil. Think about sandpaper. 325 mesh grit is about 29 microns. A 10 or 20 micron filter will let anything much smaller than that through.
What weight oil are you using? Nobody seems to have good figures on how much the bypass either in the filter or engine opens. When the bypass is open, there is no limit on what bypasses the filter. The heavier the oil, the less of it goes through the filter. A highly efficient filter may also bypass more.