@BigJohn
Former Hydraulics engineer and hydraulic system designer...
Several good answers here, pumps (especially gear pumps) do make some contamination. There is some left over from machining, that is also correct. There are some wear metals in the "clean" oil added to the system. The hoses are a pretty big contributor both when they are first put on dirty, and as they slowly erode due to fluid flow. But...THE MOST MAJOR source of contamination was right there in your statement...
"controls to the cylinders"
Those cylinders are exposed to dust, buried in water, shoved into the dirt, etc. The seals on those cylinders are not perfect. Cylinders are the #1 contributor to system contamination. Clearances in low leak valves and axial piston pumps range in the 5-30 micron range...these are precision machined components. ANY contamination can cause wear and damage.