Like I said, I left the Case dealership several years ago- so some things may have changed. But at the time I worked there, most of the skidsteers used 10w30 motor oil in the hydraulic system, along with 1 quart of Case additive for every 5 gallons of motor oil. You might have a look in the operator's manual to see which oil it calls for. I don't think this would cause your overheat issue, but you'll want to make sure you have the right oil in the hydraulic system for obvious reasons.
Are you sure the brushhog is designed to be used with the high-flow hydraulics? The 'jet airplane' comment makes me wonder if it's spinning faster than designed, which would explain why it 'tears up often'. Also, when running a low-flow attachment with high-flow hydraulics, you could very well be running at or near relief pressure, which would explain the overheating. If you have a manual on the attachment or a local dealership that can look up the info, you'd do well to check the brushhog's flow requirements and compare them with the flow rating for your machine's high-flow system and the regular auxiliary connections.
Also, considering your attachment, I think it's pretty likely that your heat exchangers are getting clogged with grass. Don't know exactly where they put the condensor on this particular machine, but if it's bolted right up against the radiator and/or oil cooler (which is common on skidsteers, but there are other places it can be mounted), then that's a good place for grass to pack in- between heat exchangers.
I had a customer a few years ago running a brushhog on a John Deere 270 skidsteer. I don't remember for sure, but I don't THINK it required high-flow hydraulics. He was having constant overheat problems due to grass getting packed in the radiator fins and between the radiator and the oil cooler. The grass would be sucked into every gap and crevice in the machine so that after only a few hundred hours, the entire belly pan was packed full of grass clippings. I sealed every gap I could find in the cab, but the grass still found its way in after a few weeks. The guy finally got tired of paying the $80/hour labor rate to have me clean his machine out, and started running a brushhog on his utility tractor rather than on the skidsteer.
So IMO, a skidsteer is not an ideal powerplant for a brushhog. It will work, of course. But with all that grass flying and with a complicated machine with multiple heat exchangers following BEHIND the mower, you just can't avoid packing it full of grass.
So yeah, to sum up: Check for grass in the heat exchangers and belly pan, and verify that your hydraulic flow is correct for the application.