Giving the caliper bracket a rough wire brushing and maybe more is necessary because the pad "slants" the wrong way while the rust is there, not in line with the factory tabs. The shim conforms with the rust.Would using SS shims help the pad "align" with the rotor?
Caliper can be checked for ease of compression. Rotors, I actually like Toyota OEM because they stay in tolerance even after going below the legal limit. But if rust is this bad, then maybe there is an aftermarket with more coating or flair than Toyota OEM.
Pad slap can work for a while, but OP appear to be in a salt heavy northern state. I'm in the mid-Atlantic, so things don't look like his caliper bracket even after 20 years and some salt.