Can you describe the residue? Does it look like a dull spot or more like a smear, like it's oily? If you rub it with your finger or a towel, does it comes off or change in some way? Do the residue spots feel tacky, like a smudge of clay is stuck to the car?
Try taking a damp towel (not sopping wet, not bone dry) and wipe down a couple of highly visible areas, places where it would be easy to notice if you made it look better or worse. What happens?
What I do as I'm claying is I clay a small section, then wipe it down with a clean towel while it's still wet with clay lube. Maybe what you see is dried clay lube which easily washes off with soap and water. Clay smudges happen sometimes but a wipe down with a towel or a normal hand car wash will easily remove them.
Try a few things and get back to us, we'll do our best to help.