What you saw under the oil fill cap was the splash baffle i mentioned. It has a rough texture for some odd reason. I've even heard of supposedly experienced mechanics (altho not toyota) mistaking this for sludge.
The blue smoke (at startup) is valve stem seal leakage, much more common than sludge and fairly common on overhead cam engines made by anyone. As long as it doesn't get so bad as to cause plug fouling and you keep your oil filled within the proper range, you'll be fine.
btw, On Toyotas the range between "Full" and "Add" is 1 qt.
Moving to 10w40 slowed mine down a lot in my 93 i4 camry, you could also try some of the "high mileage" oils with seal conditioners, many have had good results with them.