I get pretty into my detailing.
Yes, silver (and shades of silver and gray) tends to stay looking best because silver closely matches the color of light bouncing off the super fine scratches and swirls in paint.
Personally, despite that, I don't love super light silver as it can look as plain as white. I prefer just a slightly darker silver with some good metallic.
I do have one black car. It doesn't really look dirtier, I just have to work hard and polish it a couple times a year to try to keep the swirls down. And it's small so that helps.
Pretty much though, in the future I'll stick with dark silver or gray cars, or some variation. I might have a black car again, if it was really small as mine now is.
I will say when detailing, the darker the color, the more exciting it is to work on. The depth is so amazing. The lighter the color, the less thrilling it is to look into the paint, as the depth is just not there.
I also find lighter colors tend to look good on cars that have beautiful panels/design/shape as you see EVERY curve of a light colored car. If a design doesn't age to so well, the light color can make a car seem more out of date. Black can sometimes make a dated body style or design look not so bad because it hides some of the curves/design, you instead see that deep black paint.
People think white is easy to work on, but it is not. It shows any little contaminant on the paint, it stands out, so the paint must be kept really clean. Also, white does not hide micro-scratches or swirls well, it's just a plain flat white paint, so swirls can pretty easily be seen. I don't enjoy working on white paint, it's not too rewarding. Very very little depth. You won't fall into white paint no matter how polished, that's for sure. On a beautiful car, brand new, I think it can look nice though, but really brand new and shiny and in style design, like a brand new Cayman or something.