May have been a matter of what they were coming from.
A lot of traditional American car buyers bought MB diesels in the late seventies and early eighties.
This was a time when even higher end buyers were concerned about fuel economy, and a diesel Benz gave you both better fuel economy and more snob appeal than any higher end American car.
Besides, only pimps drove Eldos or Mk Vs.
The diesel Benz did have diesel clatter, especially at idle and was never a silent car, as a Caddy or a Lincoln, or even a Caprice typically was.
The Mercedes also had a hard mechanical note to it, unlike any traditional American large car of the era.
I always though our Benz diesels sounded good, but I had a different frame of reference from many who bought them.
I'd not be surprised if many buyers assumed that there was something wrong simply due to the lack of power.
Even the 300D Turbodiesel in the 123 was not really energetic and was even less so in the short wheelbase S-class in which it was also offered.
This was a time period when 80% of Mercedes' US sales were diesels.