Looking back at all the cars I've driven, it wasn't about the horsepower that got to me. It was about handling and perceived quickness. The car has to respond quickly in acceleration, exiting corners, braking, etc.
The most fun cars I've driven were an 89 Renault R9 5sp with nice suspension, 1980 Corolla 5 spd, 1984 Dodge Lancer Turbo, and my current Jetta MK4 2.0 115hp 5sp daily driver.
I like my Jetta. It's quick enough at stop lights because it's a stick, and fast enough on the highway because the gearing is low. I'm basically using all the horsepower it has in my daily drive while it keeps me out of trouble and jail.
Other cars I've driven were...
1980 Chevy Impala, V6
1987 MB 300D, 123body
1984 MB 280TE, sweet inline 6
1987 VW Scirocco, got me hooked to VWs
1995 Lexus LS400, fast and smooth yet numb
1994 Cadillac Deville, floats like cruise ship
2009 Nissan Versa, puts me to sleep.