Well, the original and probably most collectible of all fast/inexpensive combo would have to be the '68 Plymouth Roadrunner. Very potent 383 in a light (for its time) body with the cheapest bench-seat interior, rubber floor mats, radio delete, no A/C, and even heater delete as an option. And it LOOKED great, too!
More recently, I've got a soft spot for the FWD turbo Mopars from the old Omni GLH and GLHS through the Shadow and Sundance. Never really wanted to own one, but any 2.2 liter engine that can make a minivan run the quarter mile in the 12s then get 25 mpg on the drive home.... wow. Then there's the much more civilized Neon SRT-4, too.
Gotta give a nod to the 80s Turbo Buicks (Regal, GN, GNX). The latter weren't exactly "cheap," but they were the last cars ever built to meet the original Muscle Car forumla: Mid size, rear-drive, 2-door, ungodly fast.
And then there's the 80s- early 90s Fox-body Mustang. Pretty much the 'Swiss army knife' of Motorsports. No turbo, nothing complicated, just a strong 302 in a light chassis... and ENORMOUS aftermarket support.