I had and still have a few of those. I prefer 2004, but they all have strengths and weaknesses. As for the manifold issues - all years require this work, different stats though. They had all-polymer manifold through early 2002, then replaced it with partially aluminium one. Still may give you a headache. I had one 2001 that was the best one I ever had out of all panthers. 2005 has this whip antenna, which is a major turn-off for many, not for me - a good negotiation point btw. All-in-all, surprisingly reliable American car. There is crownvic dot net congregation out there, they will get you more points for micro-management. The major thing: you do shop the owner, not the vehicle description. There are plenty of panthers out there, take your time and your patience will be rewarded.
[for whatever reasons, Lincoln Towncar of the same platform has been consistently the most reliable in my case.] I had a fleet of panthers with 360k miles minumum on each of those. Rack-and-pinion steering started in 2003, the more dependable transmission ended in early 2005, the same year introduced fly-by-wire throttle, there were all kinds of funky changes later on, icluding going back and forth with the oil filter types, etc. It is all BS, the platform is an old-fashioned and refined mass production vehicle that can easily go for 500k miles with regular care. There is still plenty of great and affordable garages that cater to cabbies... the old timer talent is still there. Don't make the project of finding and buying a torturous one, make it fun. As Romans used to say - festina lente - hurry slowly and good luck!
added: 2005 is much easier to work on when it comes to the rear shocks, but it is not a deal breaker. If I had to shop, and the 2002 had a better owner, I'd grab that one... and vice versa. Shop the owner, the best is the quintessential old geezer who cannot drive himself but still took impeccable care for his wife. I have an MGM where the guy would go by the book: if it had only 200 miles, but the 6 months were up, he'd take it for the oil change... things like that malke a huge and demonstrable difference. That one literally drives like new.