Humidity makes all difference.
I would much prefer 100 with 15% humidity than I would 70 with 95% humidity. Same can be said for when it's cold...basically it makes you feel colder when it's cold, and hotter when it's "warm".
I do a lot of recreational activity just south of Houston, and my god you're dumping buckets of sweat at 6:30AM in the summers. Winter time even 40 degrees can be brutal.
I recall recently meeting up with some people (in Houston) who had driven straight in from North Dakota for the event. We were all freezing our asses off in low 40s temps, and 20ish mph winds. They indicated it was worse than the negative temps they'd left behind at home; I don't think it was trying to placate their Houston hosts either.