The passenger was drunk.
The driver was reluctant to drive him due to said drunkenness and the low likelihood of a tip.
The passenger may have been at a designated taxi pick-up line at a bus or train station such that the driver couldn't refuse.
BONUS: Maybe there were more traffic delays ahead so the driver REALLY DIDN'T want to be there (pure guess).
The difference in size between the driver and the passenger may have contributed to the driver's "activation threshold".
The passenger looked like a dippy-doo. He almost looks like the main character from the RockAuto cartoon TV commercials.