In a tie allowing the straight person to go first is more efficient in the whole world view, because in a 4way, when you're going left, you also have to go straight for a bit first before starting your turn. So both can be moving into the intersection and there's less time where cars are at a dead stop.
So in a game theory score, together both of you combined get a higher score. Next time perhaps you are the one going straight and so you'd get the benefit.
For the case of the highway turnoff or intersection where the left turn doesn't go straight as the first part of the turn, you'd still apply the same rules because it's too confusing to start assigning special cases.
Overall practically we don't live a perfect world, 100% of the people aren't going to follow the same rules, that's why we have so many traffic jams and accidents that it's commonplace and not a freak occurence. That's why there is a wealth of knowledge in traffic design to investigate and prevent this from being a choice.
But, is it a big deal? Nobody really is affected so much that it becomes a headache, otherwise it'd be changed. But changing costs money. A traffic light is more expensive than a roundabout which is more expensive than a line of paint.
If/when THIS issue becomes the biggest problem in traffic and you get backups and crashes and jams; THEN the local government will prioritize to spend the money to design or improve intersections using the latest knowledge in traffic engineering knowledge to nudge people to do the most efficient thing.