I find this debate fascinating because no one ever considers HOW you'll be doing it -- or rather, operates on the assumption that the job will be performed as well as humanly possible.
If you do it properly and use a quality seal, yes, replace it.
If you rush through and claw it out with a screwdriver, scratching the wall or the crank along the way, skip it.
Because I'm paranoid I never just "eyeball" seal depth unless it bottoms against a seat. Some float and then I find a way to make a tool so I drive it flush with a surface.
If you envision how a seal works, being cocked even a couple thousandths gives you an oval or ellipse trying to seal a circular crank.
So do it right and do it carefully, or let sleeping dogs lie.
edit: depending on cost I'll often order TWO seals. Having a second guarantees the first one goes in well. But if I don't have a spare, I'm left thinking I don't like my first attempt and wish I had another for a second try