I'm like the OP but I'm getting better.
Like most have said I would lean toward changing the oil first before the filter if I had to pick one.
If it makes you feel better, do like I do. Run a quality synthetic and a top of the line filter (RP, Amsoil EAO, M1) and go by the factory OLM. Since the OLM is calibrated for dino and a [censored] filter, you're more than safe with a good synthetic and quality filter.
I have to admit that I get nervous around the 4,500 mile mark and it occasionally gets changed early. After learning what I know about filters, I will actually go 2 intervals on the same filter when I was running the Amsoil filters and plan on doing the same with the RP.