You have NO OBLIGATION WHATSOEVER to these callers.
You don't have to answer any call your phone gets ! You certainly don't have to return those calls either.
I pretty much never answer my mobile phone unless it's family or friends calling. Even if a business that I know calls, unless I've been in recent contact with them, it goes to voicemail. Reason for that is too many calls have "spoofed" caller-ID so often it's not actually that business calling me.
Do this: Start answering those calls and then you'll reconsider answering them in the future. As for calling them back, 99% chance it won't work anyway. It will be a fake number or the number of an actual business and they did NOT call you in the first place, their number was spoofed.