Oh yeah, only Google isn't really "listening" in the traditional sense. It's scanning for trigger words, not saving anything it hears until it gets triggered. And what exactly are you afraid of google hearing about? You plot to take over the world? Or just the average murder/robbery plot? Because guess what, no one has ever been caught for any of that because google was "listening". I for one have nothing interesting for it to "hear".
Okay, here's a realistic scenario. All of this information stored on servers is extremely vulnerable to theft. Many companies, from retail, security, credit monitoring, cloud servers, and government including the IRS and state governments have been infiltrated and hacked dumping private data on the public - including bank accounts, passwords, tax information, credit cards, etc.
Do you really want your daily or occassional travel routines subject to this? Not me. That is a huge violation of OPSEC or PERSEC to have your travel routines and locations known. May as well leave your front door unlocked and some milk and cookies for thieves... And that's the least nefarious thing I can conjure... It's one thing if your neighbors have a general idea of your coming and going. It's another thing entirely if professionals can watch you in real time as you drive to another state, 10 hours away and remain there while they ranksack your house...