The principle you’re using here is Occam’s Razor, which states that the simplest answer to the unexplained is usually the best one.
A complimentary scientific philosophy is the principle of multiple working hypotheses. This principle states that whenever an unexplained observation is made, creating multiple potentially plausible explanations can be beneficial to keep biases from affecting the analysis. This keeps the effort on falsifying several hypotheses rather than attempting to prove a single one and running the risk of confirmation bias.
Regarding “everything we know about space, time, physics, etc.” suggesting that there is no nonhuman intelligence (either terrestrial in origin or extraterrestrial) with an awareness greater than humanity’s, I’m not certain that’s entirely accurate to say so definitely. To be clear: nonhuman intelligence certainly does not seem likely, and Occam’s Razor is fine to apply, but so is multiple working hypotheses.
Fermi’s paradox could be neatly wrapped up, for example, if (big if) the reason we have not made contact yet is because we are purposely not being contacted. We have very little knowledge of our own deep oceans. We have good reason to hypothesize that conditions for life exist currently in our own solar system in several locations. We recognize that H. sapiens sapiens have only existed for the past roughly 100,000 years, and have only been scientifically literate for roughly 500 of those years, which we know is an incredibly short period of time in the context of deep time. We are still struggling to create a unified theory of physics to mesh relativity with quantum theory.
All this is to say, we need to be humble as a species. Because our scientific progress has been good, ironically, there’s now a lot we know we do not know.
Occam’s Razor suggests this, but it does not falsify any other explanation. Carl Sagan said, “extraordinary claims require extraordinary evidence.” This certainly applies here. Just bear in mind, extraordinary evidence requires extraordinary change.