The short answer is the U.S. did not for a moment think Japan would be able to mount a serious threat against the U.S. And yes they viewed Japan as a bunch of backwards "Yellow People"
That then begs the question of why the US maintained a large naval fleet as well as an army air base way out in the Pacific? If there was no threat, then why bother?
For that matter, by December 1941 the capabilities and the ambitions of the Japanese should have been clear.
Surely it wasn't a secret that Japan had built a large blue water navy and that it had also developed and deployed many capable aircraft?
OTOH, many Japanese at the time saw the attack on Pearl as a turning point in the war which would inevitably lead to the destruction of Japan.
They were right.