In the crash test, the RX has a lower rating than the ES. ES is not a small vehicle inside. That is where the SUV advantage might come, headspace, etc. But ES, compared to RX, is not that much smaller.
But then we come back to the fact that ES, while not a handling benchmark, still is a better performer.
For example, I use my BMW daily, hauling 8 month old, two other kids. Compared to my Atlas I had before, or especially now Sequoia, it is a vehicle that will do much better in not ending up in a crash.