This is the case for any armour build that also effectively utilizes the armour applies to elemental damage mechanic.
The phys damage is reduced ONLY by armour. Elemental damages are being reduced by armour AND by resistances.
So even in character sheet, not even PoB, you'll see a big difference in reduction between phys and elemental damages.
This is to be expected and isn't an indication of something wrong with your build.
We also do not have a death recap, so you're kind of guessing what killed you.
On PoB or poe.ninja you can see max hit. I imagine your physical max hit is considerably lower than your elemental and chaos max hit (assuming maxed resistances). You could keep scaling armour (and the four max hits outside of chaos would scale as well) or you could scale your health pool via life/energy shield and/or mechanics like MoM and still be scaling all your max hits.
At the end of the day, there are hits that the player is just supposed to avoid.
And if you're talking 77 room temple carries, well no one is built to take those hits. Things need to be dying before the hit you. Period.