Not sure about which particular Toyota CVT, but if it is the Corolla one with starting gear then engine braking should be using that gear instead of the belt / chain CVT and it should not wear out that belt / chain / pulley.
Not sure how the one with launch gear works, but I saw a vid the other day that indicated that the CVT portion was used in "low" gear, raising ratio for a bit, then a shift into high gear. How true that is I don't know, But at highway speed, if I were to request engine braking, I would not expect it to go all the way down into that low gear, if that low gear somehow bypassed the CVT portion.