I have voiced similar concerns several times here in various forms. Clearly, my concerns are from a point of ignorance. I do not know the answer to your question. However, I do know that the answer is related to two factors that prevent the "I used it and my car is fine argument". First, the answer to the question if very much related to the type of seal material. The newer the car, the less of a problem - I think. The newer cars use materials that are tested with higher awareness to the types of lubricants that the engine may see. The second factor is the type of ester. The word "ester" really only describes a single molecular linkage. The oxygen linkage produced from the reaction of an alcohol and an acid. So, "ester" in and of itself is very non descript. It is like saying "is food good for you". Well bacon grease is food, an apple is food, cake is food, beans are food. And, even though we all know that food is a requirement for life, the food question cannot be answered very well without more specificity.
It seems as though the number of esters on the lube market is relatively low. I am guessing that most of those reagents have been sufficiently studied to think that there use would not be an issue. However, I am still concerned that the integrity of seals that see different conditioning agents in sequence is not really known. I often use silastic materials in my research. Repeated shrink/swell seem to make them fragile. I do not see a reason why a similar phenomenon could not take place in engine seals. But, I know that I do not know.