I've been having this issue a lot over the past couple months, but only recently has it consistently stopped my progress on later waves of survival.
The issue is that the Djinn on Mistymire Forest and Shipwreck are getting to my traps and auras and then flying very high up in the air. From there they are only being damaged by the lightning aura, but are also able to despawn my traps and worse my walls (which of course despawns my buff beam as well, still surprised that this hasn't been changed).
Early waves of survival this is fine, but when I have 3 ogres on the other side of that wall w