I took a look at your screenshot, I can confirm your complaint. Ive had those exact players do the same to me. Got kicked out of my own lobby by them, when I was farming the demon lord prime incursion, twice. I figured I'd leave it on public for the extra loot (I can solo the map easily, so even if people don't have much to contribute, it makes no difference) and they booted me once on the final stage, and again a short time later during the build phase.
This is why i have always held a strong belief that kicking privileges should be solely left to the host, and that you should not be able to kick past the third wave (right around when it stops people from joining).