I agree that a reporting system would be a pretty expensive option and I suggest it as more of a "stretch goal" solution. It would require additional development time to add in all the logging and systems to look into various accusations. Not necessarily an impossible option, but unlikely.
When DD2 was still relatively new, it had a rep system (haven't played in a while but I think it's gone now). It was simple and took no time to use. How I would expect it to work is, when you exit a map, whether it's from being kicked or completing it or leaving on your own, you are able to give an optional up/down to any player that was in that match while you were. As for good players turning bad, either give players the option to have the rep calculation limit by a time frame (past year / month / week), or have the built in calculation give higher weights to more recent ratings. If there's still worry that the rating system will be abused, then have up = +1, down = -1, and no vote = 0 and use an average. The actual algorithm will likely need tweaking, but you get the idea.
Also note that getting kicked isn't an immediate down vote. Neither is kicking someone giving the host a -1.
Specifically on the "Let's say a troll starts playing the game and keeps doing bad stuff and ends up with a garbage reputation but then starts really loving the game. Is that person doomed from playing with anyone who picked "good reputation only"." remark. Yes. If you are a new player and you are only being toxic and a total jerk, then you should expect to have to put in work to fix your reputation. Same as in other online communities or even in real life. Perhaps that seems harsh, but a brand new player that's also toxic and a troll isn't something I'm concerned with catering towards.
I think the idea is that if a player blacklists you their hosted games don't even show up in the list for you. Likewise, if you blacklist someone, they aren't offered as a game for you to join. There's not a "You are banned" message on trying to join.
As for getting blacklisted when you go AFK, even if unintentionally, just accept the single instance of it. If a host doesn't want AFK players then they should be able to remove one simply using host-kick privileges. If that player keeps joining and going AFK, they should be able to blacklist them. If it happens to me that after unintentionally going AFK I get blacklisted by some user, so be it. It's not a room I would care to be in since I tend to play in a way that I might go do something and end up AFK for a few waves.