Memory Leaks? Suffering progressively severe lag as time goes on

Hello, since the last 2 patches ive been suffering from FPS dropping. At first things will run smoothly and i'll have no issues, but by wave 10 and beyond things start to get choppy, even during build phase, and it only gets worse until by wave 18 its unplayable. Even after leaving the hosted game, and sitting at main menu, the game will lag horribly. Restarting dungeon defenders usually fixes it temporarily. I have tested other games (The Secret World, Diablo 3, etc) none of which suffer similar issues.

Edit: Not using controllers or anything.

PC Specs:

Quad Core 3.2ghz,
12gb ram
nVidia GTX 560 Ti 2gb Driver 301.42 (recently updated from 282 to see if that'd help, it didn't.)
Win 7 x64
Service Pack 1 Build 7601 version 6.1.7601