Yes, theyarecomingforyou, a couple of things. First, the basic principle that buying something (in this case a game) transfers ownership to the buyer. The new owner then has the right to resell the item in the future, because it's THEIRS.
Also, you seem unclear about what middlemen are. A pawn shop or any sort of reseller of used goods is not a middle man, because they're not involved with the initial sale. But a brick and mortar game store is a middleman, because they take a cut of the game's sale price.
It's arguable whether a used market even undermines primary sales, because the ability to resell it adds value to the game. But it's also a bit of a pointless tangent, because despite the fantasies of certain publishers customers still do have some rights.