I think the fee should be low. I mean, Steam is still getting its 30% cut, so there's no need for them to make money off the submission fee. However, the barrier to entry needs to be high so that we can get rid of all these blatant "stealware" games, where they put 2 hours of code together and are just trying to get a few hundred people to put money into it.
So I'd suggest they make a system where the cost of putting up a game is $5K, and then on your first $15K of sales, you earn back $4500 from the cut that Valve normally takes.
That should hopefully stop the absolute avalanche of garbage that's been infesting Steam.