Um, no, it's actually infinitely possible to play with friends. There's an explicit area to play with "only friends." I have the steam version, so I don't know how it works for others, but only your steam friends can come into your lobby.
I've found you can use this to your advantage to become the host of a public game, too. If you go to create a friend (private) game you can still change the settings before you start the game to make it public, and you'll be host (and not have to deal with slow connection hosts).