They're called Adventure games because the first one was called Adventure. At the time most games involved moving a cursor/chracter around and shooting or bouncing/reflecting something... action games. It was the first game where you went on an adventure.
As far as puzzles go I've never really had an issue with the "not making sense" thing. They usually make sense to me... I have a limited amount of interactivity in this area, I also have a few items, how can I use the interactive parts of the area and the items to accomplish something? I think people get hung up on the fact that they can't do anything they want in a game but that's not a limitation of just the Adventure genre. We all know the non-gamer who sits over the shoulder and says "punch that guy!" - "you can't" ... "well climb that ladder!" - "it's not climbable". The game is about working with what you have and what you can interact with.
I think I've come to prefer the puzzles that "don't make sense"... They're harder to figure out, more rewarding when you do. I don't like the simplification of the genre with obvious sparkly solutions. Sometimes it's good to miss and item repeatedly or have a few useless inventory items.