The 3D open world where you had the feeling that you could go anywhere, talk to anyone, interact with all kinds of objects, do all kinds of mini-games that were actually well-integrated with the world (physically and gameplay-wise) ... it was quite remarkable back in those early days of the Dreamcast - which itself had a certain magic to it. It was extremely immersive compared to other games.
The uncommon setting (1980's real Japan/Hong Kong/China!), plot, characters, and music were also pretty awesome.
And it was set up on a chapter-by-chapter basis and never finished. That right there is going to leave anyone invested thirsting for completion of the story. The first two games were a really slow build up and at the end of the second one you felt like you were finally getting somewhere ... then it was over.
The Yakuza games have kind of picked up where Shenmue left off. They have a lot of the same elements as Shenmue. Unfortunately you get the sad feeling that they had to make them really violent, edgy, and even Western in a sense in order to sell, and that's not what Shenmue was about.