The Shenmue boxart catches my attention, maybe Deadly Premonition is something worth my while checking out :P As for your question... the best example in my experience would be Little Big Adventure for the PS1. It was a blind buy when I was a kid, before the rise of websites like this and others that allow you so much information about video games. It's (unsurprisingly) an adventure game that mixes in a lot of puzzle solving and some light combat, and had a unique control system (though it would be considered very clunky nowadays). The shoulder buttons allowed you to enter one of four modes (they were normal, combat, discrete/stealth, and athletic) that changed your entire control set. It'd be hell for me to play now I reckon, but I really liked that system. As for more recent examples, Burnout Paradise is one, though that's more down to me not being a big racing fan than any hidden gem quality on the game's part.

A few games come to mind that I just took a bit of a gamble with and tried out and enjoyed it a whole lot better than I though I would.  One of these was Dirt/Colin McRae's Dirt.  Before this, I played maybe a couple of rally games like V Rally and Sega Rally and wasn't too fond of them but they were old games by the time Dirt came out.  What I got was an excellent racing/rally game that had one of the best career modes in a racing game with the way it was shaped out and had a great UI.  A lot of people that year were going on about how great Forza 2 was and that it was the racing game of 07 etc.  I got Forza 2, played mostly online but for me Dirt was my preferred game in that genre. 
 
I really want to play Deadly Premonition but one problem arises.  It ain't out here and no sign of it ever coming out.  I know there is a PS3 version out in Japan and 
might make its way to the US and if that happens, will get it.  I am sure the game sales rose a little thanks to the Endurance Run on here.

My greatest gaming surprises were Far Cry 2, Beyond Good and Evil, and Psychonauts. 

Far Cry 2, I had no idea I'd be writing these many notes about the game on a Notepad document. It continually impressed me with surprises, missions, gameplay moments like having your buddy going down, and ways to tackle combat situations. It's also a very unforgiving game, which you are reminded with by the presentation of mercernaries, grey-ish factions, and the Jackal himself. Just a very unique experience as far as First-Person shooters go. It's like an existential FPS without the pretentiousness. 

Beyond Good and Evil is chock full of great moments, characters, locations, and gameplay surprises. It never feels repetitive, which is a feat. You have so much variety, going from a sneaking mission to a boat race to photographing weird creatures. Its charm is arresting.

Psychonauts really is a hallmark of creativity. Its gameplay is not as refined as BG&E, but what it has to offer in terms of level design for each of the brains of characters is unparalleled. I have a hard time picking my favourite levels, it's a fight between Milkman Conspiracy and Black Velvetopia.