So yesterday I bought Minecraft. Due largely both to this community, and to X's Youtube videos. This game seemed to have it all, and be completely addicting, and for 13.50 I was sold.  And for the first 10 minutes or so of me playing it was everything that I wanted it to be.
 
But then I ran into I problem. I couldn't decide where to make my first home. Whether I just build a house from wood in the middle of the open field, or if I should carve my way into a mountain. The first world I generated had no big mountains any where around my spawn, so while I tried to make a little enclave in a hill side I was not satisfied. So I generated my second world. In this world it was a bunch of tiny islands surrounded by water, and although awesome to look at, I wasn't even sure where to start building. Finally in my third world I got a little hideout going in the side of a mountain, but there were to many pre-made caves around which I felt like wasn't as cool as digging my own...
 
So moral of the story is I'm just too picky, so I'm curious what are things that make a good starting location to set up camp. I'm kind of set on not building a house, but digging out my place of resident from the rock and cobble stone. But I would appreciate some tricks and hints to get me on my way.

Delete the world and make a new one, no use staying in a world you don't like .
No shame in it, I've spent at least 15 minutes finding a good one.

Nothing wrong with being picky with this game. Like tankintheair315 suggested, make a new world until you find what you like.

I disagree with the people above. Minecraft Survival is about.. Surviving. Conditions won't always be perfect and you should try and explore and find a place you do like in your world. If that means that you set off on a journey and build a temporary shelter on your way to where life takes you, so be it (this is likely how you'll have to do it anyway, you probably won't have the time to make mining picks as well as a shelter before the first night). 
 
I think you should just start a world and see where it goes. Minecraft is all about being creative anyway! 
 
The only exception I really give is whether or not you want snow. 
 
But that's just me.

In my favorite world there was a small cave high up on the side of a mountain so you had to climb to reach it. It looked so sweet I decided to keep digging there. I branched it out into 5 new caves which then led into pregenerated caves which split up even more. I can hardly find my way around inside anymore because it covers so much.  
 
You shouldn't neglect pregenerated caves though, they do work for you and give you directional adivce where to go next. (For example in my pregenerated caves I dig out the minerals first and then any sand/gravel there may be, and whichever way looks more interesting after that I'll take.)
 
I think I got a really good spawn location in that world. 
 
I agree with Pinworm though. I believe the world in alpha is pretty much infinite so if you don't like the place where you spawned, just go exploring and set up markers so you can find your way back if you die. (I don't even know if you can set your spawn location, I am such a pro) 
 
I think a good way to make your caves is make them cover a lot of ground and have a few exits/entrances where they branch out the furthest so you can get to other places on the map from within the safety of your cave. As you get more materials you can build a simple railway system that connects all the entrances and exits. When you get tired of being underground, you just waltz out one of the exits and go chop some wood and explore more of your world. I found waterfalls and a big ass field full of trees when I went exploring. Shit was so cash.

I spent forever in my first world building a castle, then decided to just generate a second for the hell of it. It was so much cooler then the first one, had a huge mountain with a waterfall, the first one was a lot flatter.
Now i'm thinking of just deleting my castle. Took me forever but i mean c'mon...a waterfall!!

My first one spawned me in a valley about 15 squares deep, but with really steep sides.   I considered just adding a roof, but right on the other side of it was a lake, so I connected my spawn point to the lake and built a house lakeside.  Then I deleted that world.  
 
In my second world I just picked a direction and ran for two days until I found a nice spot.   The world is a lot better, but if I were to ever die, I could never find my way back and would be completely fucked.   It makes for very tense nighttimes.

i got a few worlds. My favorite one is snowing and I built my home in the side of a cliff thingy with a waterfall, and a massive labyrinth of mining tunnels i built (there is no David Bowie in my labyrinth)