Rarely, but every once in a while I'll listen to NPR or podcasts. Like when I was grinding through the final achievements of The Saboteur or exploring the island in Test Drive Unlimited 2 (talk about a game you don't need the sound for).
I've been using Xbox controllers forever (I still really like those huge original system ones, too, I'm weird) and I'm used to them, but I got my first PS3 a couple weeks ago and while the PS3 controller was weird at first, now the Xbox controller feels goddamn massive and heavy.
Anyway, the key to your problem is in the following words of your post: FOUR HOURS. Four hours straight of holding a controller is going to hurt your hands. Take breaks, don't play for so long at once.