The VB may have only been in Black/Red, but the games weren't that bad at all. Wario Land, Teleroboxer, Jack Bros., Mario Clash etc. The controller was also pretty comfortable & I could think of much worse consoles like the Jaguar.
I actually owned one and other than it being a pain in the ass to play in bed, no it wasn't that bad. It did however give me a headache if I played more than an hour. Also the games telling you to take a break while helpful, got annoying. Sadly it was lost to a friend who moved away, and I still have the wario game buried in the rents basement somewhere among my childhood junk. I'd say the best thing about the Virtual Boy was it had a really comfortable controller.
I'm with the OP on this one. I've still got mine boxed up in the attic somewhere. If I thought it was that bad, I would've sold it long ago. I definitely recall playing the hell out of Panic Bomber. A puzzler obviously doesn't need 3D, but it was only on the VB and it was awesome. Wario was great, Teleroboxer was pretty cool, and Tennis was a lot of fun. The system was obviously way out of its league and couldn't compete with other platforms, but it certainly had its moments.