I'm in the same boat. I absolutely LOVED Fez, and the word-of-mouth around the indie community was that this is a very similar experience. But I actually found myself becoming extremely frustrated with this game. There's so much time investment, so many random elements, so many ways to completely mess up a level and get nothing from it, and so far (even trying to spoil a little bit to see what I'm working towards) the payoff seems to be seriously lacking.
I don't know if I was expecting this to eventually turn like Portal or Frog Fractions or something. But after about 3-4 hours, it's just not fun enough to grind through in order to see what's behind the 3-key doors.
I understand that discovery is the point. I'm sure people are completely satisfied with the colors and sounds (I was a little disappointed that the "symphonic" garden was just made up of repeated major triads =P), but I just want to know, at some point, if there's anything left to do. And the volume of "just keep working at it, you'll see" posts are completely mystifying to me.
I'd love to know from anyone who has put more time into this if I should keep going, and if so, how I might be able to gauge my progress.