I can say without hesitation that I prefer Persona 4. While I really like the characters in Persona 3, they do have a bit too much of the melodrama touch about them for me (with the exception of Fuuka, incidentally my favourite P3 character). I liked that the Persona 4 characters had more relatable issues and also that they felt far more like a group of friends than the Persona 3 characters did. I know that's kind of the point, SEES is a very forced arrangement but I really like the camaraderie you get in 4.
I see a lot of people hold P3's story above P4, but I think it's important to note that for a good long while, you're just killing Shadows for the hell of it and even after you get some semblance of a goal you only do it cos Ikutsuki tells you to. It's only toward the last months that the story gets really good whereas I feel that Persona 4 remains more consistent. Persona 3's endgame and final boss are better than Persona 4's in my opinion but that doesn't discount the 70-odd hours preceeding it.
Music is just purely down to taste and both soundtracks are excellent as I'm sure most of you will agree. Again though, I find that Persona 4 has a more consistent selection of tracks I like whereas a lot of the tracks in Persona 3 I grew sick of like the dorm theme, the city theme and the Tartarus theme. Also, I'm not as big a fan of Mass Destruction as a lot of people seem to be. I vastly prefer Reach Out to the Truth. Having said that, Battle for Everyone's Souls in P3 is my favourite piece of video game music ever.
Gameplay-wise (excluding Portable here), it's absolutely no contest. Persona 4 streamlined a lot of tedious things about Persona 3 such as not having fast travel, not controlling party members, having to talk to them separately to equip them, having to talk to them to use their healing spells. And I will concede that these things gave Persona 3 a sense of identity from other RPGs but I don't think they needed busywork to be different from other RPGs, I think they accomplished that already. Of course if you include Portable under the umbrella 'Persona 3' then a lot of these complaints cease to exist (and so does some of the character stuff). There are some things I do miss about Persona 3 in this regard though such as getting up requiring a turn, multi-hit spells not giving a '1 More' if it misses one enemy or an enemy isn't weak to it and Fusion Spells. I feel missing these things out took quite a lot of strategy out of Persona 4 although it doesn't bother me too much. On a smaller scale, the menus in P4 are FAR more responsive than the menus in P3. Good god are the P3 menus slow as molasses. And then there's Tartarus. Fuck Tartarus is all I'll say about it.
So there, I've had my say. I will add though, that it's really easy to bag on Persona 3 when answering this question but I still love it. Persona 4 is my favourite game ever and Persona 3 is probably number 2, so really much of my criticisms of 3 come from a good place and are kind of nit-picky.