I hear Renaissance of Fate is fairly difficult, but mostly for being outrageously obtuse. Haven't played it myself though. If you want good JRPGs that are also challenging, you should look into Tri-Ace's catalog in general.
I will second this comment, if you want JRPGs with really cool and complex battle systems then Tri-Ace is your company. Also Resonance of Fate shouldn't be missed. It was so stupid of Sega to release this on the same day as FFXIII but Sega wouldn't be Sega if they weren't doing stupid things.