Man hearing about this makes me think I should restart. I'm just a few hours in and already feel like I broke the curve. The magic is just ridiculously strong, and you don't even have to put points into attack since the augments do all that work for you. I can totally see why the game wants you to spread out the points now, because maxing fidget early would just be broken.
If I remember correctly there are also rings that double your magic power so that's what I did for the last quarter of the game.