Your sense of anxiety around leaving your job is normal, especially for the first time you do it. But the thing is, everyone understands how the system works. People who are good at their job move on if they're not rewarded, and your school isn't rewarding you. They know this is coming eventually and the people there should personally hope for you to find something more remunerative (that you still enjoy) if they actually care for you.
Edit: Also, it sounds like the only reason you think you enjoy your job so much is because you're classifying everything you don't enjoy as somehow not being core to the job. It's sort of like an abusive relationship - "Oh, it's not Bob fault, it's the whiskey..." No, it is Bob's fault, and it is part of your job that you have to live with the rents and walk 2.5 miles to work.