You should absolutely start skipping side stuff. Side quests can be enjoyable if you put on a podcast and just relax, but its all very skippable. The Inner Circle is the only side stuff you don't want to miss. And maybe kill the dragons. Those are pretty fun encounters.