If he doesn't want to play with you, he doesn't have to. If you don't want to play with him, you don't have to. If you want to play with him but are unwilling to change the difficulty, play a different game.
But know that with games and every social event, people are not obligated to play with you, so everyone should be having fun. So, compromise and try to reach a consensus on rules etc that works for everyone you're playing with. If this does not work, see my first paragraph.