Good question I would say depends on the game if it is trying to be more realistic then yes should be some damage but for other games it doesn't really bother me. Guess it all depends and what the game is trying to do
In Far Cry 3 you can dive head first, arms stretched out in front of you (fucking awesome), and in that case you shouldn't take damage. When bailing from a hang glider over water, falling hundreds of feet, ass first, arms flailing, you should take damage. Still not totally realistic, but it makes some amount of sense.