A mustache is defined as the unshaven growth of hair on the upper lip. It usually implies the wearer only has hair on his upper lip and has his chin and cheeks shaved.
The first mustache in a game (Donkey Kong) was used because of graphical limitations. The drawing of a mouth was too difficult, so Mario (aka Jumpman) got a mustache instead.
As a gameplay mechanic
Mustaches reached their hairy apex in 2003's Mario & Luigi: Superstar Saga. In this 2D RPG for the GameBoy Advance, players were given a "'Stache Power" stat which was equivalent to 'charisma' in other RPGs. The higher the "'Stache Power" of either Mario or Luigi, the less they would have to spend on new items, and the more they would recoup from used item sales.
Similarly to this in the recent title Mario & Luigi: Bowser's Inside Story, the "'Stache" statistic is used to increase the critical/lucky hit chance, and to get a winning space on a scratchcard, which returns money upon purchase.
Characters in practically every Mario RPG have something to say about Mario's mustache, often comparing their own to his.