Real World Context
Human beings are usually only able to hold their breath comfortably for a minute or less without training. This is because the brain has an autonomic warning that forces the person to breathe after enough time has passed. While children have a greater tolerance for holding their breath, possibly because their systems aren't fully developed, an adult can usually not manage to hold his or her breath without significant effort.
Organizations such as the U.S. Navy SEALs tend to have the reputation for training people to hold their breath for a certain amount of time, or to demand that you can swim underwater without going up for breath for a certain length, but this isn't generally publicized, at least not on their main website. Magicians and stunt artists try for general record attempts, often at the risk of their own personal safety-- many have failed, a few have died, yet those who have managed to push through
have recorded times of nearly twenty minutes.
Since you are depriving your brain of oxygen-rich blood such attempts are very dangerous, even for someone with a high level of oxygen content in their blood, so attempts are often made with little to no exertion to reduce the body's need for O2. Still, for all the successes there are many failures, resulting fatalities or serious injury.
Holding Your Breath in Games
In games the endurance is markedly LESS than 20 minutes, and can sometimes be measured in seconds. This is probably to increase the tension of the game for the player rather than to reflect any specific length of time. In Grand Theft Auto: San Andreas, the first of the GTA games to allow swimming, players could actually extend their breath meter over time by practicing swimming, which is actually pretty close to how it's really done. On the other end of the spectrum, games like Everquest or Elder Scrolls III: Morrowind don't allow the breath meter to be extended, but it can be augmented through spells and potions.
The design goal is often to create a sense of urgency, a timed exercise that pushes the player to be efficient and quick with their actions, and to create a sort of barrier to advancement that either must be overcome, or acts as a way to limit interaction with certain parts of the game world.
Paying the Penalty
Using the two previous examples, in both GTA: SA and Morrowind when the player character's breath meter drops to zero, you begin to take damage directly to your health bar. This is actually pretty common in such games, although some of them will actually drown the player character outright should the meter deplete, or cause damage as it depletes, killing the character by health damage when the meter reaches zero.
Sometimes the breath meter can be extended. Some games utilize bubbles (or kissing blow fish in the case of Radical Rex), while others such as the game of the Thief franchise use breath potions (which actually makes sense, when you think of it. They're probably just little bottles of air).
When coming up for air, recovery differs as well. In Grand Theft Auto, depletion takes a while but recovery is slow, often gaining back breath as slowly as it depleted underwater. In Morrowind, breath is recovered instantly, allowing you to come up for air and dive down with no penalty as long as you do it before it expires. Others vary the rates, but usually it doesn't take longer to recover than it does to deplete the meter.