Backtracking is often a derogatory term used to describe a situation in a game where the player must return to previously encountered locations in order to continue advancing the game. There are two major flavors to this concept, described below.
Backtracking as level design
Many games include backtracking as an intrinsic way to advance the game. Games famous for this type of design include Metroid and the later Castlevania games, though there are many other examples, often of the Adventure genre. From the first Metroid on, players were often presented with a seemingly impassible location that had an impossibly high door or an obstacle that interfered with normal movement in a given area. Players might then later find an object or weapon which would help the player clear that obstacle, advancing to a previously unseen area of the game.
In a sense, by returning to that previously explored area, the player backtracked, or followed the track they used to leave the area only to return back along it later.
This form of level design is often frowned upon, as it is said to increase the length of a given sequence without a proportional increase in game content. Players are forced to see old scenery, and sometimes defeat the same monsters, despite having felt as though they had conquered that area already, leading to dissatisfaction. This dissatisfaction, though, is contingent on the repeated area's lack of variety.
Backtracking as player strategy
A phenomenon not directly related to the above is that of player backtracking as a strategy. For example, a player may miss clues which show how to advance in a level, the player may not have collected all of an item, or may need to refill his or her health. Backtracking is a way to, again, return to a previously explored area in order to remedy such situations.
In this case, it may or may not be the designer's fault, but it can be said that such backtracking is often less common in games on the whole, as it's usually a uniformly irritating experience for the player.