Benchmarking is when a game or program runs through a simulation of what will most likely be the most graphically demanding parts of a game, and records your computers performance during that scene. Once it is done, it will usually display to you things such as your highest FPS (Frames Per Second), the average FPS your game will most likely run at, and what your lowest FPS was when things really hit the fan. Some of the better ones will even recommend to you what settings you should set the game at to get the best mix of graphical appeal without needing to sacrifice a good frame rate.
Occasionally a benchmark isn't included with the game itself, but is provided as a free download separately (sometimes as part of a demo).
Game specific information
To enable the in-game benchmark you have to add '-benchmark' (without the quotes) to the startup parameters.