This artifact of game design is one of the oldest on record, as it has existed (though not contemplated) since the days of arcade games advanced enough to allow a character to roam on-screen, yet not advanced enough to allow the character to scroll the screen - thus, the character is "boxed in" to the single screen area, surrounded by invisible, impassable barriers on all four sides.
The same is usually taken for granted in platformer, sandbox and RTS games. Mario begins a level on its left end, and cannot go any further left than where he began. The protagonist in Far Cry 2 can travel out into the african desert, but once he passes a certain rectangular border surrounding the game area, he is immediately dumped back into his cordoned-off zone, without explanation. Realtime Strategy games will often feature no natural barriers along a map's edge, but the camera and your units are not able to travel beyond it.
Despite all these examples, the term "invisible wall" is usually only invoked when it stops making sense. This most often happens in 3D games. The protagonist attempts to leave the game area, only to discover that he is trapped from proceeding by solid, unrelenting wall extending all the way to ground level. Perhaps a protagonist that can jump attempts to vault over a waist-high wall, only to find himself smashing against it, as though it extended all the way into the sky, thus sparing the game designers for having to account for the protagonist proceeding beyond a small enclosed zone. Perhaps you are playing an FPS game with flying vehicles; You climb onto your vehicle, start flying straight up and suddenly smash into an invisible ceiling. You are playing a 3rd-person adventure game. You attempt to walk your protagonist towards an area with dense geometry, like a desk with office chairs, or a piloting cabin of a ship, or rows of lockers - only to be repelled and pushed back, as the game engine is unable to handle moving the character through these tight spaces. It is then that the term "invisible wall" comes up and is used in derogatory fashion.
Invisible walls always have been and always will be an important gameplay element, and like all important gameplay elements, can be done correctly to add to the experience or incorrectly to add frustration. Correct use of invisible walls is for plugging up dangerous areas the player might get stuck in or fall out of the game world through, or for "estimating" collision around very complex geometry to enable more fluid gameplay. Bad use of invisible walls is for inexplicably trapping a player in an environment that otherwise seems open, or preventing the player from walking towards a place that visually seems to be reachable.