An in-engine cinematic is a cutscene that, opposed to full-motion CG cutscenes, uses the in-game graphics engine. Usually used for minor cinematics that happen during the action, they especially stick out when a game also has CG cinematics. For example, in Final Fantasy XIII, Lightning's hair moves around a lot more during the CG cutscenes than in the in-game ones.
In some games, these cinematics can be downright ugly, especially in games where the camera is panned out as we're getting a close up look of characters whose faces are just a bunch of muddy textures.
Some games, however, manage to pull it off. Games like the Metal Gear series, Half Life 2 and Just Cause 2 only use in-engine cinematics. This lets for a more consistent experience where the characters don't suddenly get prettier and more detailed during a cutscene, as is often the case in a lot of games from Square Enix (and previously Squaresoft), which is known for producing some of the best CGI yet this often overshadows the quality of the in-game cinematics as a result.
There are mainly two types of real-time, in-engine cutscenes:
The older technique used to render real-time, in-engine animated cutscenes was by using 2D animations, in the form of animated bitmap images, similar to how the GIF animated image format works. This technique was introduced by Squaresoft for the 1985 sci-fi visual novel / adventure game Will: The Death Trap II, programmed by a postgraduate student from Keio University (two years before the GIF format became available in 1987). In 1986, Squaresoft further developed this technique with Will's successor, Alpha. The success of Will and Alpha in the Japanese computer game industry soon led to many other video game series also using this technique to render real-time, in-engine animated cutscenes, including Valis, Ys, Snatcher, Ninja Gaiden, Far East of Eden, and Lunar, among others. While uncommon today, this technique is still used quite often in the visual novel and adventure game genres.
The most common technique used to render real-time, in-engine animated cutscenes, is simply using 3D polygon graphics. Initially, 3D polygons lacked details, so many game designers relied on FMV (full-motion video) to render CGI animation, but as the quality of real-time 3D polygon graphics improved, most developers abandoned the use of CGI animations recorded with FMV, with the exception of some companies such as Square Enix.