Rock out with your... knife out.
The idea of 'running faster with a knife' is most commonly associated with the series of multiplayer First Person Shooters, Counter-Strike, where the character being controlled by the player has a faster movement speed when holding a knife as opposed to when holding a gun. Since the knife is the lightest weapon available in that game, the player moves fastest with it selected, as movement speed correlates to the weight of the weapon the player has equipped. Subsequently, the refrain became well known in that community and turned into something of a cliche.
Realistically, it's faster to run without anything equipped, and one wouldn't run faster with a knife equipped when they have several guns on their person either way. Nonetheless, it's a fairly well established mechanic at this point, although more recent first person games generally opt to allow the player to holster their weapon to access a faster movement speed as opposed to equipping a knife since, fundamentally, that doesn't really make a lot of sense.