When the round end the top 3 players and a top opposing player are shown, the "spotlight" system is how choose with execution to do on the opposing player. Kinda cool, but sometimes the opposing player can counter attack, but it really isn't clear how this works. Anyone know how you're supposed to pull off this? I end up hitting every button and accidently leaving the match

It's a quick time event, you choose an execution and after a brief period of nothing happening there will be a button prompt that pops into the middle of the screen saying something like "(X) EXECUTE!"

You have roughly a single second to press the face button shown, if you press the wrong button or miss your window of time, it plays the counter animation for the hostage.

Choose "Kick in the Nuts". Always choose "Kick in the Nuts".