Deflag someone

I keep accidentally flagging posts, its something that's happened 2 or 3 times now. It happens when I go to click somewhere neutral in the window to switch to it, but when I do, the "Flag" box appears, and I've flagged some innocent victim.

Is there a way to deflag someone, or a confirmation box I can opt into or something?

@videorob: Yes, just hit the same button again.

After you click "Flag" the button's text should change to "Unflag". Just click that, you should get a notification saying that it was successfully processed.