Despite Brad claiming that it's a "freebie" these days I found lowering the AA made a significant FPS difference on my machine.
Also make sure to turn off god-rays. It increased my frame rates quite a bit.
What are God Rays exactly, I mean I assume they are beams of light but when do they show up?
God Rays are rays of light like when the sun is shining through clouds on a semi-cloudy day. In Assassin's Creed 4, it when light is shining through the trees especially in the jungle settings.
This game has horrible adaptive vsync. If people with high end rigs are having frame rate issues then it possible that's the reason why. On my 770, I was getting 30 when running through the jungle. I know the card can handle 60 perfectly fine. The results of googling shows that it's because the adaptive vsync was kicking in. To disable the adaptive vsync, I had to download a program called D3DOverride to enable triple buffering, which forced adaptive vsync off.
As for the OP's issue, it could also be thermal throttling. If the CPU is heating up too high, it'll start to downclock itself. It's the same with the video card.