For those who might be dumb like myself, I was using overcast to play the podcast and forgot to turn off smart speed so it always ended up out of sync. Took me a while to realize lol.
BrosephR's comments