After sequence 3 Desmond says: "Why don't you get in the animus, he's your ancestor too!" And his dad just kind of ignores the question by telling Desmond to grow up. But you know what, that's an excellent question. I has assumed before that all the ancestors were from his mom's side of the family but I guess not. I am assuming this never brought up again in the game the same as Lucy who is obscurely reffered to in that scene as well.