If so at what point of the game is that?  I know all of the splitting up and what not is part of the story, but I find it really annoying that the party keeps changing.

At one point (not exactly sure) you can change your party on the go and choose your party leader.

At around the 25 hour mark you can change your party at will.

It took roughly twenty hours or so for me until I was able to freely select my party, but it's going to vary for a lot of people.

It took me 33 hours to be able to control them all. 

Not really no. You can eventually end up choosing leaders, but when the leader dies it's game over.

near the end of chapter 9 = control of who is in your active party, except leader. 
Chapter 10+ = full control over your entire party, including leader.