It's a great podcast game, but yea that can be infuriating if your audio setup doesn't allow for that. I personally have my PS4 running to both my speakers and my TV, so while I generally mute the TV to avoid audio echo I can easily turn down the speakers or unplug the system from them to separate audio when I'm attempt to podcast Destiny. Where I first noticed these options were missing was when I started partying up for the harder missions prior to most of my friends ditching the game.
The headphones I use (Klipsch Image x-11) come with a speaker which makes them more ideal for chat than the pack-in headset, and they're also fantastic gaming headphones since the sound is so rich and stage so sealed from the outside world, but they are SO good that the game audio easily overpowers the voice chat with all that bass and rich orchestral swagger. I quickly found that the only way I could party up was to start a voice chat separate from the game where you can easily balance the headphone audio between voice chat and game audio.
Yeah, tried setting up a one man party where you can slide the slider all the way to the headphone side, still the game is too loud.