The crew mentioned it on the bombcast a few times. I think specifically around PAX East. All about swapping different characters or items over SpotPass to defeat enemies in some kind of labyrinth... It sounded like it was possibly a nintendo built-in. After dusting off my 3DS, and finally quitting out of Zelda: Phantom Hourglass to do system updates, I didn't see anything that looked like what they were describing.

Does anyone know which app or game they were talking about?

It's in the Mii Plaza, there are two games, one is Find Mii, where you can use Miis picked up through SpotPass to defeat ghosts, and there's Puzzle Swap where you collect puzzle pieces from Miis to complete 3D images.

Oh yes, this is the thing! Thank you.

To clarify, that's StreetPass, which is what's used when 2 3DSes are physically near each other. SpotPass is what's used by companies to push information to your 3DS via the internet.