Any editing/encoding software with decent customization controls should let you do the settings I'm going to suggest. Keep in mind that H.264 is not the best codec to have to edit later, so only export to H.264 when you're ready to upload.
File-type and Codec: .mp4 H.264
Resolution: Whatever you want it to be (if you want to be above 5 minutes or so and not more than 150 or so MB, you're going to want to top it out at 640x480 if you're doing 4:3)
Framerate: Whatever FRAPS captures at. I have to imagine it will be
Video Bitrate: Variable and I would advise between 1 Mbps and 1.5 Mbps; the higher you go the higher the file size
Audio Bitrate: 128 Kbps is probably the lowest you would want to go if you're playing music. If you're doing voices, you probably want to have 192 be your lowest bitrate
With those options, I can make a 12 minute video at 640x480 that's 150 MB. You could probably get away with lower bitrate if you're truly concerned about filesize, or even go a little higher since Youtube will be the one doing the streaming after the initial upload.