Microsoft hasn't had the greatest record with security either. The last thing I want is some douche hacker getting access to my Kinect to use it to harass me or my family. If the Kinect wasn't such an integral part of the system, I would just disconnect it and not worry. From what I've heard so far though, that's not really an option if you want to use Xbone.
Edit: I have to add that I think the Kinect is super cool and could be amazing for horror games and such. If Microsoft could at least insure that we can play offline and that no Kinect video/audio is recorded, than I really wouldn't mind. But as is, Kinect + Always On + Potential for devs to force Always Online while playing = I'm very worried.