You interact with it to further the story and explore the environment. Is it a game? Hm...... depends on what you think a "game" is. Certainly Gone Home has no challenge whatsoever. No obstacle to overcome. No puzzles to solve. No enemies to defeat. There is very little fun to be had in interacting with the environment in Gone Home itself - the interactions are to further the story which is quite poignant and interesting.
Interactive Experience is a better term for it, and one that is just as good as "game". Calling it an interactive experience does not diminish its power or its worth. We've gone from "Don't call it a game" to emphasize its worth to "It HAS to be called a game" in order to emphasize its worth - what a strange development!
Something doesn't have to be a game for it to be extremely worth your time. "To the Moon" is my favorite piece of media, ever. I like it more than Catch-22 the novel, I like it more than any film I've ever seen - it's my favorite story. And I don't think it's a game at all. Yes, you interact with it a little. There are a few "action" pieces which you can't lose and a few absurdly simple puzzles put in there, I think, to justify calling it a game, but it didn't need any of those things to be absolutely special. It's an interactive Experience or an Interactive Story - not a game, really. That doesn't mean it is less moving than "The Last of Us" or any game or film or book.
Whether or not its a game SIMPLY DOESN'T MATTER, if you really think about it. And to anyone who wouldn't play "To the Moon" or "Gone Home" because they don't think it's a "game" - well, sorry. You're missing out.