After watching Vinny's amazing Bot Colony experiment I decided to check out the Steam page.

What greeted me was an ostensibly hilarious "Reviews" section:

Watching even a minute of Bot Colony you can tell these reviews are ridiculous. So I decided to check out the actual articles and see if Vinny was just missing something. Much to my chagrin the Destructoid review is taken out of context and is actually a preview of a trailer. The Gamasutra quote doesn't actually exist. The Game Informer quote is just a back-of-the-box quote from a preview that doesn't impart any subjective criticism required of a review.

"If it works -- and that's a big "if" -- then Bot Colony might be the first game to really nail the dream of full vocal interaction. However, being Scottish, I can't help but feel I'll have a natural handicap."

- Alasdair Duncan, Destructoid, 2013

Gross.

Bot Colony is a bad videogame. That voice recognition video is amazing though.

That's what happens when you let Jimmy prepare your Steam page.

Quoting a review by Alasdair Duncan/ Destructoid is pretty questionable by itself.. and taking those quotes out of context is.. well, just wrong.

That Alasdair Duncan fellow should lend his name to the next Dragon Age game though.

Edit: Oh, and it's not even a proper review.. just shady. Just like the game. Shady as hell.

@superfriend: There are already major characters in Dragon Age named Alistair and Duncan.

Talk about misinterpretation.

WHO ARE THESE NORTH SIDE FUCKERS WHO PISSED OFF VINNY AND HOW DO I DESTROY THEM?

Hahaha, of course they're just straight up lying.

Yeah, but they're two seperate dudes (and both could be dead, depending on your actions)

So Alasdair Duncan would just be a perfect fit, right?

"Jimmy, pick up the gawddamn pepper grinder. Jimmy. No. Pick up. Jimmy. Shutdown. Just turn off..."

Giant Bomb

Hey, at least they're not as desperate to use one-word pull quotes which is all too often :P

IGN: "Amazing"

Eurogamer: "Spellbinding"

This happens a lot on Steam for these smaller releases.

Right. You name is A cursory examination of Bot Colony's Steam page.

But anyway, a ton of games do this with their boxarts and Steam pages. I find it shady and I don't like it, but it's all part of the hustle.

Yeah, reading up on the game before the live show I was amazed I never heard about this game by how the game's Wikipedia page touts it as some groundbreaking new piece of interactive entertainment, containing the "ability to engage in intelligent English conversation with the game’s characters."

After watching the live show, there is nothing intelligent about their speech-to-text/text-to-speech technology. In fact, that technology seems just as accurate as it was 7 years ago on Windows Vista. It's really disgusting how hyperbolic the descriptions of this game have been, and those "reviews" are just plain lies and false advertising.

Watching Vinny descend into madness was pretty entertaining, though.

Well that's delightful and not at all shady!!!

Even if the voice recognition worked, it would be a bad game.

If someone buys a game based on quotes from the game's store page, then they deserve the pain of buying a bad a game.

Without giantbomb, I wouldn't know what to believe.

You might also notice that of the positive user reviews 4 of the 7 are that person's only review. Since they've got a bunch of games on their account my guess would be developers or friends of the developers.

These videos

https://www.dropbox.com/s/ginpub8ztnmhfae/Bot%20Colony%20Robot%20Visual%20Memories.mp4

https://www.dropbox.com/s/pai63dfy04530gx/Text-to-animation.mp4

show how the game performs when it's played with a trained voice profile. Can you please take a bit of time and actually WATCH them? Of course, you're welcome to buy the game on Steam, check it out yourselves and report actual problems you may find in the Community Hub on Steam (I did acknowledge the ones Vinny encountered). It is a very new kind of game. We did work very hard on it, and I find that Vinny's 'review' amplifies the negative to make points about speech to text and using language in games.

My post in the main thread on giantbomb.com enumerates the real technical problems highlighted in Vinny's session. They're manageable. I feel that Vinny did make the game look terrible, to put on a good show. Using keyboard correction, for example, would have solved 90% of the heartburn, but then he wouldn't have such a juicy show, right? He only used it once or twice at the end. Also, he asked for the robot's commands, but never used the information. Why? To issue unsupported commands?

There were several Gamasutra pieces that dealt with the Bot Colony's potential, concept, story, etc. (the first prototype was shown at GDC in 2009 and the second at E3 in 2010 - it really was a very long haul effort)

http://gamasutra.com/view/news/202072

This is more dated http://gamasutra.com/php-bin/news_index.php?story=22513

and this

http://www.gamasutra.com/blogs/EugeneJoseph/20140626/219765/Natural_Language_Understanding_and_TexttoAnimation_in_Bot_Colony.php

you'll probably find the quote there.

I don't think Vinny actively tried to make the game look terrible. He went through the 20+ minute voice profile training, spoke more clearly and slowly than a normal person should have to while in the game, and it still produced some hilariously inaccurate results. Even if it did all function immaculately, what I saw didn't look like it would be fun. Just my two cents.

So any comment on the Destructoid quote deliberately chopped and taken out of context to mislead potential buyers??

I think I would rather play Hey You Pikachu!. Atleast it has Pokemon and not stupid robots.

You're telling me to find a quote from a review that is from the Producer of your company?

ARE YOU SERIOUS RIGHT NOW?

It's not a review, it's a technical article. If you want a review, this came out recently

http://www.rockpapershotgun.com/2014/07/07/bot-colony-alpha-review/

@botcolony: I just watched the visual memories video and the whole thing felt rigidly scripted at not at all like a normal pattern of speaking.

For example when the robot gives a bedtime for the child, it is very unlikely that anyone would parrot back the name and exact time as part of a question. They may ask 'where is he?' or 'why is he not in bed?' but would not say "where was *name* at *time*".

Based on everything that I've seen so far, I get the impression that there is a singular path that requires you to know and repeat exactly what the writer was thinking. If you deviate from that path or come at it from a different angle then the game does not respond in a natural way at all. As we saw with Vinny. I've done similar myself - I've put something together and then watched as someone struggled with part of it because they weren't replicating what was to me an obvious thought process. But that is why we test; so that we can determine where we can simplify and improve and allow something to be used and enjoyed by a variety of people - some of whom approach situations in ways that are very different from our own.

That is without even touching upon the voice recognition issues that clearly exist for users. Which, yes, can probably be improved by training but it also needs be made obvious what the training will do, why it is needed and then how to activate it. If it is the case that the training is essential - say so. If a user tries to click start with a default profile it is trivial to add a screen that states - 'caution you are starting a new game using an untrained speech profile. Do you wish to continue?'. At the very least you've then warned the user that they are in for a sub-optimal experience, but also lets them know that they may have missed loading their profile such as in the video we just saw.

Between the logic/unnatural speech patterns and the issues inherent with voice recognition the game as it stands just looks like an interesting idea whose limitations make it very frustrating to play. Maybe this is something that can be worked out during testing, but given your comments on this site so far seem to take the 'your doing it wrong' rather than 'how can we help make this better' approach to customer feedback, I am unconvinced that the issues will resolved.

It's listed as a "review" on your Steam page. It's in the first post of this thread.

This thread isn't about Vinny's stream. This thread is about intentionally misleading customers with fake reviews. One of which is apparently from the producer of Bot Colony and sourced as Gamasutra.

I'd like to explain the time business in robot visual memories business (the 3rd Gamasutra link explains perception-based Question Answering in detail). Where is he? or Why is he not in bed? (I agree this is what the obvious thought process would lead to) may get "I don't know" answers if the robot didn't see or hear anything about that. Then the question is what do you do next. Questions like when did you last see X? will return a specific time, that you can then use to ask a specific question. It works by narrowing down to a time-segment by elimination.

You're making excellent points (about singular paths, testing and community feedback), and trust me that this session and the conversations surrounding it give a lot of fodder for thought and improvement. About voice training - I'd even go to the distance of not making it available if the training score is not good enough. Players can type until they get a precision enabling them to have a good experience through speech.

I'm not necessarily defending the game or its systems (your latter point) but I think it should be clarified that the speech recognition software was not built into the game - it was just using whatever was available on the computer in question - in that case the basic Microsoft speech software. I assume if you had Dragon or something it would work a lot better. It's not like the game was doing anything beyond accepting the interpreted data from speech recognition software. So blame MS for their crummy voice solution, I guess?

@mrsmiley: It doesn't require more training, it requires a speech model for bigrams and trigrams, that Microsoft speech can take as input. We don't have the speech model in the version that Vinny used. But the best solution would be to reach a deal with Nuance (it is a commercial issue, not a technical one).

Your points about commands that should have been understood but weren't is well taken. Here's what I posted on that in the main GB thread:

"Shut" seemed like another problem, and when he tried "close the drawer", Jimmy let him know that he was already in "close" proximity. Infuriating.

*** No, Jimmy let him know that it's already closed. I agree that 'shut the drawer' should be supported. Other things that Vinny tried and should also be supported:

- 'more' (we support 'again').

- 'lay down X', 'turn this', 'repeat X', 'speak X', 'slide X', 'turn off X', 'turn off', 'shut down',

- put X back (we do support 'put X where Y was' so 'put the pepper grinder where the salt was')

- 'are we done?', 'what am I doing?' (only what we can tell from the interaction), 'tell me about your commands' (we have 'what are your commands'. Maybe not 'where I click', it's out of the experience).

- We need to improve generation (I am now for the first step ready)

- Vinny was right about the shoe rack, it is an item that had changed between photos.

- When teaching Jimmy a new command, Jimmy insists you teach him based on commands he already knows, one step at a time. Jimmy fucked up simple commands already, there is no reason to assume he would be smart enough to understand a multistep process of picking up the grinders and putting them down in each other's places.

*** As I said, we actually support 'put X where Y was', so that actually does work.

This is an important point, and one I am hoping @botcolony actually addresses. This is more than misleading, it's a straight up lie. I think if @botcolony is here to defend the game, he should start by addressing this very important and distressing piece of information. If you're going to charge real money for a game, even in Early Access, you should be honest about it. I wonder what our friends at Gamasutra will think about this?

@mb: Thank you. Those quotes are not coincidence. They're deliberately crafted, and I'm sure the devs behind this game are smart enough to know what they were doing. It's simply unethical, and @botcolony is sidestepping the core problem with random links saying "I dunno, it's somewhere in here?", in addition to ignoring the other two that are equally gross. He's also diverting people who call him out in comments, like @vierastalo's, to this thread...in which he addresses nothing but the technical complaints (which are also of substantial concern).

Well let's give him an opportunity to respond, since I directly asked him the question. We just aren't going to permit anyone to use the forums as a platform to both complain about how someone played their game while also touting it's technical achievements whilst simultaneously ignoring direct and important questions from the very community they are trying to engage. But like I said, let's put away the pitchforks for now. It could just be a mistake...

Yeah someone else posted about the fact that the game isn't using it's own speech engine (hence having to use your version of Microsoft's speech calibration tool). That certainly explains a lot, but as I mentioned in my reply, the issues with the game transitioned from speech to actual gameplay in the last half of the video or so. I agree with what you said in response to someone else about not allowing voice recognition unless a specific score is reached. In it's current format, there is literally no feedback as to how well you are doing in training. This could just be a limitation of the MS speech software, but that doesn't make things any better. Direct feedback on how many words were recognized for each sentence read, are at the very least a final score at the end of the training, would be great if it's possible.

But again, the main issue itself doesn't seem to relate to voice recognition as much as does the commands themselves, or the robot's actions in response to valid commands. You guys are really fighting an uphill battle with this game, and I feel for you. Not only are you dealing with Microsoft's inferior voice recognition software, but you are creating a game that runs off specific, user-created commands. Any game that allows the user to type in or say commands to make things happen is always going to frustrate the user in some way. I remember old games like the Quest for Glory series that allowed direct command input from typing. Those games were frustrating because there was always a disconnect between what the game wanted you to say, and what you wanted to say. Essentially, it's always the developers vs. the gamers in this genre of game, and there's no easy way to get around that.

My closing thoughts for you and the team, since this is still just an alpha build of the game, is this:

PLEASE test this game by having "normal people" play it. Not developers. Not family members. Not game critics. Not anyone else who is aware of this project. When you do this, takes notes, and keep an open mind. One of the biggest issues I've seen during my time as a game critic (and also a QA tester for some games) was the "developer bubble", especially with independent studios. In the developer bubble, everyone is super passionate about their project, but that passion causes them to make a game that they want to play, rather than something that a larger audience can play and enjoy. Sometimes getting honest feedback hurts, but there is no downside to discussing and considering it, as you seem to be doing here. I really hope you guys continue to consider feedback for Bot Colony, as it really is an interesting game, and the premise isn't like anything I've seen before.

Cheers!

People are acting like it's Vinny's fault, like the main problem was it mistaking what he was saying. Why that game sucks is that even when he typed the correct thing, in multiple grammatical ways "place the shaker down" "put the salt shaker down" "move the salt shaker to the left" the robot never did anything remotely right. Even when Vinny clicked on one side of the huge counter, the robot went all the way to the other.

The only one mentioning it was remotely Vinny's fault is one of the developers. Plenty of people (including me in my above comments) have pointed out that the game didn't even seen to work when Vinny typed commands.

Oh boy. Maybe take the money that you didn't spend on 3rd party voice recognition software and hire a PR rep.

Complaining that someone was playing the game wrong because they didn't manually "correct" the voice input (which is the entire and only hook of the game) is honestly just sad. He didn't not do it to "put on a good show", he resisted because the point of the game is to talk to the robot. There have been tons of interviews on this site with actual devs themselves saying that, despite how challenging making a game is, and despite how impossible it is to anticipate player behavior, the end user shouldn't care about any of that. We can't give certain games a pass for not working because making them is hard.

And of course, with a game like this which uses specific commands, there is bound to be at least some frustration that the dev can't stop. But when I'm typing the wrong words into Zork, it clearly tells me that it doesn't understand... instantaneously. If I entered a command and then had to wait for a bland robot to animate for a second, stand still for a while, and then respond with something confusing, I don't think I would have ever played any text-based adventures.

I'm usually the first to jump in front of forum nay-saying for projects like yours which, though currently pretty cumbersome, are taking huge and potentially exciting risks on new types of gameplay. But your behavior here has pretty much soured any of that good will.

It's great that you're trying to engage with users and answer some questions, but maybe try and use a little restraint in the future.

@mb: I agree this needs to be addressed.

@botcolony As pointed out, all 3 of these "quotes" are severely out of context and don't belong in the "reviews" section of the game if anywhere at all. The Destructoid quote is actually commentary on a preview trailer, chopped in a way that changes the meaning and context. In this case you actively changed the meaning to sound like a favorable review rather than wild, hopeful speculation from viewing a demo reel. Though bad, this is less distasteful than:

The Gamasutra quote isn't even from Gamasutra, but from a featured blog on their site from your own producer which you then attributed to the gaming site that hosted the blog. This goes quite a bit beyond just chopping quotes for a more favorable understanding, and moves a bit more into active deception. You need to be VERY careful doing this kind of thing if you want a community to look at your work favorably.

"The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.

The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company."

Lastly, the Game Informer quote is the least terrible of them, at least you actually added what was written and attributed it to a proper outlet, though lacking the context. Though still not a review, its simply a features listing essentially.

And finally, I don't think Vinny went out of his way to actively show your game in a bad light. In fact, I've picked up lots of games played by him that weren't finished/polished that he has had similar troubles with though not nearly as severely as your game. Some of the stuff is fun to watch him try and break, and he does that admirably at spots in this quicklook, but once he's trying to actually play it's a train wreck. The simple fact is the voice recognition is dodgy at best and should either require or steer players down a path of helping it better understand commands through more training. Additionally the voice recognition seems to have *absolutely* no sense of context to the surrounding environment. Calling something a sautee pot then failing entirely to recognize the voice command directly after isn't acceptable if voice recognition is going to be a central focus of the game. And lastly, it has the issues of every text parsing game ever, if you don't know the exact string of commands the computer wants it acts really dumb or simply doesn't understand. This is frustrating and not a fun mechanic to play with for the majority of people.

yo phillips this is hilarious

@mb: We did not make any deliberate attempt to mislead anyone. Our company started as a R&D company in AI and Natural Language Understanding in 2001, and until 2007 we had no idea we would one day put out a game. We learned about making games by attending local chapter meetings of IGDA. We don't have any senior person on board experienced in the game industry (although we employed at one point more than a dozen Ph.D.'s). Most of our developers never worked on a game, unfortunately. The Steam store page has a field Reviews, and the same store page format is used for Indie games (that don't have reviews) as for other, AAA games (which do have reviews). Since we didn't have a game before Early Access, how could we have game reviews? The next best thing we had was articles about the concept, the technology, the story, etc. - and we quoted them in the Reviews field, since it seemed to be the only place we could mention them. There's a lot of novelty about Bot Colony - you can shoot, wreck cars, fight in a million other games, but we elected to do something different. So, I think that posting articles about something that's truly genre-defining, to explain what the novelty was, was OK. If you define review as a hands-on gameplay session by a game journalist, what we have there is not reviews, but people do write reviews of films, books, shows, etc. which are conceptual, so it's a fair use of the word. In the meantime, there are gameplay reviews on Steam of people who actually have played the game, and prospective buyers can rely on those.

@botcolony: That's all fine and good and everything, but as stated by a couple of Giant Bomb community members in this topic, the Gamastura link on your store page goes to an article by Mike Rose where the quote doesn't even exist. Instead, the quote actually comes from Eugene Joseph's personally submitted blog, here. I'm sure that is just an honest mistake that is easily corrected, though.

The same misattributed quote can be found on the official Bot Colony website as well.

Can the quote technically be found on Gamasutra? Yes. Is it misleading? I think any reasonable person would also have to agree that it is.

Sorry Phillips, I do not understand But people do write reviews of films, books, shows, etc. which are conceptual, so it's a fair use of the word.

You just admitted to the core issue of this game: it feels like it was made by people who don't know how to make games. I have tried to offer my own experienced feedback from a gamer/critic's standpoint. You have heard countless bits of feedback from actual gamers both in this thread and the comments on Vinny's video. In addition, you have seen that, whether or not you deliberately attempted to mislead anyone, you are misleading people. I feel like your next steps seem pretty clear here, but in case they aren't, let me lay them out for you:

- Remove the out-of-context quotes from the Reviews section on Steam. It doesn't matter what other people do. Finger pointing in this business makes you look worse.

- Start listening to feedback from actual gamers about your games. If almost no one on your staff has any experience making games, doesn't it make sense to listen to those who have played/made/reviewed games?

- Instead of stating that players don't know how to play your game, take a step back focus on how you can betterteach players how to play your game through the design itself. Vinny has been playing video games his whole life. If he can't figure things out after playing for 90 minutes, that seems to say something about the game's design, no?

What you do from here is up to you, but you can't say we didn't try to help.

@botcolony: I really don't understand what you mean by this:

If you define review as a hands-on gameplay session by a game journalist, what we have there is not reviews, but people do write reviews of films, books, shows, etc. which are conceptual, so it's a fair use of the word.

I have always understood "reviews" as criticism by a professional in regards to the worth of a product and that criticism is directed towards a potential consumer of said product. I believe that most people share that viewpoint as well. By holding random blogs and preview coverage as being on par with "reviews" you are stating that the following articles are providing advice to consumers that they should buy Bot Colony. The fact that some of these articles do not do this makes your use of these quotes highly questionable.

Honestly you do not NEED to have a "Reviews" section on your Steam profile until you have finally released the game and it has been assessed by members of the gaming press. Why not wait until then?

Exactly - a good example of this is Starbound, which is also still in Early Access. Check out their Steam store page. It has the Early Access notification, news from the dev team about updates and features, has some video and screenshots, lists the system requirements, and then there are clearly marked Early Access reviews from users. That's it.

That would be my advice. There are plenty of games on Steam that don't have any reviews listed, and there's certainly no obligation for there to be any on the entry for Bot Colony. The currently listed pull quotes don't come from sources that would hold up to even the loosest definitions of the word "review", and any attempts to depict them as such is frankly dishonest.

Did you mean?