The hardest part of level design in Mario Maker is playtesting. You can't do any sort of limited release and you definitely can't see how everyone plays it even once it does release. All you can do is play it yourself extensively and hope for the best. This is what I've been doing. Designing levels while occasionally testing each part and trying them in as many different ways I can think of, making minor tweaks along the way. I've done this because I believe that making as many different approaches as viable as I can manage is the right approach. I do this and see clear rates in the 10%-30% range and feel good. I’d never seen my levels played, but the average person took a few deaths, clears it and is likely to star it. Seemed a good enough impression to me.

Yesterday however that changed for me. Yesterday, for the first time, I did actually get the opportunity to watch people play my latest level. Beneath The Runway was one of seven courses in the second week of Mario Maker Mondays, a blind race of Super Mario Maker levels which dozens of runners and players participated in. Exciting! I would actually see players (albeit skilled players) take on one of my stages! Unbelievably cool!

The first two levels go quickly. My level comes up. I excitedly IM my buddy watching the race. The commentator sells the idea of this rundown abandoned runway as runners run down conveyors speeding along its length. At the end coins rise into the sky where becaped runners will eventually fly into the sky and soar to the exit. First though they must venture into the unseen area beneath where Boos, Thwomps and lines of yellow blocks breakable through spin jumping lie. Things go about as well as I wanted for the runners. There's a few deaths here and there for the four on stream but they get through the level in short order all the same. My friend complements my stage and I feel good. It showed really well.

The race goes on and racers start to finish around 22 minutes in. They switch out runners further and further behind in levels 7 and 6 until I tune out around 50 minutes in. At least, I tune out until I'm told that my level was back on stream a few minutes later. I didn't see how that was possible. I pulled back up the stream to witness a struggle.

I like to believe my levels are fair. I’ve been conscious that there’s a lot of things Super Mario Maker does not teach. There are people who’ve never played Super Mario World. Players who don’t know how to spin jump, don’t know spin jumping breaks blocks, don't know it allows you to bop on Thwomps, don't know how to fly. I tried to account for this as much as I could. I introduced spin jumping to break yellow blocks by trapping Mario in a room with a breakable floor, a mushroom and the letter R. I showed that Thwomps can be spin jumped on by making the next time you break blocks you landed on a Thwomp. But flight? Couldn’t do it. Didn’t see a way to test player’s understanding of that before the actual flight.

You can imagine my joy as Mario Galaxy runner allyoucaneat1620, who’s never played Super Mario World, cleared the last actual jump in the level (tricky to do if you don’t hold Y in the air turns out) and you can imagine my horror as allyoucaneat1620 went out onto the runway, jumped up and slowly floated to down to his death. What was meant to be the ultimate payoff backfired. I didn’t like it any better when it happened again a few minutes later.

I panicked and did what you shouldn’t do during an actual playtest. I went to him stream and explained that he had to hold down Y to fly. You really shouldn’t do this because you’re not always going to be there to explain to everyone who struggles in this way. I wasn't exactly alone in this but still. Soon the main stream ended and hundreds of viewers flocked to cheer him on. His Twitch chat became a blur of “I believe you can fly!” Once he got back to the end he took a few quick practice flights over the runway itself and, after confirmed he could in fact fly, he soared off towards the exit finally finishing my level. Course cleared.

Whether or not you see them players like allyoucaneat1620 exist and that’s what playtesting can highlight. It reminds you that not everyone plays in the same way, even if you honestly believed you were accounting well enough for that. In all my tests I never tried doing runs where I let go of Y because that would honestly have never occurred to me. Additionally it had never occurred to me that the space over the runway could be used to practice cape flight for those who are inexperienced. If I had seen allyoucaneat1620 play my level before it went live maybe I could’ve put some extra coins above the runway to encourage that sort of bonus flight and give him people who struggle with flying the idea to experiment sooner. Those are things you don’t see without playtesting.

Currently Beneath the Runway sits at 14% completion rate which is well within my normal range, but I feel uneasy about it. It turns out looking at a statistic is one thing and seeing something struggle is quite another. The difference between suspecting there might be issues and seeing and believing. That's the difference playtesting makes.