Making Music Through Video Games

Written by Michael Manning

mickdadude@gmail.com

myspace.com/mikmanning

The Future’s in Our Hands

Kevin Kelly, founder of Wired Magazine talks about an idea of technological evolution. He uses the internet as the embodiment for this technology through a talk he gave for TED. TED is a forward-thinking annual conference for Technology, Entertainment and Design. Kevin states that since the conception of the web, all the computers, handhelds, cell phones, laptops and most consumer electronics, all these connections are creating what he calls, ‘The One Machine.’ All of our handhelds and electronics are acting as a window to this single one machine. This machine he says is the most reliable machine ever made; it is always on and contains over two hundred and fifty five Exabyte’s of memory. This machine also doubles in size and complexity every two years and his calculations show that by the year 2040, the machine will exceed humanity in processing power. One of the consequences of this is that humanity will become co-dependent to the machine.

 Natural evolution of this technology over the next twenty years will show that everything in the world will be linked in a way that this paper will be linked to you, your room, the items in your room, your house, the contents of your house and so on all the way back to the one machine. These links will physically exist within the machine and not just in our minds. Kevin’s most interesting point is that the total personalisation of this requires total transparency, he uses Google as an example, ‘Google, I can’t remember my phone number, I’ll just ask Google, we’re becoming so dependent on this I don’t even try to remember things, I just Google it. If you think about it, the dependency we have on this other technology, the alphabet, we’re totally dependent on that, it’s transformed our culture, we can’t imagine ourselves without the alphabet and writing, so in the same way we’re not going to imagine ourselves without this other machine being there.’

The relevance of all of this is to say that the future really does mean ‘connected,’ and this future already exists within our phones, laptops, mp3 players, game consoles and etcetera. There is not going to be a technological breakthrough where a scientist somewhere reinvents the wheel in order for this connected future to happen it is going to happen completely naturally and, unavoidably. It is what is happening right now.

So with that in mind, this paper will talk about the ways people can interact with sound, music and each other through this new technology. I will cover the concepts of new ideas and analyse some examples of software that exists today. The main focus is on consumer aimed software development and I will be using the iPhone as an example for the current standard in hardware and its software distribution network. Also I’ll be putting a lot of attention to the artists and developers. There are many people out there who are really pushing the envelope for what I’m going to call, ‘interactive sound.’ Brian Eno, long-time music composer, producer and consistently innovative thinker, is taking full advantage of what the iPhone can offer people through use of his co-created Bloom software which is a touch based minimalist music program. Its’ simple and elegant interface makes creating minimalist music instantaneous and responsive. Similarly, Toshio Iwai, both a seasoned software developer and installation artist has spent his life creating spectacularly forward thinking software’s, his most recent being Elecktroplankton for the Nintendo DS which is similar to Bloom in its touch style interface and minimalist influences. This game features a suit of play areas, each area providing a unique interface which allows you to improvise music. This objective free game was met with many plaudits from game critics.

This takes me to my primary argument, interface. The hardware of these new handhelds, especially the button less iPhone, is forcing people to consider new ways of developing software. So too however, does it give developers the opportunity to reinvent music interfaces for the general public. If you wanted to make a simple music sequencer for the iPhone, would you copy the template set by the PC or Mac equivalents? I do not think this would be a success, the small screen forces detailed track layering and basic editing functions a frustration, plus the touch sensitive screen would mean your fingers would be covering up a large part of the image, making navigation a difficulty. Aside from that, creating this style of sequencer would put off a large number of the installed iPhone user base who, do not have any knowledge of how sequencers work, let alone have any knowledge of how to make music or even if they would want to. So a new style of interface could bring some interesting developments; by taking Toshio Iwai’s example, an interface that takes inspiration from video games instantly opens up an audience that would be put off using a typical music programme. So is it possible to create stylised and abstract music interfaces for handhelds? Or even making a goal focused video game with narrative, which includes all the basic functions of a music sequencer wrapped up in a fun and interactive package? Most defiantly yes, I will explain how, but know that the latter hasn’t been created yet.

But what are the benefits of creating such a thing? Well for one, I personally get excited about anything that allows me to compose or improvise music in a new way; any new perspective will make you think about music differently thus, can offer the freedom for new genres of music to emerge. On top of that, a program such as this could make music composition universal again. In the way that someone with no prior knowledge of how to make music would be at the same skill level of someone who is a seasoned Logic producer. Also it can act as a tool in which to educate people, in a form which isn’t forced or too orate about music and sound. Getting people to listen not just to the programme but such a thing could inspire people to suddenly realise the soundscape which surrounds them. I am aware however, that such programmes could never be as detailed and intricate as a full-featured computer sequencer on a device such as the iPhone, full traditional compositions should be left in the studio.

I, Phone

Just as author Isaac Asimov defined a series of rules and guidelines towards the future creation of artificial intelligence, so too does it seem that Apple are defining their own set of guideline as to the future of how we perceive what a portable device can be. With a strong enthusiasm for user created content and interactivity with its portable hardware and an aesthetic that is universally accepted,  Apple are quickly making its mark as the leader of mobile devices, so much so that rival mobile phone and handheld gadget companies seem to be copying the template set by apple providing its users with similar features. Now that the machine is getting into the hands of more and more people, how does the software reflect the capabilities of the device? With such a large user base and effective use of new technologies, for the innovative programmer there is a potential for not only financial success through the digital shop the ‘app store’ but also for the invention of new ideas.

Since its release on June of 2007 there have been thirteen million iPhone’s sold. In comparison, the total number of Microsoft Xbox 360’s sold after one year was one point five million units. I’m using the Xbox to compare because this is a machine that has an online service which matches that of the iPhone in terms of how consumers interact with it and each other. The Xbox has a functionality about it which could potentially enable its live connected user base to invent new software. It is the same for PC and Mac also but those platforms do not have a structured service or a universal interface which links each user to each other. The PC could use a service like Steam which is a game software distribution and player communication service. Which brings users together in a way that allows them to communicate, play games or share information with each other, but not all PC users use steam, and not all PC users use the same machines or operating systems to run it on. The Xbox live service or apple’s handheld technology brings people together simply because all users have that service/device in common; this is quite a powerful thing. 

Firstly, as is the case with the iPhone, every user is familiar with the device, how it functions and how it links its various technologies together. Let’s take an upcoming iPhone video game, Drug Lords. This is a fully online connected video game. The objective is to buy and sell drugs on the virtual street; the game uses the iPhone’s in built messaging system to talk with both the computer AI and real life players. Here you can buy, sell or get a loan of money from the other players/AI. There is an exciting feature which uses the inbuilt GPS system to display where all the other real-life players are in your area. So you can see physically who is playing the game and where, some would say this is an infringement of privacy or personal space but you have the option to turn the feature on or off, it is in fact by default off. The game uses the familiar iPhone interface as a means of playing the game, so you don’t need to learn much to start playing, it is as seamless as it would be to access your emails or send a text message. Fantastically, none of this interferes with your other iPhone apps, so you won’t be getting messages from other drug lords while you are not playing the game. So here is a game that couldn’t exist in the same way as it does here on the iPhone, by taking advantage of the inbuilt messaging, internet and GPS features the developers, (A-Steroids,) have crafted something unique.

Let’s take this principle of creating unique software for the iPhone a step further and more relevant for this paper. Brian Eno, musician, composer, producer and theorist, a man with the title of the ‘father of ambient music’ for his early work in the genre. A legendary character in the world of music and sound who has worked with David Bowie, U2 and John Cale of the Velvet Underground to name a few.   He is an active part of a relatively new field known as Generative Music, the name was given by Eno himself, what it means literally is ever-different and changing music. Brian Eno teamed up with musician/programmer Peter Chilvers and created the application called Bloom. Technorati.com describes it very well ‘Part instrument, part composition and part artwork, unlike most music -creation apps, it doesn’t take an existing musical instrument and cram it awkwardly into the iPhone’s interface. Instead it creates a completely new instrument designed specifically for the iPhone.’

When you start the programme, you are welcomed with two simple choices, ‘listen’ and ‘create’. The Listen mode generates a constantly evolving composition for you to listen to making use of all the sounds and features of the iPhone. A simple shaking of the device resets whichever pattern it is on and a new one starts over again. The Create function allows you to influence the music, at first all you hear is a drone but when you touch the screen a small circle will grow out of where you pressed, this circle represents a sound and when the sustained note is finished, the circle resets itself. Further presses dependant on height and width, play different notes of the same sound and the rhythm to when you press is quantised, creating a repetitive pattern of melody. You also have the option to change the scale of the notes and the sound of the drone.  This is a simple interface and it works very well, it is not an extremely deep application and the sounds do get a bit tiresome after a while of play. I think this application achieves its goal though as a meditative and clever musical improviser which is open for all not just musicians to use. The key to its success (as well as having Enos name to it,) is its title, Bloom, it is very sweet and universal and is much more appealing than, Generative Minimalist Improvisation Application.

There is another application called Beat Maker to contrast Blooms user friendly approach and price (£11.00 to Blooms £2.50.) Beat Maker was made by a bedroom producer Vincent Bongiorno, there is very little information on the web about this man, and there most certainly aren’t any books which relate to him so I’ll focus on the application exclusively. This programme assumes the user has prior sequencer knowledge as the layout is very similar to the current generation of computer music applications. Surprisingly detailed this basically allows you to make your own step-sequenced loops and then arrange them as blocks within a sequencer. There’s effects and mixers and even the ability to import your own sounds into the programme. Impressive stuff, within moments you can find yourself creating intricate patterns and arrangements all from the palm of your hand. This is however, assuming you have prior knowledge of sequencers. The programme starts off by welcoming you with pages of instructions on how to use it. This would be a necessary read for anyone not familiar with programmes such as this and with that; there are many consumers that may possibly get frustrated with it. However, as someone who is used to this sort of interface the creator has made very good use of the hardware capabilities.

According to thinkabdul.com, the iPhone’s hardware comes equipped with an ARM1176JZF chip with processing capabilities of up to 700MHZ, 8GB of Flash Memory, a 3.5inch colour screen, 2.0 Megapixel Camera, Wi-Fi and Bluetooth, rechargeable battery, Accelerometer and Touch Screen architecture. This is a relatively powerful machine as compared with other Phones or PDA’s of its liking. There are other devices which surpass it for raw hardware statistics like the Nokia F800 but none can compare to Apples software distribution networks iTunes and the App Store.

The iPhone itself runs off a modified version of OSX called OSX Mobile, this OS comes pre installed with a YouTube channel, GPS Mapping system, Weather, Clock, Calculator, Stocks channel, Calendar as well as the iPod, E-Mail, Phone and SMS functions. So out of the box already there is a suite of applications which make use of the phones capabilities. On the App Store you will find 3,456 third-party applications. This range from content made by independent developers or LTD companies, fantastically either party can decide what to charge for their content or to release it for free.

Even though there is a large quantity of content readily available for the iPhone, a great deal of it are muddled, pick up and drop style toys like a 50 pence Zippo lighter application which is simply an interactive animation of a lighter opening. Edge, a very respected gaming magazine did a feature on the iPhone and its applications saying, ‘Despite the instantaneous success of the App Store as a money-making machine, there’s something indistinct and gently underwhelming about a lot of its more playful output to date. IPhone games aren’t bad, many of them just aren’t terribly memorable; you may touch them, shaking, prodding, japing and tilting your way through marble runs, trivia quizzes and side scrolling shooter, but they often fail to have any lasting impact in return.’ The music applications fall under the same bracket in that they do not provide a service like Fring which is a Hotmail and Skype messaging service. Also the line between music and game software’s for handhelds have defiantly blurred in recent times as exemplified with Bloom and the work of Toshio Iwai.

Toshio Iwai

Toshio Iwai was born in 1962 in Kira, Aichi Prefecture, Japan. He has been described generally as an interactive media and installation artist. He studied Plastic Art and Mixed Media at the University of Tsukuba where he began producing installation art which combined early animation techniques with modern ones. In 1985 he won the Grand Prize at the 17 Contemporary Japanese Art Exhibition for his piece Time Stratum 2, Iwai himself explains ‘In this installation, I placed 120 paper human figures on a motorised spinning disk. I set up a video monitor above them, while strobeing the light down, the paper figures all burst into motion. By using a video monitor as a strobe light, I could change the colour and speed of strobe lights with the music. The images are transformed with changes in the light.’ This was taken from an autobiographical essay called A Short History of the Works. Toshio created his first videogame in 1987 called Otocky.

Otocky has been credited by the Game Innovation Database, as the first Media art videogame. The game was developed for the Famicom Entertainment System and features colourful, abstract graphics which are impressive for the Famicom. You essentially play a side-scrolling shooter where you guide, what looks like an odd, alien superhero...Or something like that! The simple graphics of the time leave a lot for interpretation; it is only recently that gamers have not had to leave much to the imagination as they play. So as you control this superhero you can fire in multiple directions around you, up, down, left, right and the diagonals in between. It is here that the game fuses music with your actions. When you fire the multicoloured balls from around you, you trigger the sounds of the melody which accompanies the bass and drum backing track. So as you fly around and shoot things you are also dictating the performance of the melody. There is a blog post from a developer who works for Hand Circus which is a micro-studio which creates games and exhibits. In this post you can find his very informed reaction to playing Otocky including a video which I recommend you see. 

The post entitled Ode to “Otocky” (by Toshio Iwai) compares the game to that of a similar, more popular music/game hybrid Rez which was released first on the Sega Dreamcast and Sony Playstation 2 and is now featured on the Xbox Live Arcade store, Rez was created by Tetsuya Mizuguchi in 2002 (I also recommend you experience this game). ‘When I first played Rez, the connection between the audio and your ability to affect that audio as a player really made an impact on me. The way that each level was constructed so that elements were created to guide you to add percussive events to the soundtrack and so that sections of the level corresponded to sections of the main track felt so unique and such a suitable marriage.’

The poster who, strangely is unnamed goes on to say, ‘It’s only after playing Otocky, created in 1987(!) that you realise that Rez certainly wasn’t the first to explore this area. One of the things I love so much about Otocky is that you can’t really go wrong with the composition you create. As you pass through the level, you pass through successive chord progressions, and the sounds generated by firing map onto a single note from the current chord, so it’s always in tune. It also snaps to the nearest beat so that the note always hits on time - so while you feel total ownership of the notes created, the notes always sound good.’

Toshio was a resident artist at the Exploratorium in San Francisco, California from 1991-92 where two of his works are now a permanent fixture of the museum, Well of Lights and Music Insects. Toshio describes Music Insects, ‘You can see four small Insects walking on the screen. These Insects have their own sound. They are kind of walking musical instruments. Using the mouse, you can choose a colour from this palette; you can draw something like paint software. If an insect crosses a colour dot, the insect makes a sound and light pattern.’

The concept behind Music Insects led to his work for the Nintendo DS which uses a similar interface. Electroplankton was released in 2005 and offered ten different interactive pieces where the player can compose their own music within the governed rules of the area. Edge describes the game as ‘off the wall...In one part of the game, neon-pink tadpoles are shot toward a water fern. When a tadpole hits a leaf it makes a dulcet tone. Using the stylus, players can rotate the leaves to create a tune when the little guys hit them.’ Other areas include; places where you spin in turn six disk-like creatures to create padded overtones, each generating different colours. A four channel sampler where you can make noises into the DS inbuilt microphone and the fish on-screen record the data and play it back looped. Electroplankton is an incredible example of how music software can make full use of the specific hardware it was designed for. This is not my idea of a fully fledged sequencer disguised as a game, but it’s defiantly a creative, improvisational music focused game powerful enough to use as a performance tool. One of my main gripes with the game is the lack of an ability to record your performances; this requires an external device to record the sound and a separate one to record the visuals. The lack of this may have been due to the limited memory capacity of the Nintendo DS hardware.

Unfortunately on release Electroplankton did not do so well only selling 5,000 copies in its first week in Japan, although it is speculated that this is because one of the DS flagship titles Nintendogs was sold in the same week, which reached figures of 150,000 copies sold. Aside from that though, Electroplankton has gone on to achieve critical success and a solid cult following, it is a game which is firmly placed in history as a fine example of interactive sound media.

Toshio’s latest work is in takes form of the Tenori-On. This develops the things he’s learnt over the years with software and applies them to hardware. Released in 2006 and Co-produced by Yamaha this £684 hardware digital instrument could well be Toshio’s proudest moment. It features a 16 x 16 LED button matrix which acts simultaneously as an input controller and display. Through this you can play and record over sixteen layers through multiple modes. You can give your melodies and beats characteristics which you could never have done with previous hardware for example; you can set up a sequenced loop, then press the line of a melody in time with this loop. The device will remember and repeat this line but, as the track bar reaches those notes again you could set it to make them bounce. So they bounce with a sort of gravity onto the bottom of the device. As well as this the device features the same LED’s on the back which you can hold up to show to an audience. In an interview with Future Music, Toshio explains. ‘I wanted to create a very unique hardware and software together. This is not just a musical interface, this is also a musical instrument...I have created software for the Nintendo DS, Electroplankton for example. But writing software is so easy, anyone can do it, but making hardware at the ‘’big factory’’ [pause] ‘it’s a kind of dream!’

So it would seem that Toshio Iwai has his heart set on the hardware/software synergy, I personally couldn’t find that more encouraging for such a visionary developer of software. The ability to create purpose-built hardware for his software puts him in an extremely rare position which very few artists can find themselves in. If an artist is to have such power however, I’m glad that it is Toshio Iwai. He has constantly proven himself to be a figure of whom to take great inspiration from, and if the day comes when we are creating vastly different interfaces to make music on, we must take note of this man’s body of work.

Concept

From my research over the years I have found there to be many ways to interpret sound. The information I’ve learned has completely re-shaped the way I approach my audio projects and, generally how I hear the landscape around me. It is my intention to create music software for the iPhone and other handhelds in the guise of a videogame, the ideas/concepts I have written take direct inspiration from what I’m about to talk about. I feel that these findings should be one of the cornerstones of when it comes to creating any audiovisual composition. So I’m going to briefly talk about listening modes and other concepts.

French composer, film maker and critic Michael Chion in his book Audio-Vision talks about the three listening modes. ‘When we ask someone to speak about what they have heard, their answers are striking for the heterogeneity of level of hearing to which they refer. This is because there are at least three modes of listening, each of which addresses different objects. We shall call them casual listening, semantic listening and reduced listening.’ Casual listening, he describes, consists of listening to a sound in order to gather information about its cause. Semantic listening is what refers to a code or language to interpret a message; he uses spoken language as an example. This listening mode is similar to something that sound artist R Murray Schafer calls signals. The book, Tuning of the World and now sourced from a previous paper of my own says how signals are sounds which are used to communicate; cities are full of them, from the beeps you hear telling you it is safe to cross the road to the ‘ding’ the elevator speaks when you reach your desired floor. Finally, Chion talks about reduced listening, ‘Pierre Schaeffer gave the name reduced listening to the listening mode that focuses on the traits of the sound itself, independent of its cause and of its meaning. Reduced listening takes the sound as itself the object to be observed instead of as a vehicle for something else.

I don’t want to elaborate too much on the subject but with this is mind you can break apart these new and emerging software’s of Brian Eno and Toshio Iwai and place the audio into these separate categories. I feel that it is important to be aware of the information provided from Michael Chion and R Murray Schafer for when software is created with audio in mind. This allows you to think more deeply about what your sounds are doing in the software, what role they play and the meaning of their existence. I do not think that sounds should be placed just because they sound nice but they should have a meaning of their own; why do they sound nice? What is the relevance of its niceness within the context of the programme? This is only really pertinent for the music software which behaves more like a videogame, obviously for a programme such a Cubase, the ideal for these programmes is for the user to have total creative freedom, to start from a blank canvass.

My realisations through the research I have found have shown that in order to create some software for which to make music on, aimed at non-musicians but in a fun, easy and vastly different way the creator has to inhibit certain limitations on the user. In Bloom, the notes available for you to play are pre-picked from a scale, so where you can choose to play a note, you do not have the freedom that you would have with a piano. It is impossible in Bloom to play the ‘wrong note’ if such a thing exists. So Musicality, is something you have to be aware of, most non musical people wouldn’t know what keys fit with what modes or scales, in that regard there has to be a musical structure already in place within the programme. Making something fun is what I’ve learned from Electroplankton, if you can wrap up a neat, detailed and powerful music creation package within a videogame or similar format you then give the opportunity to people who would never before have much interest in composition or music in general to try it for themselves, even if they do not realise this themselves as they play!

Finally, in preparation for the new technological evolution, dare I say, revolution, that is happening, designers, artists, musicians and programmers should take a distinctly different approach to software development. These new devices such as the iPhone or even handheld game consoles have their hardware designed in such a way that completely new ideas which couldn’t have existed elsewhere can emerge from it. If we take the examples set by Brian Eno and Toshio Iwai including the fundamentals of how to work with sound, as talked about from Michael Chion and R Murray Schafer, plus a keen eye on what is to come, impressive, unimagined things can come. This could potentially change the way that people see, work and play with music and the world around them.

I’m going to finish on a quote from Kevin Kelly, of Wired magazine ‘Imagine Van Gough being born before the technology of cheap oil paints, imagine Hitchcock without the technology of film. Somewhere today, there are millions of young children being born who’s technology of self-expression has not yet been invented. We have a moral obligation to invent technology so that every person in the world has the potential to realise their true difference.’