January 4, 2008

Toshio Iwai talking about the visual-musical interface

Filed under: Imagination,Interactivity,Sound,Transposition — Graham Coulter-Smith @ 12:01 pm

Toshio Iwai, Composition on the Table, 1998–1999.The video footage provided below consists of an extract from a major presentation given by Toshio Iwai at Ars Electronica: Simplicity the Art of Complexity, in 2006. In this segment he gives insight into the inspiration for his remarkable visual-musical interfaces such as his gallery-based interactive visual music installations, his compilation of such ideas into Electroplankton for the Nintendo DS and his invention of a new visual based musical instrument the Tenori-On, which Iwai developed in conjunction with Yamaha (link 1 [uk] link 2 [global]).

Iwai’s English can be a little difficult to understand and it is especially important to appreciate the beginning of the video in which he explains that the original inspiration for his excursion into visual music was his fascination for a traditional music box that played music using holes punched in a strip of card. One scroll played ‘Happy Birthday’ and Iwai explains that he looked at the pattern of holes that produced this melody and considered the relationship between the visual and musical patterns. He found the visual pattern as aesthetically pleasing as the music and wondered what would happen if he reversed it. He experimented by turning the scroll upside down and feeding it through the music box. The result is very interesting:

Watch the segment from Toshio Iwai’s Ars Electronica 2006 presentation (VIDEO CLIP 8:42)

The resulting melody is quite beautiful but it sounds as if it were transposed into a minor key making it somewhat melancholic, Iwai calls his inverted retrograde theme ‘Unhappy Birthday’. What is especially interesting about Iwai’s experiment is that a visual pattern transposed into sound can possess aesthetic value in different orientations. In his presentation Iwai notes that some visual works of art can appear equally pleasing even when upside down. Interestingly, the pioneer of abstraction Wassily Kandinsky made a similar observation in his 1911 book Concerning the Spiritual in Art (also On the Spiritual in Art in the superior Peter Vergo translation).

Iwai studied computer science, and from a mathematical perspective one can consider the relationship between the pattern of holes in their ‘correct’ orientation and in their ‘inverted’ orientation in terms of isomorphism. The mathematician Douglas Hofstadter author of the Pulitzer Prize winning book Gödel, Escher, Bach, defines isomorphism as ‘an information-preserving transformation’ (Hofstadter 1980: 8). It is this quality of ‘information preservation’ that Iwai saw in his music box scroll experiment. But what is most interesting here is that the the aesthetic character of the pattern is preserved across two different sensory dimensions.

One could call the pattern of holes in the music box scroll a gestalt, which is to say an integrated perceptual structure. Via the tuned metal tongues of the music box the orientation of these holes is mapped onto the diatonic scale which is a sonic gestalt or integrated structure. It is impossible to play discordant music using the diatonic scale alone and Iwai’s various excursions into the visual-musical use this fact to map minimalist grid based patterns onto aesthetically pleasing sonic outputs. The diatonic scale can also be understood as a species of narrative structure where narrative is understood in the broadest sense akin to concepts such as pattern, gestalt, structure, system, game, language game etc. It is also possible to refer to Einstein’s connection between creative thinking and what he calls the ‘picture’:

What, precisely, is ‘thinking’? When, at the reception of sense-impressions, memory-pictures emerge, this is not yet ‘thinking.’ And when such pictures form series, each member of which calls forth another, this too is not yet ‘thinking.’ When, however, a certain picture turns up in many such series, … Such an element becomes an instrument, a concept. (in Holton 1996: 197) [emphasis added]

Einstein’s ‘picture’ is not ‘thinking’ until it recurs across numerous series. In other words thinking can be understood not only in terms of the “picture” but also: system, structure, pattern etc. It is also possible to make sense of the same picture, pattern etc. from multiple points of view. If we invert the pattern it makes sense in a different way. If we play a music box scroll backwards then the same pattern opens up a new perspective. Considering cognition and cultural construction in terms of repetition and transposition is evident in structuralism and poststructuralism and represents one of their most valuable contributions to the concept of art as involving multiple language games.

As an artist, however, Iwai developed the intersection between visual pattern and music further using his programming skills to create interactive installations that combine visual formations and sound formations. Unlike Starling’s more traditional approach to art where the viewer focuses on how the artist constructs and plays his or her creative game, Iwai’s work is in the form of an interactive installation. His brilliance lies in his capacity to move beyond the artist’s game towards the construction of creativity games that involve the viewer in a creative process. In Piano As Image Media, 1995, the viewer-participant uses a trackball to construct a (virtual) music-box scroll represented as a sheet of transparent material made up of a matrix of dots of light that moved upwards towards a piano keyboard. When the light dots reached the keyboard they activated the piano creating tonal structures without the need to know how to use the piano keyboard. A vertical screen ascended from the piano to the roof generating three-dimensional abstract patterns as interpretations of the tonal picture. Piano as Image Media was exhibited at the Play Zone at the Millennium Dome in London, 1999–2000. Currently (2008) Iwai’s work remains unrecognized by the fine art system. Perhaps he is considered too commercially oriented which is risible considering the contemporary fine art star system.

Hofstadter, Douglas, R. 1980. Gödel, Escher, Bach: An eternal Golden Braid —A Metaphorical Fugue on Minds and Machines in the Spirit of Lewis Carroll. London: Penguin Books.
Holton, Gerald. 1996. Einstein, History, and Other Passions: The Rebellion of Science at the end of the Twentieth Century. Reading MA: Addison Wesley (current publisher Perseus Publishing).


  1. links from Technoratidownload the whole presentation (2 hours): mms:// (copy this link into your Videoplayer VLC, QT…. consider the codec) Links Interview with Toshio Iwai on pixelsurgeon.summary of Iwais Ars Electronica lectureYouTube-Videos with Toshio Iwai relevancy The Tenori-On is of course interesting for my work – especially because it’s a commercial product. The older projects of Toshio Iwai that are software and hardware based are also quite exciting.

    Pingback by ResearchNotes — topotropic — April 10, 2008 @ 7:37 pm

  2. Kramer auto Pingback[…] der robert lippok hat das ger?t f?r die de:bug ziemlich anschaulich erkl?rt… und hier und auch hier gibts noch 2, 3 hintergrund infos… der herr iwai baut schon ziemlich lange am […]

    Pingback by Tsunami Crew Board ~ Thema anzeigen - Synthesizer und PC ade ??? — May 19, 2008 @ 11:55 am

  3. Kramer auto Pingback[…] at Ars Electronica 2006. It’s nearly impossible to hear what he is saying so you can read it at It’s a bit boring for the first 4 mins with a fair amount of unintelligble speech and a […]

    Pingback by AudioLemon: Toshio Iwai talking about the visual-musical interface — September 19, 2008 @ 10:18 pm

  4. Kramer auto Pingback[…] Toshio Iwai talking about the visual-musical interface at Ars Electronica 2006. For text on this video go to […]

    Pingback by Toshio Iwai talking about the visual-musical interface - YouTube — January 3, 2012 @ 9:00 pm

  5. Nice read. Just wanted to say so…

    Comment by KC Piano Player — May 23, 2012 @ 1:36 am

RSS feed for comments on this post. TrackBack URL

Leave a comment

Powered by WordPress