Forms and Transgressions Regarding ‘Music Visualization’

I.

In 1993, Patti Lather sent out a call for an expanded definition of what constitutes ‘research’. In ‘Fertile Obsession: Validity After Poststructuralism’, she outlines her ‘post-epistemic’ intention as one which seeks to upend discourses of validity, particularly those in relation to spaces of research and ‘masks of methodology’,1 in favor of a new method which ‘recognizes… the “temporary contract” of any consensus. Its goal is something not entirely subordinated to a system’s goals, yet not so abruptly destabilizing of a system that it is ignored or repressed’.2 Among the areas to be included in Lather’s expanded notion of research are the arts: an area in which her dictum of ‘research as praxis’ has evident meaning.3 Here, I will explore the possibilities raised by the phenomenon of, and ambiguities within, the art and research praxis of music visualization (Fig. 1).

1. Stephen Malinowski, Große Fuge in B-flat Major, opus 133 by Ludwig van Beethoven, 2017, MIDI computer animation.

As a form of academic and intellectual research that is also deeply artistic, music visualization interrogates the boundaries that generally justify academic research, troubling many of the defining features we accept as necessary for valid research, even as fulfilling them in other ways. The reason music visualization’s criteria for validity are highly ambiguous is its questionably dual-presence in the domains of both musical, and visual composition. But while it seems ideal for its content to advance the epistemologies of both mediums, whether it is able to accomplish this goal is, at the present, unclear. One analogous field of research can be found in data visualization, the concepts and practices of which might help to specify the aims of music visualization. Edward Tufte begins his 1990 monograph, Envisioning Information, by examining the ways in which increasing ‘the number of dimensions that can be represented on plane surfaces and…data density’ can liberate data from the two-dimensional page, an intrinsically insufficient mode to express what he calls our ‘daily…perceptual world of three spatial dimensions’.4 In relation to data visualization, music visualization’s two-dimensional page, or ‘flatland’, can be viewed similarly: the written score. Certain music visualization practitioners view the score as being inadequate for expressing a variety of musical forms, including those outside of the Western canon, and others which rely heavily on improvisation. Music visualization is also, however, a practice with a complex status as art that is the current endpoint of a long and complex history of artistic investigation: one that showcases precisely how amorphous the object of analysis can be when considering two mediums in simultaneity. Where music visualization resonates outside of projects such as data visualization is in this duality: Tufte’s elucidation of these often abstract forms of representing data is pedagogical, beginning from data and concluding with animation, whereas music visualization, in addition to acting as a form of musical pedagogy, is also itself an art practice. The ambiguity in the project of music visualization lends itself to a revised concept of what such a mode of inquiry might look like, and how the notion of validity might be expanded to include such a manifold area of thought and work. An area which, akin to Lather’s own aims, reexamines research itself, making inquiry into whether its goals are aesthetic or practical.

The question arises whether it is even possible to talk about music without mention of the visual. Since the inception of written music – visual directions for sonic production – the two have always existed in an ongoing and mutually imbricated mode of encounter.5 This intersection indicates how certain art forms are, indeed, not closed, and do not offer themselves as prescriptions of methodology so much as, via Lather and Giles Deleuze, “‘curves of visibility and enunciation’”.6 These are, in Lather’s words, ‘provisional space[s]’ for new forms of science to occur, in avoidance of such neat, closed, medium-specific distinctions.7 The question of import is not whether the two have a relationship to each other, but rather what the most productive avenue is for the mutual negotiation of the two to take. Music visualizations are only the most recent example of this enquiry, coming after nearly a century of audiovisual experimentation in the hybrid medium of visual music.

II.

Pairing sounds with moving images has been a polarizing practice since the commercial inception of the talkie in the late 1920s, resisted by principal Soviet filmmakers – Sergei Eisenstein, Vsevolod Pudovkin, and Grigori Alexandrov – as being antithetical to the ‘perfection of the cinematographic art’.8 For Eisenstein and his colleagues, the only artistically viable route for the synchronized sound-film to take was in a contrapuntal use of the two mediums, utilizing sound for its ‘pronounced non-coincidence with the visual images’.9 Stan Brakhage held a similar ideal half a century later, which he expressed in a letter to Ronna Page in 1978 as a deep dissatisfaction with sound becoming ‘mere referendum to image in movies’, becoming a supplement rather than a companion art to be interrogated.10 In making the majority of his films silent, Brakhage took up a seemingly paradoxical challenge: to create ‘visual music’.11

Precisely what this particular cross-medium is has been defined, and redefined, by artists and theorists alike since its conception in the early twentieth century. Brian Evans, a prominent theorist in the field of visual music, published his ‘Foundations of a Visual Music’ in 2005, defining this medium as ‘time-based visual imagery that establishes a temporal architecture in a way similar to absolute music. It is typically non-narrative and non-representational (although it need not be either)’.12 In referring to visual music’s ‘similarity’ to absolute music, Evans is citing Igor Stravinsky’s definition of absolute music, being a music which ‘means nothing outside itself’, in other words Evans envisions visual music as being non-representational, at least in its impetus.13 In Evans’ article, he draws upon the works of an eclectic group of early- to mid-twentieth century artists, including Oskar Fischinger (Fig. 2), Jordan Belson (Fig. 3), and Stan Brakhage (Fig. 4).14

2. Still from Oskar Fischinger, An Optical Poem, 1938, 35mm film.

3. Still from Jordan Belson, World (excerpt), 1970, 16mm film.

Belson and Fischinger’s works exemplify a similar ethic, being a visual music which uses both mediums to complement each other – its visual component amplifying its aural information, and vice versa. This contrasts with Brakhage, who uses the visual not as a means of supplementation but rather to create a visual language that operates autonomously, as is the case in music. In other words, Brakhage was a composer of images rather than sounds.

The visual music theorist William Moritz has set out a historical trajectory for visual music, claiming that the medium’s evolution ‘parallels that of cinema itself’ in its lateral development alongside the burgeoning studio system in Los Angeles through the 1920s and 1930s.15 Moritz attributes this correlation to two phenomena: an increased availability of the necessary ‘industry supplies’ for the creation of visual music made possible via large studios such as Metro Goldwyn Mayer, and the influx of European immigration into the United States during World War II, particularly those of German descent who bore a certain relation to a visual and musical aesthetic (Fischinger being his primary exemplar).16 Visual music has also been said to belong to a varied tradition of artists who have sought to bridge the fabrics of the optical and the aural. Moritz traces a fairly abstract lineage for the medium, calling upon the abstract musical experimentations of Louis Bertrand Castel’s Ocular Harpsichord in 1730, and later instruments which produced colored light, e.g. Zoetropes, Phenakisticopes and Praxinoscopes, as well as the painterly in Paul Klee’s re-imaginings of Johann Sebastian Bach’s music. And, certain musicians who approached this question from the other side, such as in Alexander Scriabin’s ‘synaesthetic symphony’, Prometheus: A Poem of Fire, wherein he indicated for ‘precise colors in the score’ to be projected across the performance hall with a Color Organ.17 What these seemingly disparate examples have in common is a shared intention in using one medium to experiment with the other. Visual music, specifically, enacts a form of research into the aesthetic question, now via technological means.

4. Still from Stan Brakhage, Black Ice, 1994, oil paint on 16mm film.

Whether the medium, as an artistic enterprise, stems primarily from the visual or the musical seems unclear. For artists such as Brakhage, the output is often purely visual, but his intention is musical in nature, as shown in Black Ice (Fig. 4), an example of Brakhage’s attempt to ‘compose’ images without the aid of sound. Whereas for Fischinger, the visual and the musical are both being used to create something which combines the two into what is an ambiguous result. In Fischinger’s Optical Poem (Fig. 2), visual and musical qualities are completely intertwined – when the music changes in character, or reaches a distinct point of consonance or dissonance, the visual follows (e.g. 3:47–3:54). This also means that Fischinger’s visual material is largely determined by the artist in advance, and always in accordance to the musical text on which the film is based. Brakhage’s intentions seem to be such a radical departure from the earlier form that for them to be viewed as products of the same practice seems reductive. If Brakhage’s silent films are to be considered alongside the works of such overtly aural filmmakers as Fischinger, and Jordan Belson, all under the aegis of visual music, the medium’s criteria for validity seem ambiguous, if not tenuous.

What separates music visualization and visual music is elusive. They do not appear to be identical, nor entirely separate, or in a coherent artistic dialogue with each other. Music visualization stands in an indefinite, highly dynamic, critical situation. As a form of research that attempts to answer the questions posed by this history through a new means, its contents can be seen by some as art (e.g. Brian Evans), some as novelty, and others as pedagogy. I propose that we think of three distinct constituent types. The first is as an analogous form of visual music. These are pieces which are designed to visually represent sound/music in a classically cinematic mode. They seek to make music visual. The second is in more prevalent, commercial, automated forms of music visualization such as in the iTunes visualizer, and software such as Adobe After Effects, both of which have the capacity to create, algorithmically, abstract visuals in adherence to the wav-forms of a particular sound. The third occurs in the musical domain, being those forms which seek to visually transcend the limitations of Western notation.

The first type of music visualization, here referred to as visual music, is examined from the perspective of visual art by Brian Evans. It is in this territory that Brian Evans seeks to create a ‘groundwork for a practical theory of visual music composition’.18 In an attempt to codify visual music’s parts, Evans’ ‘Foundations of a Visual Music’ begins by adopting two foundational musical traits in assignment to image-making: consonance and dissonance. Evans’ notion of ‘visual consonance’, also referred to as ‘visual rightness’, is formulated through a primarily compositional approach, along with constituent bases in color theory, and montage.19 To Evans, apt composition is a binary metric:

If rightness is codified and understood, wrongness is easily defined by not being right. The might call this wrongness visual dissonance, that is, visually active moments of tension in a temporal design. Progressing from visually wrong to visually right moves us from dissonance to consonance or tension to release, just as in music.20

To substantiate his criteria, Evans explores the qualities of certain artworks according to dimensions such as proportion, symmetry, and ratio, specifically the phi/golden ratio. Color and camera movement are also posited as methods of either creating and/or resolving tension. And, while Evans concedes that none of these are ‘panacea’ for visual rightness, he insists that they are still ‘undoubtedly useful and often used’.21 In this, I find echoes of Lather’s call for standards of validity that stem from assessments that are unable to be measured through foundational methods – standards of validity that stem from a desire for what is ‘at play in our practices of constructing a science “after truth”’.22 Evans alternates between filmic and musical vocabularies with fluidity, using terms such as ‘phrasing’ and ‘cadence’ in address to filmic phenomena, and consequently his criteria become open-ended, though perhaps overly dependent upon a shared, assumed comprehension of ‘tension and resolution’.23 This dichotomy underpins each of his criteria – ‘tension/release’ – to varying degrees of specificity.24 It would seem that, according to Evans, a work’s validity depends more on how well it can be judged from a musical perspective, or at minimum one that is based more in a musical vocabulary than one which is distinctly visual.

The divisive lines between the latter two categories of music visualization appear at a different point in the process of production, being: intention. While commercial endeavors attempt to amuse, to entice a vague interest in the connection between music and imaging from a casual observer, other more far-reaching projects begin to interrogate the nature of that connection. The ultimate goal of these endeavors is the creation of an alternative to the traditional Western style of musical notation. One of the most substantive endeavors in the realm of this type of music visualization began in 2011 through the combined efforts of Ingrid Monson and Alexander Rehding, and one of the medium’s most prolific pioneers, Stephen Malinowski.

III.

Malinowski is a formative figure in the domain of musical visualization. Malinowski’s animation of Igor Stravinsky’s The Rite of Spring (Fig. 1) is one of over 600 videos that Malinowski produced for public use, all of which follow a similar audiovisual template. Malinowski’s work received what is, thus far, its most significant commercial following by being shown during Björk’s live performances since the two started collaborating on her Biophilia music-application in 2011.25 However, his current formal practice is a product of decades of experimentation. His first attempts, not surprisingly, resemble those of Klee and Fischinger, the latter directly cited by Malinowski as an inspiration for his efforts.26 Malinowski’s work facilitates an optical experience in adherence to the aural, linking visual (often geometric) data to individual notes and larger musical patterns in real-time, thus creating a language which is able to communicate visually and sonically simultaneously.

Monson and Rehding saw Malinowski’s work in music and animation as a means of surpassing the capacities of Western musical notation. The trio set out to explore ‘the sensory experience of listening, [viewing] it as a kind of knowledge that’s not necessarily text-based’.27 The traditional Western system, to Monson and Rehding, seems not only to be biased towards expressing Western music alone, but also antithetical to the oral traditions of musical communication, and other musical phenomena that are more difficult, if not impossible, to codify into the Western system (e.g. improvisation, other musical styles and techniques composed in performance). His work with Monson and Rehding developed into a more substantive venture – similar in intention – helmed by Malinowski, Etienne Abelin, and Lushen Wu, titled Music:Eyes. In its current form, Music:Eyes is an interactive computer program which enables music students to create animations similar to those of Malinowski’s – in its impetus, ‘A new art form. And a powerful music education tool’.28 Music:Eyes is still in a relatively early stage, but even in accounting for Malinowski’s prior decades of experience, music visualization’s competency to address the finite scope of Western musical notation seems, at least at the present, to be supplemental rather than restorative.

Malinowski’s work is, by design, a form of research. Funded and supported by universities and musical ensembles since the early 1980s, Malinowski continues to focus on concretizing images and sounds in innovative ways for visual artists, musicians, and software engineers alike. His role in the history heretofore traced is most explicitly that of researcher, as he holds the status of composer, musician, educator, engineer, and inventor. Malinowski’s hybrid status as both scientist and inventor makes what he does most obviously research, based in the domains of engineering and technology. His work has always been experimental in character, and ongoing, having not yet arrived at a singular solution to these dilemmas of intermingling the visual and the musical.29 But his aims have remained the same, as a culmination of this history of music visualization: to identify the nature of the connection between image and music, and to manipulate it to discover more about the intricacies of that connection. Malinowski’s experiments attempt to answer, through technological means, the ongoing questions of his discipline. In this, his work resounds the call to research in both a classical mode and in Lather’s mode of an expanded mode of validity that would include the arts.

The ways in which Malinowski and his collaborators intercross the musical and the visual in such elemental ways forces a confrontation between music visualization as a discrete art practice and as a vehicle for pedagogy. The new breed of music visualization proposed by Malinowski is self-consciously undertaken within the experiment, recalling Lather’s call for an expanded model of research: one that involves enhanced understanding of the subject through practice. However, if Malinowski’s work did develop itself into a more standardized, accessible practice, how exactly it would circumvent the process of becoming another purely semiotic format – teachable, by some – is uncertain. While technological innovations in the audiovisual sphere have been so intertwined with new forms of experimentation, beginning with the Hollywood studios of the 1920s, spanning to the evolving software of Malinowski, there is also the possibility of this form of research to depend less on the technological and more on the anthropomorphic, for example, positing Busby Berkeley’s ‘semi-abstract’ dance numbers as a less orthodox form of visual music.30 In films such as Footlight Parade (1933), Berkeley would arrange dancers into complex patterns, shot from a variety of more distant angles which created similar types of optical illusions and synchronous audiovisual interplay to the films of Oskar Fischinger and Jordan Belson. Though outside of the realm of dance, the human has not played an integral role in any of the visual content produced by artists in this cross-discipline. Is this because integrating the human into the production of works of visual music instigates a third, unavoidable question: can the work advance our understanding of musical, visual, and humanistic ‘notation’? Such an enquiry begs questions of relevance to the province of ethnography, wherein works coexist as both artworks and forms of empirical anthropological research. Such a medium can perhaps help to reconceptualize the dual-praxis of visual music, as, according to anthropologist-filmmaker David MacDougall, visual anthropology itself continues to struggle with its dual nature, continually in the process of liberating itself ‘from expectations imposed upon it by anthropologists whose intellectual goals had been formed in another medium, the medium of words’.31

Whether the heretofore works ‘succeed’ in either field of research is representative of the questions which continue to face the domain itself: can music visualizations exist as a medium which answers questions of both visual and musical pertinence? Such binary designations also run counter to the possibility of a ‘validity of transgression’, which is particularly pertinent to the audiovisual inquiry as it is itself a product of a persistent defiance of adhering to a sole medium’s supposed functionality.32 These new lines of research clearly evoke old questions raised by artists and theorists such as Brakhage and Eisenstein, as the nature of the relationship between sound and image continues to be explored through such new paradigms. And, due to the work of artists such as Malinowski, I propose a new form of research is recognized– one based in both engineering and artistic practice. The audiovisual hybrid is perhaps an ideal intersection through which Lather’s proposed ‘counter discourse/ practices of legitimation’ can be developed – one medium constantly ‘unjamming’ the other, and vice versa.33

1. Patti Lather, ‘Fertile Obsession: Validity after Poststructuralism,’ The Sociological Quarterly 34:4 (1993): 675.
2. Idem, 679.
3. Patti Lather, ‘Research as Praxis,’ Harvard Educational Review 56:3 (1986): 257–77.
4. Edward Tufte, Envisioning Information (Cheshire: Graphics Press, 1990), 12.
5. Christopher Gibbs and Richard Taruskin cite a ‘Babylonian cuneiform tablet’ from ca. 1200 B.C.E. as being the first known example of transcribed music; see: Christopher Gibbs, Richard Taruskin, The Oxford History of Western Music (Oxford: Oxford University Press, 2013), 2.
6. Giles Deleuze, ‘What is a Dispotif?,’ in Michel Foucault: Philosopher, trans. Timothy Armstrong (New York: Routledge, 1991), quoted in Lather, ‘Fertile Obsession,’ 687.
7. Lather, ‘Fertile Obsession,’ 673.
8. Sergei Eisenstein, Vsevolod Pudovkin, and Grigori Alexandrov, ‘The Sound Film: A Statement from U.S.S.R.,’ Close-Up 3:4 (1928): 10.
9. Eisenstein, Pudovkin, and Alexandrov, ‘The Sound Film,’ 12 (italics in original text).
10. Stan Brakhage, ‘Letter to Ronna Page (On Music),’ The Avant-Garde Film: A Reader of Theory and Criticism, no. 3, ed., P. Adams Sitney (New York: New York University Press, 1978), 134.
11. Robert A. Haller, First Light (New York: Anthology Film Archives, 1998), 81.
12. Brian Evans, ‘Foundations of a Visual Music,’ Computer Music Journal 29:4 (2005): 11.
13. Igor Stravinsky, The Poetics of Music in the Form of Six Lessons (London: Harvard University Press, 1947), 76.
14. Evans, ‘Foundations of a Visual Music,’ 11–12.
15. William Moritz, ‘Visual Music and Film-As-An-Art Before 1950,’ in On The Edge of America: California Modernist Art, 1900–1950 (Berkeley: University of California Press, 1996), 224.
16. Moritz, ‘Visual Music and Film-As-An-Art Before 1950,’ 224.
17. Moritz, ‘The Dream of Color Music, And Machines That Made it Possible,’ Animation World Magazine 2:1 (1997), par. 5, accessed May 22, 2017, at https://www.awn.com/mag/issue2.1/articles/moritz2.1.html.
18. Evans, ‘Foundations of a Visual Music,’ 11.
19. Idem, 13.
20. Ibidem.
21. Idem, 14.
22. Lather, ‘Fertile Obsession,’ 687.
23. Evans, ‘Foundations of a Visual Music,’ 16.
24. Idem, 11.
25. Nicola Dibben, ‘Visualizing the App Album with Björk’s Biophilia,’ in The Oxford Handbook of Sound and Image in Digital Media, eds. Carol Vernallis, Amy Herzog, and John Richardson (Oxford: University of Oxford Press, 2013), 691.
26. Ibidem.
27. ‘Animating Musical Analysis, part 2,’ YouTube video, 2:51, posted by DARTH Crimson, 29 August, 2014, accessed  May 19, 2016, at https://www.youtube.com/watch?v=MO7QW_L8kdQ.
28. ‘About Music:Eyes,’ Music:Eyes, accessed January 31, 2017, at http://www.musiceyes.org/#about-musiceyes.
29. They began as literal experiments of a pseudo-scientific nature, having begun his first music animations in the early 1970s in reaction to a drug-induced hallucination he had while listening to Bach’s Sonatas and Partitas for Unaccompanied Violin, and see: ‘1996 Music Animation Machine (documentary),’ YouTube video, 0:26, posted by musanim (Stephen Malinowski), 13 July, 2010, accessed May 20, 2016, at https://www.youtube.com/watch?v=SQfzNJGnqnw.
30. Moritz, ‘Visual Music and Film-As-An-Art Before 1950,’ 224.
31. David MacDougall, The Corporeal Image: Film, Ethnography, and The Senses (Princeton: Princeton University Press, 2006), 267.
32. Lather, ‘Fertile Obsession,’ 676.
33. Tony Bennett, Outside Literature (London: Routledge Press, 1990), 277, quoted in Lather, ‘Fertile Obsession,’ 676.