What is a Projection Mapping

Projection mapping¹ widely defined as a projection technology allows converting usually irregular shaped objects —either big or small, solid or liquid— into a projection surface, achieved by using specific software that creates an exact correspondence between the surface and the images in the projection. This technique makes possible to add additional dimensions on to the projected surface and to create optical illusions and a sense of motion to previously static objects. Rossella Catanese (2013) explains: “Combined with audio equipment, you can ‘tell a story’ or let the audience live [...] a synaesthetic experience [today it would rather use the definition of an ‘emotional’ experience]” (p. 165).

Literature about projection mapping as a multimedia art form, as already mentioned, is scarce. Some writings only address the visual effects and analyze projection mapping only from the subjectivity of the graphic designer or the animator, while others discuss the technology with which to achieve the projection mapping itself. In terms of documenting the live experience, Catanese (2013) writes: “Videomapping is one of the most ephemeral contemporary art forms; for this reason we have additional problems for documentation and archiving. Like other events with a strong performative character, theatrical or video installations, the memory of the event is delegated to video recordings of the occurred screening, leading to a gap between actual public use of the event and its recording” (p. 168).

Brief History

Projection onto a non-flat surface finds its known origin in the 1969 opening of the Haunted Mansion ride in Disneyland where a film was projected on five individual busts singing Grim Grinning Ghosts and on the disembodied head of Madame Leota. In 1980, media artist Michael Naimark, a precursor of today’s projection mapping, presented his immersive film installation Displacements. He projected a film in a white painted living room to create the effect that a couple of performers were interacting with the objects of the room (https://www.youtube.com/watch?v=bMDr_CFFgWE). In 1998, researchers from the University of North Carolina Chapel Hill, who worked on the project Office of the Future, studied academically for the first projection mapping.

From the mid-late 2000s, projection mapping started to become more popular as it thrived in the consciousness of artists and then it found a commercial route in advertising. It has been gaining popularity ever since and nowadays projection mappings are created for different purposes. It can be experienced as part of a show, in a concert, in a corporate event, used as a branding strategy or with the intention of a narrative audio-vision spectacle itself. The latter finds buildings’ façades as a common surface to project, and it has become a standard practice to use them to show a 3D architectural mapping for an audience.

Framework to Analyze Music and Sound Design in Projection Mappings and Why It Is Important to Analyze It

Writings about the role of the sound design in projection mapping are arguably non-existent as far as I could confirm through my academic research, and references to music and sound design in projection mapping go only as far as to say that “[w]ith sound design, the aim is to supplement audio elements for supporting visual show” (Ekim, 2011, p. 11), or “[c]ombined with audio equipment, you can tell a story” (Catanese, 2013, p. 165). However, narrative projection mapping as an audio-visual art form contains and shares many particularities seen in film, TV and video games. Above all, the audio-visual contract (Chion 1994) or the marriage between sound and image, where the sound aids the film or the projection narrative at many different levels by providing continuity, mood, sense of locale, time of day and very importantly, the emotional nuances and states of each moment.

There is a substantial amount of research in sound in the context of film, and authors such as Chion (1994); Altman (1992); Gorbman (1987); Cooke (2008); Kalinak (1992); Karlin and Wright (2004); Brown (1994); Cohen (2001); Whittington (2007) or Prendergast (1992), amongst others, investigate the impact sound-on-film makes on our intellect and emotions in terms of perception and meaning. Authors from a cognitivist perspective, such as Cohen (2001), have provided theoretical and cognitive frameworks to describe how we process and associate music (non-diegetic) with an image in our brain. Cohen (2001) explains how the viewer extracts the emotional information from the non-diegetic source to match the visual stimulus through what is referred as to an associationist congruence. The Congruence-Associationist framework “represents the two primary ways in which the brain operates” (p. 258) and explains how the auditory and the visual stimuli interact together allowing the audience to create a meaningful story from the information collected from both sensory channels. In applying the concept of cross-modal congruence, studies have proved that music can sometimes determine the visual focus of attention and that when the association between the optic medium and it is congruent, music can contribute to fixation and consciousness, as well as leading to a stronger long-term memory of the experience. Authors, such as Tagg (2004), Sonnenschein (2011) or Fluekiger (2009), amongst others, who provide writings on the meaning of music and sound in relation to the visual stimulus, study sound and music in the film context in terms of semiotics.

As it happens in video games, a large number of projection mappings hope to provide an immersive experience for the audience. Both sound and image participate in the achievement of such a venture. Together the visuals of a metamorphic building’s façade ² and the emotions that the sounds generate aim to drag the spectator into the semi-virtual environment in which the experience takes part and facilitate the audience into a suspension of disbelief to connect with the narrative. Stefania Serafin and Giovanni Serafin (2004) analyze ambient sounds (i. e., wind, street noise) and sound events (i. e., doors, footsteps) to understand the presence of sound design in virtual reality and, borrowing sound design concepts from the entertainment industry, they write: “In movies, sound effects exaggerate reality to create an immersive experience. Virtual environments are created to immerse users in order that they may experience a so-called suspension of disbelief”. They conclude that “[i]t is likely that also in virtual reality, enhanced sound events will increase the sense of immersion” (Serafin & Serafin, 2004). Karen Collins, who has written extensively about video games audio, has provided evidence form embodied cognition theory to explain how the player experi­ences empathy and envelopment (or presence) to facilitate immersion. In her book Playing with Sound, Collins (2013) articulates that “[o]ne of the key functions and effects of sound in games is to immerse us in the virtual world through the sense of sonic envelopment” (p. 54), defining it “as the sensation of being surrounded by sound or the feeling of being inside a physical space” (p. 54). Although there should be more empirical research in this area, I would argue, from my own experience, that sound effects and music in projection mapping have the exact same essential function as that of video games in terms of envelopment. Nevertheless, as the viewer of a projection mapping happens to be placed in known public space, the sense of presence, to an extent, comes from attending the projection itself (being physically there) and sound contributes to the perceptual alteration of the space.

However, projection mapping enjoys some qualities as an audio-visual art form that vary from that of film or video games. These dissimilarities, in my opinion, are what make analyzing its sound so necessary. The experience of projection mapping differs from that of traditional film in that (A) “[u]nlike simple projection on a screen, the video-mapper interacts with the display surface, offering it a new reading” (Catanese, 2013, p. 165), and (B) the audience is placed within a known space that is transformed to a semi-virtual environment. Furthermore, in my view and as I will try to explain, sound effects play a crucial role in the emotional and cognitive perception of the interac­tion with the display surface, and they provide an emotional depth that enhances the appreciation of movement of the metamorphic façade, thus, contributing to the immersive experience and the ‘realism’ of the visual phenomenon.

Methodology

In this paper, I discuss the way sound design in projection mapping provides the emotional level that guides the viewer and facilitates an immersive experience. I concentrate mainly on the analysis of sound effects and explain my approach and intention on how to affect the images with emotion. Although this paper is theoretical, I provide evidence from both theoretical and empirical research in the areas of cognitive psychology, embodied cognition theory, and musicology in the context of film, animation, and videogames.

Synopsis of Projection Mappings

Bilbao:

The projection mapping Bilbao illustrates the history of the city of Bilbao, from its origins with its whale hunting people to the contemporary Bilbao with its Guggenheim museum Bilbao and its modern and vibrant city life.

Tricenari Mas Guinardó:

The projection mapping Tricenari Mas Guinardó is a history based illustration of the siege of the city of Barcelona during the end of the War of the Spanish Succession in 1714.

Voluntariat:

As Barcelona won the volunteering city 2014, the projection mapping Voluntariat illustrates the different humanitarian areas in which the city has excel.

Emotion in Sound of Projection Mapping

In contrast to the traditional screen, which is “a pretty cold proposition” without music, as Annabel J. Cohen (2001) quotes from Aaron Copland (1941), I would argue that projection on a building’s façade is a very ‘warm’ proposition because the projection surface convincingly appears to be the auditory object itself —the sound source— and the canvas that contains an interaction of “sound object(s)” (Schaeffer, 1966).

Sound Effects: Tangible Perception of Movement

When the projection creates the visual illusion of movement in one or in several of the structures of the façade, an accurately synchronized sound accompanies the image and imbues it with meaning in terms of its source and its qualities. Sound can communicate the material, the texture or the speed and mass of an object, or it can suggest friction between objects. At times it is the sound that is first perceived —as the dimension of the façade can be wider than our visual field— and thus it guides the viewers to the focus of the action. Either way, this investigative listening is what Michel Chion (1994) calls écoute causale (or causal listening). Chion (1994) classifies causal listening as the most common way of listening and, as he explains, it “consist[s] of listening to a sound in order to gather information about its cause (or source). When the cause is visible, sound can provide supplementary information about it; for example, the sound produced by an enclosed container when you tap it indicates how full it is” (pp. 26-27).

Expanding Chion’s three listening modes, musicologist David Huron (2002) produced a classification that added the emotional dimension evoked by the auditory stimulus and proposed a six-component theory to classify the emotional experiences. Huron calls Denotative listening to that that associates the sound to its source or action, coinciding with Chion’s causal listening. In this paper, I will use Chion’s term when discussing the association between sound and source. In the context of projection mapping, as it happens in cinema, audiences’ causal listening, I would maintain, enhances the perception of ‘reality’ even if “causal listening is constantly manipulated by the audiovisual contract itself, especially through the phenomenon of synchresis” (Chion, 1994, p. 28).

In other words, sound effects in projection mappings are responsible for the palpable perception of the façade’s movement, as the sound is related to motion and can influence the quality of visual objects. Perhaps it is our need to understand what surrounds us and the fact that “[w]e can never not listen casually: it is the dominant mode of hearing” (Collins, 2013, p. 23) that eases this synaesthetically tangible perception of the metamorphic façade, which requires precise synchronization between the auditory and visual stimuli. It is this very synchronization that endows the sound with the power to deliver great emotional weight by means of added value (Chion, 1994), and helps to provide ‘realism’ thanks to synchresis. Synchresis ³ is extremely important in projection mapping because, as it happens in animation, the sound in it “is not tied to production audio (sound acquired on a live set)” (Beauchamp, 2005, p. XX) and so it needs to be created. Synchresis is what allows the audience to believe that the façade has become the auditory object itself.

The following sequence from the projection mapping Tricentenari Mas Guinardó exemplifies the concept of synchresis (from 01:20 to 02:03) https://youtu.be/S4Nmhb3NL-I?t=1m17s. Synchresis provides the sequence with a sense of ‘realism’ and contributes towards a perception or illusion of ‘truth’. Audience’s causal listening allows the association of the sound they hear to the action that created it: the sound of stone blocks and bricks that magically build a wall.

The ‘realism’ in terms of the aesthetics of sound, in narrative projection mapping, I would argue, is linked to that of the film in terms of “cinematic expectation” (Whittington, 2007, p. 149). All the more when most of the projection mapping seem to belong to a genre of fantasy, sci-fi or even horror, where “codes of ‘realism’ are in fact created and passed along from production to production” (Whittington, 2007, p. 149). Certainly, the construction of the wall of the medieval city of Barcelona that we see on the sequence has a fantasy genre attached to it. The audio’s mix in the sequence assists the audience to simultaneously hear the contrast between the small bricks’ clean close mic recording sounds and the sounds of the movement of the big stone blocks, which creates a sense of artificiality —in real life, the sound of a big stone block would mask the sound of a small brick— that has been deliberately created to control and guide the audience’s attention, while contributing at the same time to the contemporary cinematic expectation.

Technology has contributed to the development of the sound effects and its aesthetics in current forms of media. Barbara Flueckiger (2009) refers to it as the “autonomous sound object”. As she explains, sound designers of the New Hollywood aimed at recreating a sound in precise detail by analyzing its complex whole. They manipulate the autonomous sound object, which is different from any other of the same source. Flueckiger (2009) elucidates how “the new creative freedom, combined with new flexible and highly expensive technologies, have made possible the intensive detailed work on sound which is characteristic of the autonomous sound object of contemporary film” (p. 156). These influence the perception of the visual phenomena through what Chion (1994) refers to as the materializing sound indices (p. 114) or MSI. The very sound-detailed autonomous sound objects are capable of engaging with the diegetic reality and fulfilling the contemporary cinematic expectations, and its “extreme artificiality […] is in itself so coherent as to be immediately acceptable as natural, as is confirmed by audience reactions” (Flueckiger, 2009, p. 156). I hope to provide with the 33 seconds test of the same sequence shown above from the projection mapping Tricentenari Mas Guinardó https://www.youtube.com/watch?v=MN03-hM0psk a good example of the “artificiality” of the sound object (sound of friction between rocks, sound of small bricks, sound of big stones and sound of revolving stone blocks) as well as its relation with the sonic point of view. The artificiality is achieved by the manipulation of the audio-mix and the creation of the detailed sound object with its materialized sound indices on the rendered sound, which aims at affecting the perception of the visual phenomenon and to elicit emotions via a “reflexive response” caused by the reflexive listening, a pre-conscious listening mode, which focus “is on automatic and fast audio evoked responses” (Tuuri, et al., 2007, p. 15).

Sound Effects: Emotional Qualities and Metaphors

The sound design created for the test sequence emphasizes on the mass of the different bricks; thus, focusing more on the effect (the impact of the stone blocks and the bricks on the wall) and less on the cause (stone blocks and bricks moving). Tuuri et al. (2007) proposed an extended taxonomy from David Huron’s (2002) listening modes and stated that the pre-conscious “reflexive responses represent clearly an important way by which meanings and emotions can be evoked by sound” (Tuuri, et al., 2007, p. 15). My idea behind the creation of the sound design for the city walls was not only to enhance the fantastical reality of a wall moving by itself and the manipulation of the stones’ sound to the extent of being fetishized (a concept I have borrowed from Whittington), but also to create an anthropomorphic illusion, a metaphorical expression through sound, and to give an organic life to an inanimate wall; thus, imbuing it with an emotional quality. This can be heard from 00:13 when the movement of the blocks purposely adopts the quality of a roar.

Synchresis yields the possibility to re-associate the sound object with the sound source, without it, sound effects would not be able to provide the depth in meaning and emotion that can be associated with different sound objects. Re-association, as explained by Walter Murch in Chion’s Audio-vision: Sound on screen (1994), “is the fundamental stone upon which the rest of the edifice of film sound is built, and without which it would collapse” (p. xix). The projection mapping Bilbao contains a sequence which perfectly illustrates the process of re-association. The sequence running from 02:50 to 03:05 shows a thick forest that grows rapidly on a green hill (https://youtu.be/uws5PDic49I?t=2m49s). We hear a low rumble of young trees protruding from the ground followed by the cracking noise of wood as the trees grow and finally the sound of leaves from the trees. I hoped to create a sound montage that conveyed effectively the ‘birth’ of the forest we see on the building façade. However, it was employing sounds of “destruction” that I achieved the “construction” of the forest. I used and manipulated recordings of the landslide, breaking wood (for the growing trees) and the crackling of fire (as the sound of leaves). I also employed this sound re-association, although further manipulated, in the projection mapping Voluntariat (01:27 to 01:46, https://youtu.be/I6aFrqu1YtE?t=1m26s).

Re-association of sound and source is a fundamental practice amongst sound designers and a usual technique designated to evoke the sound designer’s desired emotions, which creates new meanings and generates “new embodiedness” (Collins, 2013, p. 21). Research in cognitive embodiment through a multisensory approach to describe the role of sound in the context of audio-visual experience has been gradually gaining presence. Our neurological condition of synaesthesia allows a bodily association with the audio-visual. As Collins (2013) elucidates, “[t]ogether the phenomenological and semiotic approaches seem to indicate that sound’s connotative abilities are due to the synesthetic, sensory integration that takes place in the mind” (p. 24). It is perhaps easier to think about a radio play to understand that when we listen to it, we also feel and see the drama unfolding before our ears. It is the intertwining of the sensory inputs that allows us to sense texture, temperature or presence through sound. “In the context of cinematic media, an embodied cognition approach suggests the primary function of sound design is to elicit affective imagery which, in turn, shapes cognition and consciousness” (Ward, 2015, p. 155). The synaesthetic fusion between the auditory and the visual stimuli “acts as a neuropsychological driver behind the most powerful and basic building block of cinematic sound design: audio-visual metaphor” (Ward, 2015, p. 162).

In the book Audio-Vision, Michel Chion (1994) explores the difference between rendering and reproduction and affirms that “the film spectator recognizes sounds to be truthful, effective, and fitting not so much if -they reproduce what would be heard in the same situation in reality, but if they render (convey, express) the feelings associated with the situation” (p. 109). The notion of rendering gives way to the metaphoric use of sound, and it offers an approach to understand how the perception of sounds can arouse emotions and alter the mood of a visual stimulus. Kathrin Fahlenbrach (2008) has written extensively about audio-visual emotion metaphors and has examined how objects and spaces “can be conceptualized metaphorically via their very materiality, by sound design” to enhance their qualities and to aid communicating “complex cultural and narrative meanings” (p. 85). Following the theory of conceptual metaphors, Fahlenbrach elucidates the mechanism of metaphorical mapping to synaesthetically fuse the auditory and visual stimuli of a film to create an emotional gestalt of the object (figures, spaces or emotions). In her analysis of audio-visual metaphors, Kathrin Fahlenbrach (2008) argues that “the synchresis of pictures and sound is most effective when embodied image schemata are used by sound design that guide, on an unconscious level, our perception of film” (p. 85). Using again a sequence from Tricentenari Mas Guinardó, the audience can see how the façade turns into a wooden ship (05:41 https://youtu.be/S4Nmhb3NL-I?t=5m40s). At 05:57, once the ship has been built, its wooden windows slide open and cannons appear from inside the ship. However, the sound of the wooden windows delivers a metallic grind. On the one hand, the metallic sound carries information about the physical qualities, such as the mass and texture of the object which is not apparent in the visuals. On the other hand, the sound of scratching metal, almost like a scream, can be associated with the horror genre through intertextuality. It also creates an audio-visual metaphor that aids the narrative and endows the image with connotations and meaning. The sound of the windows contain an added value, “an expressive and informative value with which a sound enriches a given image” (Chion, 1994, p. 5), that helps the audience perceive both the materiality of the windows —which in itself contains a deeply embodied gestalt— and, in terms of the narrative, its semiotic meaning that becomes a sign that heralds the imminence of a bad forthcoming. Projection mappings are mainly non-vococentrics, and employing the metaphoric use of sound, I would argue, supports and enhances the audio-visual both at the emotional and narrative level.

Sonic Continuum

On to another level, the unification of the music and sound effects heightens the anticipation that the sound design provides in the above example of the wooden ship. The music cue, during the war-ship sequence, is built upon metallic sounding synths and a religious choir. The synths provide ambient and, although their ambiguous place in the diegesis, are connected with the metallic sound of the sliding doors and contribute as an element in the creation of horror. The other element in the music is a soft religious choir which, in the sequence, announces the death. It provides emotional depth to the narrative and binds the sequences together when the choir accompanies the images again after the war at 08:25 (https://youtu.be/7B5YFMaV-Iw?t=8m24s) showing dead bodies and the aftermath of the bombing.

It is, I would argue, the “nature” of architectural mapping and the “warm” proposition of the façade what allows with no other limit than that given by the director, a unification between music and sound effects to create a “sonic continuum” (Donnelly, 2009, p. 104). Donnelly (2009) explains how technology and the use of electronic sounds allowed a fusion between music and sound effects, a trend used in recent films, and how “this has led to an aesthetic rather than a representational conception of sound in the cinema” (p. 107). This holistic (to use Donnelly’s term) approach to sound design is something that I have increasingly developed, both because of the aesthetic interest and because of the “music-sound-design” that I am required to create for a number of productions. In the projection mapping: Voluntariat, (https://www.youtube.com/watch?v=I6aFrqu1YtE) I have been able (within the limitations given by the client and the director) to purposely use this approach, and to create a sonic continuum of an entwining sound track of music and sound effects. The first example comes at 00:30, where the sounds of synths in the music track also play as the sound effect for the folding façade. At the end of the sequence, a loud drum hit marks the end of the folding process which functions both, as a sound effect representing the bang of the collapsed façade, and also as a musical hit point demarking the end of a sequence and the beginning of a new one. At 00:37 the synths start a riser effect which, again, exist simultaneously as a sound effect and as a music soundtrack. The sound of the riser is used as the sound effect for the zoom out of the planet earth, and it also works musically as a way to glue the end of a sequence with the start of a new one. In the sequence starting at 01:15, the sound of the shakers contribute both in the music score [non-diegetic] and as a sound effect [diegetic] for the seeding. More obviously, from 04:49 the sound effects and the music function as one unit fused together. The sound effects provide a musically rhythmic ambiance of city sounds within the diegesis while a bass line keeps a groove in the non-diegetic orbit, until at 05:04 where music and sound effects conjoin to be part of the diegesis.

Bilbao is a projection mapping that, in a sonic level, depends heavily on the construction of ambiances or geographic spaces. As I have previously mentioned, narrative projection mapping, arguably because its interaction with the surface, gravitates towards the fantasy and science fiction genre narrative, or even horror, hence, the sonic link between those film genres and narrative projection mapping is, in my opinion, reasonably evident. “Science fiction is ultimately about constructed spaces; therefore, sound ambiences are crucial in supporting and commenting on spatial geography” (Whittington, 2007, p. 153); at the same time, “general ‘ambiences’ function to immerse the audience in the film more effectively in sonic terms than might be available as a visual effect” (Donnelly, 2009, p. 118). Just to finish with another example in relation to the unification of music and sound effects as a sonic continuum to create ambiances, and the ambiguity of the diegetic and non-diegetic spheres in which sound can exist, I would like to draw attention to a couple of sequences in Bilbao. From 01:11 to 02:18 (https://youtu.be/uws5PDic49I?t=1m9s) drones from electronic instruments together with the sound of bubbles and running water create the underwater ambiance sequence. The sounds of running water and bubbles are clearly understood in the given context and provide a direct relation with the visual stimulus. The drones illustrate the sense of an uninterrupted vastness and emptiness of the ocean depths. Also from 04:55 to 05:15, the time travel sequence is composed in conjoin of musical synths and sounds of clocks, functioning in a similar way as it does in the underwater sequence. To understand how these music elements work and how they can exist as part of the ambiance, Annabel Cohen uses the Congruance-Associationist Framework for understanding film music communication and the paradoxical role of background music. She explains that music adds consistent affective information with the narrative and inconsistent acoustical aspects with the narrative: “Somehow, the brain attends to this affective meaning, while ignoring or attenuating its acoustical source” (Cohen, 2000, pp. 373-374).

Fortune seems a recurring voucher, Walter Murch also makes use of it when he writes about re-association: “It might have been otherwise —the human mind could have demanded absolute obedience to ‘the truth’— but for a range of practical and aesthetic reasons we are lucky that it didn’t [...]” (Chion, 1994, p. xix). However, it is, by chance or luck, sound evokes emotions and conveys meaning. Sound designers can use these qualities to enhance the experience of 3D projection mapping.

Conclusion and Future Research

My research in the use of sound has shaped my understanding on the decisions that the sound designer can make in order to enhance the audience’s experience of the projection mapping as well as how the sonic narrative aids the story in terms of meaning and emotion. In my experience, the use of the sound in a metaphorical way, as well as the use of sound effects to exaggerate the reality, with the very artificiality of the sound object in projection mapping are necessary in order to make effective and to enrich the visual artifice.

Projection mapping offers an opportunity to create sound effects and to use them in a very creative way in order to enhance the fantastical ‘reality’ that appears on the metamorphic façade. By manipulating the psychology of the sonic perception, sound can permeate the visual stimuli with an emotional depth and facilitate the audience into an immersive experience. Movement or friction between bodies generates sound (at least in our earthy reality), and architectural projection mapping contains both of them as its main way of visual expression. With its “warm” proposition as a screen or display surface, projection mapping provides an almost perfect canvas for the sound objects to lure the audience’s causal listening into a new, even if ephemeral, reality.

Although I theorized about the audience’s experience in projection mapping, further empirical research in this area would shed light on how the audience perceives immersion within a known public space.

Other aspects to be analyzed in relation to sound in projection mapping could look at how the “polluted” environmental outdoor sounds, in which the projection might take place, alter, if at all, the cognitive and emotional experience, as they would inevitably “interact” with the projection mapping. Also, and from a technical point of view, projection mappings, at least in my professional experience with the outdoors projections, are only equipped with stereo sound. Although the majority of sounds happen within the confines of the façade, the possibilities of displacing the sound in a sound surround system are negated, and one could examine how this limits the immersion experience and the perception.

References

1. Altman, R. (1992). Sound space. In R. Altman (Ed.), Sound theory sound practice (pp. 46-64). New York: Routledge.

2. Beauchamp, R. (2005). Designing sound for animation. Burlington: Elsevier.

3. Brown, R. S. (1994). Overtones and undertones: reading film music. Berkeley: University of California Press.

4. Catanese, R. (2013). 3D architectural videomapping. Retrieved from https://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XL-5-W2/165/2013/isprsarchives-XL-5-W2-165-2013.pdf

5. Chion, M. (1994). Audio-vision: sound on screen. Claudia Gorbman (Trans). New York: Columbia University Press.

6. Cohen, A. J. (2000). Film music: perspectives from cognitive psychology. In J. Buhler, C. Flinn & D. Neumeyer (Eds.), Music and cinema (pp. 360-377). Hanover: Wesleyan University Press.

7. Cohen, A. J. (2001). Music as a source of emotion in film. In P. N. Juslin & J. A. Sloboda (Eds.), Music and emotion: theory and research (pp. 249-272). New York: Oxford University Press.

8. Collins, K. (2013). Playing with sound: a theory of interacting with sound and music in video games. Cambridge: The MIT Press.

9. Cooke, M. (2008). A history of film music. New York: Cambridge University Press.

10. Donnelly, K. J. (2009). Saw heard: musical sound design in contemporary cinema. In W. Buckland (Ed.), Film theory and contemporary Hollywood movies (pp. 103-123). New York: Routledge.

11. Ekim, B. (2011). A video projection mapping conceptual design and application: Yekpare. The Turkish Online Journal of Design, Art and Communication, 1(1), 10-19. Retrieved from http://acikerisim.iku.edu.tr:8080/jspui/bitstream/11413/656/1/tojdac_v01i102.pdf

12. Fahlenbrach, K. (2008). Emotions in sound: audiovisual metaphors in the sound design of narrative films. Projections, 2(2), 85-103. Doi: https://doi.org/10.3167/proj.2008.020206

13. Flueckiger, B. (2009). Sound effects. Strategies for sound effects in film. In G. Harper, R. Doughty & J. Eisentraut (Eds.), Sound and music in film and visual media: a critical overview (pp. 151-179). London: Continuum International Publishing Group Ltd.

14. Gorbman, C. (1987). Unheard melodies: narrative film music. Bloomington: Indiana University Press.

15. Huron, D. (2002). A six-component theory of auditory-evoked emotion. In Proceedings of the 7th International Conference on Music Perception and Cognition (pp. 673-676). Sydney, Australia.

16. Kalinak, K. (1992). Settling the score: music and the classical Hollywood film. Madison: The University of Wisconsin Press.

17. Karlin, F., & Wright, R. (2004). On the track: a guide to contemporary film scoring (2^nd ed.). New York: Routledge.

18. Prendergast, R. M. (1992). Film music: a neglected art; a critical study of music in films (2^nd ed.). New York: W. W. Norton.

19. Schaeffer, P. (1966). Traité des objets musicaux: essai interdisciplines. Paris: Éditions du Seuil.

20. Serafin, S., & Serafin, G. (2004). Sound design to enhance presence in photorealistic virtual reality. Retrieved from https://smartech.gatech.edu/bitstream/handle/1853/50913/SerafinSerafin2004.pdf?sequence=1&isAllowed=y

21. Sonnenschein, D. (2011) Sound spheres: a model of psychoacoustic space in cinema. The New Soundtrack, 1(1), 13-27. Doi: https://doi.org/10.3366/sound.2011.0003

22. Tagg, P. (2004). Gestural interconversion and connotative precision. Retrieved from https://tagg.org/articles/xpdfs/filminternat0412.pdf

23. Tuuri, K., Mustonen, M.-S., & Pirhonen, A. (2007) Same sound - different meanings: a novel scheme for modes of listening. Retrieved from http://www.auditorysigns.com/kai/papers/Tuuri_etal_2007_Same_sound_different_meanings.pdf

24. Ward, M. S. (2015). Art in noise: an embodied simulation account of cinematic sound design. In M. Coëgnarts & P. Kravanja (Eds.), Embodied cognition and cinema (pp. 155-186). Leuven: Leuven University Press.

25. Whittington, W. (2007). Sound design and science fiction. Austin: University of Texas Press.

Notes

¹ Also known as video mapping, architectural mapping or spatial augmented reality.

² A public space well known to the spectators, which transforms itself and offers a different ‘reality’ even if ephemerally.

³ Described by Chion (1994) as the “irresistible weld produced between a particular auditory phenomenon and visual phenomenon when they occur at the same time” (p. 63).