In this set-up, the interaction is not only reactive to the stimuli from the users, but it is composed by a wide range of site-specific elements, including the live performers, the multiple digital frescoes, showing the embedded code on the images, and the space itself. In this way, kondition pluriel creates a hybrid environment in which elements are interconnected and react to each other. The installation behaves as a “living environment” communicating with the audience in many ways.

This is how the artist duo has managed to create a participatory experience, in which interactivity is hybrid and multisensory. Throughout their career, the artists have exhibited numerous methodologies to reflect on the potentiality of human-machine interaction, creating staged installations with cutting-edge technology and choreographing the relation between the audience and the machine.

Martin: Yes, that is one thing I forgot to mention earlier. When observers are close or in front of the building, they can see the words and letters and eventually feel embedded into sound and images. As the visitors go further away, the dynamic and figurative content of the images become very clear. For example, if one is standing around 50 or 60 meters away from the building, one sees almost a clear video image projected there. There are, for example, 3D renderings of sequences of migrants walking, and there are also some virtual landscapes, where we can perceive a world map. Additionally, there are images depicting refugee boats on the sea and burning forests, etc. But as we come closer, it becomes a very abstract visualization of words. The texts serve as reminders, of the human rights and of our current state of climate crisis.

Flood is also based on the movement of visitors, who perceive completely different things as they approach or move away from the projection surface. As Marie-Claude mentioned, projection mapping is a field that has been mainly explored, by focusing on the phenomenon of architectural trompe-l’oeil. What we do is work with the physicality of the building and the space, and the animations we create generate optical, but furthermore, perceptual illusions. These illusions are not based solely on the visual architecture of the building, but we use the dynamics of the building. For example, last year, Flood took place onto a round building, and the animation was made differently to evoke a stronger sense of being sucked into or repulsed from the image. Now in Bad Ischl, it is the same; we adapted the animations significantly to achieve these kinds of perceptual illusions.

Alejandro: Understood, because this is what I liked; that in “Flood” you are more focused on a social aspect. And I am also interested in the content that you selected, but let's continue with the next question.

As an art-group, your works are born from merging various disciplines. How would you describe your process of co creation given that you come from different backgrounds?

Martin: The term interdisciplinary is very often used and sometimes too easily used. In our case, when we decided to merge interactive media with professional dance and choreographic performance, each of us had different interests, expectations, and motivations. For my part, what interested me most was that I was eager to investigate about the difference of impact of the presence of “professional bodies” (performers) interacting with the interactive environment I create, in comparison to those of ordinary spectators. To return to the term of interdisciplinarity: During the course of our collaboration, we realized very quickly that on a very short term, disciplines cannot merge so easily. We came to the conclusion that this requires a long process, and now after we have been working together for almost 25 years, our disciplines are very much intertwined with each other.

At the beginning of our collaboration, it was a difficult process.  We believe that one big challenge with interdisciplinarity is that one needs to let go from one’s own discipline to be ready to give up certain elements, to be able to develop new avenues and new concepts and methods. And this was a long journey, and since our work was very much research oriented, interdisciplinarity itself became part of our research: how to create with heterogenous elements combined on an equal footing?

Marie-Claude: When we began our work together, I remember our determination about resisting against the idea that one field of art would be a decor for the other. To avoid that, we tried to get at the essence of the deeper principles which are at the source of each of the different inputs and disciplines. I had a background in performance, choreography, and dance, and at the same time, I had a strong interest in working with systems. In the contemporary dance world, I felt a bit of an alien because I already had this desire to work with installation, and audience participation.

When we started, working with sensors in public space or in a space where the public is not sitting in the dark was rather uncommon. A situation in which all bodies are sharing a common space, the body of performers, of dancers, and visitors. In such a context, we can see the space itself becoming a “body” in a sense. We intertwined a lot of different media and used different type of projection surfaces. And the installation-space became like a huge “body” composed of different bodies, with the aim of “de-hierarchizing” the various elements involved.

Martin: I just want to add that our approach has become very fluid in that sense. Over all these years, I learned a lot about dance, performance, bodywork, and presence, as much as Marie-Claude has learned about the media technologies involved and about the media grammars used. But at the beginning, it was a cumbersome and a difficult process. I think, it is a question of time, how much time you investigate to develop the artistic core elements. And I believe that this is something which is sometimes overseen in the notion of interdisciplinarity. Because for people, it can be easy to say “okay, we have a group of interdisciplinary researchers and now we work for two months”, but I think that usually it really takes a lot of time before starting to understand the formation of a new artistic language.

I remember for example that at the beginning, the development of our work took often place in the frame of residencies, five weeks here, three weeks there, etc. My colleagues and me were setting up the media elements, taking hours for designing an interactive setup and then, when we were finally ready and said to the performers “okay, now we can start working together”, the dancers often started to talk, they talked and talked, until finally they could be ready to begin.

This kind of situation led me to understand that one of the performers’ main tools is communication. They communicate, to define what the performative rules are, and how they apply them, how they embody them. We, the media environment part, we need hours to prepare our setup, but then the dancers move for two minutes and then we say “okay, stop, we have to change”, it can take another two hours to change something. So, only the time dynamics was a very difficult challenge because for sure, the performers can change the movement in the course of some seconds, but for us it took us hours to perform changes, and sometimes implied modifying the whole system.

Marie-Claude: Yes, it was challenging because just in terms of means, for example, we sometimes had a team of five-six dancers that we were paying per hour, and they came to warm up before the rehearsal and then the media environment system suddenly blocked. So, they had to go home, or they had to wait for two or three hours while we had to keep paying them while they waited. And this was creating a lot of pressure and misunderstandings. Such situations demand to be Zen, and the more we know about all these production issues and have them in mind while working, the easier it becomes. Because not everything can be planned. One has to integrate the spontaneous and unpredictable as well; one has to adapt.

Alejandro: When designing an interaction, where do you begin? And how do you know that the interaction is ready to be shown in public and used by the spectators? Take for instance, “Future Room”, how do you decide to use voice recognition technology?

Martin: Most of our works are in one way or another interactive because we are very much interested on the notion of “being live”. The notion that the artistic work is not pre-written, but that it is open-ended fascinates us. Many of our works involve the idea of a process, in which the spectator plays a crucial part. For me, this is a very exciting idea: that the spectators become part of the artwork. How do we decide on the interaction? Most of the times, it is always in combination with the topic, the format, and the technology of the project. For example, when we were developing the concept of the “Future Room,” it was very clear that we wanted to make it as a dome work. A considerable amount of people worked together on that piece, under my direction. The work was commissioned at the occasion of the celebration of the 150 years of the University of Applied Arts. It was a collaboration with Peter Weibel and Gerald Bast, with whom we decided that the Future Room should become a sort of metaphor for a machine that could predict the future.

To my opinion, when working with a dome, it seems to make sense not to have a linear experience, because the space is surrounding the user. As artists working with this medium, we felt the urge to foster interaction. The difference when using a VR Headset, for example, is that it is not a very social situation. The dome is, on the contrary, a very social environment, where the immersion can be experienced together with several people. When we started developing the “Future Room”, we decided to project words into the dome environment, and invited the audience to choose a topic by speaking it into the microphone. And these “word clouds”, as we called them, had an immersive quality embodied by the way they moved, the way they behaved. And this is how very quickly, the mode of interaction of that work took shape, because it was for us the most logical access for this work.

Marie-Claude: Talking about the social space, the interaction, or the configuration of how people behave in this specific context and the situation it creates, I believe, are an integral part of the artistic proposition. The fact that visitors observe each other, and that when one person gets up to go to voice his choice of topic in the microphone, have a strong performative component. That is to say that it is not only the dome-space, or the imagery projected on its surface that constitutes the piece, but the whole social choreography of how people behave, for example: some people are shyer, and they observe the others for a long time before entering themselves into action. Sometimes they even wait that the others are gone, to get up and intervene. All these dynamics are fully part of the artwork as well. This is one example, but it is contributing to portray, to come back to your question, how we conceptualize, prototype, try out and finalize the interactive aspect of each piece.

Alejandro: Yes, exactly, because when you focus on this social space, then a participatory dimension emerges. In relation to that I would like to ask you the next question. Many of your artworks can be accessed and shifted by the performance and the audience inside a specific context. Examples of this could be “Swarming Launch” and “At Play”; in the first one by using mobile phones and in the latter by the movement of plastic storage bins. How relevant is the participatory element for you in the process of conceiving an idea?

Martin: The participatory element is very important. In that sense that for us, the spectator almost becomes part of the performance and becomes another performer. For sure, this is a delicate situation, because, for example, if we imagine some of the interactive environments developed in the 80s, beginning of the 90s, some of these systems were very complex. For example, the Very nervous system by David Rokeby, which was only revealing its full potential if somebody would really move in a very elaborate manner inside it. But when it was installed in a gallery space, there was never really people freaking out, jumping or screaming, because the general public would be shy. And this is what was sad about this very rich work, that the common visitors never have had the impulse to explore its depth. So one could say that with his very nervous system, David Rokeby, at least in this work, forgot - or let’s say he did not forget, because he realized afterwards and we had discussions with him about it - that at the end there has to be a balance between the amount of energy the viewer has to provide to activate a system, and the system's ability to activate itself independently of the user.

As interactive media artists, if we make it too complex, the visitors will often not be able to explore the work. Because people are shy, they will touch maybe, but they are not going to perform a complicated choreography. So, we need to always find a balance between what is expected of the audience and the performativity of the proposed system. But the participatory factor is for us very important because we believe in an open-ended artwork, and we believe in the intermingling of life performance and digital art. We think that this is a relevant combination because processes are always reiterated only live with the spectator and this is what makes it a unique experience. And that is why it is a totally different art experience than looking at a picture hanging on the wall.

Marie-Claude: If I can add, you were mentioning Swarming Lounge and At Play. So, if I think of Swarming Lounge, this piece in reality has maybe a hidden theme, - or several ones - but one of the hidden themes for me - as a person who is occupied with performance - is the interchangeability of roles between visitors, performers and virtual figures. So, the whole movement vocabulary of the professional performers and of the virtual figures is based on finding “in between” daily gestures of visitors, which are kind of magnified because they are performed by the performers. It is like magnifying the banal. This whole spectrum in which visitors are watched by performers who start behaving even less like performers and more like visitors leads to a confusion of roles.

Interchangeability is a very interesting theme for me, especially in the context of mixed reality. Because this is the context of a piece where the visitors via their phone are capturing or watching virtual characters, other visitors like them, and performers. The idea is that all these people, these protagonists, are not clear about which role they have, and this is what interests us. It is this kind of erasing of borders between them. It could also be said that this artistic approach also deals with the breaking of the fourth wall with digitally augmented performance.  This question of the fourth wall was intensively discussed in the 60s or the 50s, but emerged as early as in the 19th century, and in our case, there is a reinvestigation of this notion with means of today.

Martin: I would like to share an anecdote in connection to one of our earlier works, “passage”, where the public was invited to touch the performer as such, when we went to Paris in 2008 to perform this project at the théâtre du Casino. At the time, we were surprised and somewhat frustrated by the reactions of the audience, many of whom were more concerned with documenting the performance with their smartphones than experiencing it. So, it is at this occasion that we decided that we needed to develop a performance-project in which the audience musts use a phone, not as a tool to document their experience, but as part of the artistic experience itself. It took quite a while until we finally made “Swarming Lounge”. This is an example of what we meant earlier when we were talking about intuitive processes, motivations for creating a work of art that sometimes go back a long way, as in the example I have just given you. We wanted to question the very fact that spectators were running with their phones and filming everything was taking them away from the performance experience. This idea followed us unconsciously, and it was only a number of years later that we created a work in which using the phone is at the heart of the performance.

Marie-Claude: Today this phenomenon seems normal, but it was all the more of a shock for us in 2008 because we had just arrived from Canada, where cell phone networks took longer to spread because of the distances involved. And suddenly, presenting our work in Paris, we were shocked by the fact that, for the first time, the shared moment of co-presence with the visitors, without being disturbed by the outside world, was broken by the use of the phone, and that was a painful turning point for us.

Alejandro: It’s very interesting because the work somehow takes the ordinary and transforms it into a piece like, as you said, breaks the fourth wall. One cannot say this is the ordinary or something extraordinary.

Then, I would like to ask you about improvisation because last time we talked, you mentioned its importance during the conceptualization of the project. Would you like to elaborate on the notion of improvisation within an interactive system? For instance, in 'Swarming Lounge,' the topic we just discussed, how does this process work for the performers? How are they trained or prepared? Could you describe the process of setting up a dramaturgy for them?

Marie-Claude: Martin named the expression “open ended”, and I would like to add to this idea the fact that most of the time, we structure our work from a kind of matrix. It is not what we sometimes call, in dance, “a written piece”. A written piece in dance is like written music, where there are musical notes on a score where you could read the music. There is also written choreography where each movement is a fixed element, and which functions as a score that the performers go through in a linear way. Our way of working, particularly as we work with interactivity, is geared towards audience participation. So there has to be room for them to intervene, and flexibility within the score/scenario of the performance. The question for us is more of setting up a kind of matrix, providing access to a bank of possibilities.  Because there are many possible situations, and we need to be prepared for different scenarios to deal with the unexpected. That is where improvisation comes in, but a structured improvisation with very precise parameters.

Martin: The performers have a set of rules they can rely on. So, they are improvising but nevertheless, they have a rigorous guidance inside their improvisation. For a period of time, especially in our work “passage” and some other projects, we went even further as the musician, and myself, as the person in charge of the visuals, were also on stage, and were changing the interactivity setting during the show sometimes. We wanted to explore the notion of a reactive or interactive environment, in a form where the system even continues to transform itself while the audience is interacting with it. And we were interested in the type of situation that can come out. One could say that these projects have been like improvisation sessions with the public in real time, and that we were generating new media environments, adapting to their behavior, on the fly.

Alejandro: This is quite interesting because then the interactive system somehow absorbs what is happening in this place. For example, the audience or interruptions become integrated into the piece itself.

Martin: Just to add a little bit more: There are different types of spectators. Some of them come and they want to check out how the piece works, but only technically, they are not that interested in the artistic aspect. Then, others enter the work carefully listening, watching and sensing the experience, and only later start getting interested about the technology behind it. To give an example: we once had one specific sensor technology embedded in the installation, that was producing an intense sound, when triggered by the public, let’s say a big bass drum or so. If someone would activate this sound 20 times in a row, always the same, it would become very annoying. And this was sometimes happening, as some audience members always want to “test” the limitations of the system that is offered to them to play with. That is an example of why, in a case like this, we allowed ourselves to change the behavior of this one sensor over the course of the performance. It is also about making the audience aware that they have a certain responsibility, and about making them part of the experience.

Alejandro: Exactly. And now that you've mentioned this, I want to ask you about human-machine interaction and how it feeds back into the device itself. So, what part of the concept do you think could be defined by the systems of the machine itself, or is this not the case?

Martin: We are currently working on some projects which are going in this direction. There are different degrees of interactivity, we have projects that are time-based, they change according to time, for example "passage" or "At Play". Depending on the audience’s behavior, the system reacts differently, but once this person goes away, it goes into standby mode. We are now faced with the question of what can be defined as intelligence within a system. And in this sense, the intelligence of these types of systems may not be so high.

Nevertheless, they are still interactive systems because they react to the user input. There are many human-machine interaction systems, but they don't live on their own. They are not really generative entities in that sense. This is something which we already started to emphasize in our works. In the last few years, we started to experiment more and more with artificial intelligence. These technologies have become more accessible for us to integrate and now we are working intensively on the notion of adaptive behavior.

We want to achieve a certain system scenario, where the media environment will adapt to the behavior of the public, based on a learning algorithm, because at the moment most of our systems are based on mapping. There are time layers intertwined. There are certain conditions which change over time, but most of the elements are scripted with a certain parameter of randomness. In that sense for example, many of our projects work with thresholds, with input over time. But they are not living systems in themselves. They come to life only when somebody is using them, and then they can come very much to live depending on how one behaves.

Alejandro: Another relevant feature in your immersive installations is the fact that the artworks can be experienced simultaneously with all the senses. The importance of movement and holistic bodily experience is often emphasized on your oeuvre. In artworks like 'At Play' and 'Diver,' to mention a few, spectators perceive the installation as a total experience, which includes kinesthetic elements. Whenever you consider an artwork, in what way is this relevance taken into account?

Marie-Claude: I would say that in almost all our work, kinesthesia plays a central role. This is the case, for example, in our current works. They are not based on the traditional definition of choreography or performance, but on the other hand, they are relying on one of its most central components, kinesthesia, which is at the very source of what underpins these fields of practice. It is all about choreographing movements in space. And as Martin was saying, because our approach is intrinsically linked to interactivity, our works are not like artworks that exist outside the people for whom they are intended, like a painting on a wall, and a 2D video or film, for example. For us, the perception coming from the user, performer or visitor is an integral part of the work. It is a three-dimensional experience.

Martin: This is important because when we talk about going beyond the boundaries of the real, with our visual, auditive and multisensorial experiences, we believe that the human brain, the human cognition, will embody and function when the body is incorporated in a process much better than when it stays static. There is a different embodiment taking place in a different perception and recognition of the environment when the whole body is involved in an experience. So that is why we are not that interested in that the people are standing fixed, while they are encountering the work, we like them to be able to be active. Because we believe that in this way, the quality of exchange and the reception process is much higher.

Alejandro: To conclude, I would like to ask you about your research at the Dome Lab. How has it been over the years that you've been working with the dome? What aspects are consistently important when you're working in the dome, and what factors should be taken into account for your research?

Martin: For us, the dome's ability to generate kinesthetic empathy is a crucial feature from which we develop our projects. This same property to stimulate proprioception is also found with virtual headsets, but virtual headsets are limited to a solitary experience.

I remember when I got first invited for working in a dome, I was already very much interested in the virtual environments called “caves”, developed in the 90ies, but it was rather cumbersome to work with them. These environments are square in shape, but the image has a physical impact, a bit like in the dome. For me, the physical resonance generated by these environments is what makes them most exciting. And the other interesting aspect is that the observer has multiple points of views and is therefore not only looking in one direction. Somehow this works in accordance with our approach to our work, that one might miss something in one corner of a space, but on the other corner can experience something else. So, there is no linear situation, and this is somehow amplified in the dome. It is a 360 grades imagery, so it means that one can look in one direction while something has happened in another direction, and this is what is called “fair visualization”. How to balance these elements? We continuously research about this. We think we learned a lot, but we believe that a lot of grammar can still be explored.

Marie-Claude: When we started to work with the dome, because this was in 2010, this was a field - and still is - which was not so much investigate by contemporary artists. It was more used somehow in entertainment and education contexts. We thought that it made sense that artists would explore that media as an artistic field in its own right. And for us, working with the dome was a natural pathway because of the works we did before. Before we started working with the dome, we had several times developed installation works with multiple screens that generated panoramic environments, so when this new medium became available to us, we didn't hesitate to explore it.

One challenge when one wants to use the dome in contemporary art, I would say, is the spectacular aspect of the dome. Because we are talking about huge projection surfaces. How can we manage this spectacular aspect while creating tension, intimacy, doubt, ambiguity, concepts that relate more to contemporary art and not just generate a kind of flashiness, more related to pop and entertainment?

So, we have always been preoccupied with the idea of figuring out how to work with the spectacular dimension of the dome in an interesting way. In addition, we talked about kinesthesia in the dome, vertigo, imbalance. It can be so extreme that some people lose their balance because of it. We have already seen this happen to the point where some spectators had to leave because it was too much for them.

This is an extremely physical property. And it can be an artistic tool, so we are trying to find out how to use this, how to compose with it, how to create the artwork, including that physical element into the art concept. Then also working with the architecture of the dome, as what can be seen as a “negative space”, in the sense of a void, an empty shell on which virtual images are projected. And this empty space can be a perfect performative and social space. So, one of our great interests is to create relations between what is happening in that physical space and the virtual content of the projection. To find how these two types of spaces interact with each other and contribute to the general content of an artwork.

Alejandro: Thank you very much. Thank you, Martin and Marie-Claude. Do you have anything else that you would like to add?

Martin: We are currently working on developing a Holodeck. The idea is to develop a speech recognition system that works like a prompt. And in real time, as real time as we can get, that one could really start building one own landscape into the dome by saying some words, on which another user could build, this environment shaping into a holodeck. At the moment, it is only a technological research, but we will be extending it into the creation of an artwork. We are not yet sure how, but we will investigate in that direction.

The Archive of Digital Art would like to thank kondition pluriel for their exceptional work and research in the field of Digital and Interactive Art, as well as for their participation in the feature article.