The Advanced Media Research Group is an interdisciplinary research group at the School of Interactive Arts + Technology at Simon Fraser University. The group includes scholars from the fields of humanities, computer science, music and cognitive science whose work focuses on video art, new media theory and analysis, artificial intelligence, computer graphics, metacreation, computational systems for creative applications, music composition, and soundscape design.
The Generative Media Project has been funded by the Social Sciences and Humanities Research Council (SSHRC) of Canada. It is exploring the use of computation and generative algorithms in moving image and sound art. It addresses a series of related research questions and creative challenges. The first is “How can computationally generative presentation systems maximize semantic coherence and visual flow?” This work is a refinement of the PI Jim Bizzocchi’s existing work – a fully functional generative system for the random combination and ongoing presentation of video clips. The second question is “How can a visually generative system be improved through the inclusion of an integrated audio stream?” This new sound channel will complement and extend the expressive capability of the visual system. Finally, “How can a generative presentation system incorporate narrative and storytelling capability?”
These questions address a number of non-trivial challenges. Computationally-generative artistic processes require the combination of scientific and technical sophistication with a clear sense of aesthetic purpose. The effective integration of image and sound requires a deep understanding of the capabilities embedded within the individual domains and the potential for synergy when they are combined. Narrative is a powerful and pervasive human phenomenon, one which is dependent on a high degree of semantic coherence and ordered presentation – both of which are difficult to combine with generative computational processes.
The project team is building upon a strong foundation of existing research and creation. Bizzocchi and Calvert have built a generative system that presents an ongoing flow of visual imagery drawn from a database of shots and joined by a set of visual transitions. The system runs indefinitely, and continuously varies the visual flow through constant recombination and variation of clip selection and transition choice. Pasquier and Eigenfeldt have been building a series of computationally generative audio systems that combine sonic flow with constant variation that have been shown to be indiscernable from professional handcrafted compositions. For example, Coming Together: Freesound continuously uses sonic features and metadata to create ongoing soundscape compositions from a database of audio recordings,. All of the team members have strong records of scholarly publication and collaborative artistic creation across joint aesthetic/technological projects and teams.
Our research-creation approach includes the close reading and analytical deconstruction of exemplary generative moving image and sound works, leading to the development of an integrated conceptual framework that supports both analysis and creation. The conceptual framework will include the understanding and the design of visual coherence/flow, the complementary relationship between generative sound and image, and the incorporation of narrative processes and outcomes. Artistic creation with its associated technical development will be fully integrated. The review of the critical discourse and the analysis of exemplary works will clarify the initial project directions and challenges to be addressed. Prototype creation processes will include the artistic application of techniques and tools from multimedia, signal processing, artificial intelligence and machine learning to identify potential solutions and fresh challenges. Mature artistic works will instantiate and consolidate our intellectual and aesthetic findings. Our results will contribute to understanding and appreciation of generative art and its cultural potential, and will be shared in a series of scholarly papers and presentations, gallery exhibitions and public artist talks.