Fused Twins: A Cognitive Approach to Augmented Reality Media Architecture

Jascha Grübel, Michal Gath-Morad, Leonel Aguilar, Tyler Thrash, Robert W. Sumner, Christoph Hölscher, Victor Schinazi

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

8 Citations (Scopus)
201 Downloads (Pure)


Recent advances in Augmented Reality (AR), the Internet of Things (IoT), cloud computing, and Digital Twins transform the types, rates, and volume of information generated in buildings as well as the mediums through which they can be perceived by users. These advances push the standard approach of media architecture to embed screens in the built environment to its limits because screens lack the immersive capacity that newer media afford. To bridge this gap, we propose a novel AR approach to media architecture that uses a Digital Twin as a platform for structuring and accessing data from various sources, including IoT and simulations. Our technical contribution to media architecture is threefold. First, we extend the possibilities of media architecture beyond embedded screens to three dimensions by presenting a Digital Twin using AR with a head-mounted display. This approach results in a shared and consistent augmented experience across large architectural spaces. Second, we use the Digital Twin to integrate and visualize real physical sensor information. Third, we make artificial occupancy simulations accessible to everyday users by presenting them within their natural context in the Digital Twin. Observing the Digital Twin in situ of the Physical Twin also has applications beyond media architecture. Fusing the two twins using AR can reduce the cognitive load of users from consuming big and complex information sources and enhance their experience. We present two use cases of the proposed Fused Twins in a university building at ETH Zürich. In the first use case, we visualize a dense indoor sensor network (DISN) with 390 IoT sensors that collected data from March 2020 to May 2021. In the second use case, we immerse visitors in agent-based simulations to enable insights into the real and projected uses of space. This work brings forward an ambitious vision for media architecture beyond traditional flat screens, and showcases its potential through fusing state of the art simulations, sensor data integration and augmented reality, finally making the jump from fiction to reality.

Original languageEnglish
Title of host publicationProceedings of the 5th Media Architecture Biennale Conference, MAB 2020
PublisherAssociation for Computing Machinery (ACM)
Number of pages6
ISBN (Electronic)9781450390484
Publication statusPublished - 28 Jun 2021
Event5th Media Architecture Biennale Conference - Virtual, Online, Netherlands
Duration: 28 Jun 20212 Jul 2021

Publication series

NameACM International Conference Proceeding Series


Conference5th Media Architecture Biennale Conference
Abbreviated titleMAB 2020
CityVirtual, Online


Dive into the research topics of 'Fused Twins: A Cognitive Approach to Augmented Reality Media Architecture'. Together they form a unique fingerprint.

Cite this