Augmented Tangible User Interfaces (TUI)

Novel Human Computer Interface based on Cross-Reality and Artificial Intelligence

This project explores the design of tangible, AI‑enhanced interactive systems that combine projection, real‑time sensing, and cross‑reality interfaces. By linking physical models with camera‑based tracking and intelligent simulations, users can manipulate complex scenarios through direct, hands‑on interaction. The approach supports a wide range of applications, from military planning and vulnerability assessment to urban redesign and environmental analysis. It enables learners to understand complex systems by visualizing how their actions reshape projected information in real time. Through interdisciplinary collaboration, students develop technical, analytical, and creative skills essential for modern decision‑making environments.

Context

Tangible user interfaces augmented with artificial intelligence, cross‑reality technologies, and insights from neuroscience create powerful environments where users can think, learn, and decide through direct physical interaction. By blending real objects with digital intelligence, these systems allow complex information to be explored intuitively—leveraging gesture, spatial perception, and multisensory feedback to strengthen cognitive engagement. AI personalizes the experience in real time, adapting the interface to users’ behaviors and cognitive states, while cross‑reality immerses them in rich, contextualized scenarios. Neuroscience‑informed design ensures that these interactions align with how the brain processes attention, memory, and decision‑making, resulting in learning experiences that are more natural, more efficient, and more impactful.

In military and defense contexts, they allow planners to physically manipulate terrain models while simulations of troop movement, line‑of‑sight, vulnerabilities, and operational risks are projected and updated dynamically based on camera‑tracked configurations, enabling faster and more intuitive scenario analysis.
In urban planning, the same technology supports collaborative redesign of neighborhoods or critical infrastructure by letting stakeholders explore zoning changes, traffic flows, or environmental impacts directly on a manipulable 3D model, with AI generating predictive overlays in real time.
For climate‑change and biodiversity studies, these interfaces help scientists, policymakers, and the public visualize ecological interactions, rising sea levels, habitat fragmentation, and mitigation strategies by physically reshaping ecosystems on the table and immediately seeing data‑driven projections of long‑term consequences.

By grounding abstract data in a tangible, interactive medium, such systems foster deeper understanding, collective intelligence, and more informed strategic decisions across domains where complexity, uncertainty, and human collaboration are central.

General Overview

A tangible interactive system of this kind operates through a continuous feedback loop between the user, a physical model, a camera system, and a projector. The user manipulates tangible objects placed on a table, while one or more cameras capture the geometry, configuration, and movement of the setup in real time. Computer‑vision algorithms interpret this evolving scene—detecting positions, shapes, orientations, or patterns—and feed this information to an AI‑driven simulation or decision‑support engine. The system then computes appropriate visual responses, such as overlays, dynamic maps, indicators, or projected simulations, and sends them to the projector. The projector updates the surface instantly, allowing the user to see the consequences of their actions directly on the physical model. This creates a tightly coupled perceptual loop in which every gesture modifies the projected information, and every projection influences the user’s next manipulation—merging the physical and digital worlds into a unified, interactive mixed‑reality interface.

Provided Materials

All material to design a TUI

Developed Skills

Computer Vision & Sensing
Projection‑Mapping & XR Technologies
Artificial Intelligence & Simulation
Human–Computer Interaction (HCI)
Software Engineering & System Integration

Python/C++ for CV/AI pipelines. Unity/Unreal or similar engines for rendering and simulation. ROS/ROS 2 or equivalent middleware for modular data flow. Real‑time system design, concurrency, networking.

Pedagogical Goals

Understanding Complex System Design
Developing Cross‑Disciplinary Collaboration
Mastering Real‑Time Perception–Action Loops
Applying AI to Human‑Centered Interaction
Prototyping and Iterative Innovation
Enhancing Decision‑Making and Visualization Skills
Developing Creativity Through Real‑World Applications
Strengthening Communication & Project Management Students document their processes, justify design choices, present results, and work in agile, multidisciplinary teams.