Engineering case study / 2026
Beyond the Bloom
Connecting body motion to a shared folktale.
Team lead · AI/ML · Unity system integration

The problem
Beyond the Bloom turns a Korean folktale into an embodied, multi-visitor exhibition. A visitor needs to see their movement reflected in a character and understand how an action changes the story. This requires more than recognizing a gesture: tracking, animation, story state, and visual response have to work together.
The interaction, in motion
My contribution
I led the five-person capstone team and worked on AI/ML, motion retargeting, interaction development, and Unity system integration. My scope centered on connecting the visitor's body to the running experience. Image generation, archive development, and the broader visual production were team contributions.
From movement to a story branch
The interaction pipeline connects Azure Kinect body tracking to avatar motion and an EfficientGCN-B0 action-recognition model. Recognition results drive branches in the folktale; the Unity scene brings those branches together with the character and exhibition visuals.
- Capture the visitor's body movement with Azure Kinect.
- Retarget the tracked motion to the character using the Unity animation and FullBodyIK workflow.
- Interpret actions and connect them to the current story branch.
- Present the response as part of the shared folding-screen experience.
Correcting motion at the engine boundary
Kinect joint coordinates and Unity Humanoid bones did not share the same orientation. I mapped JointId to HumanBodyBones, applied a 180-degree Y-axis correction and joint-specific basis corrections, and handled left/right joint mapping for a mirrored visitor view.
Tracking noise created another problem: small fluctuations in the skeleton could become visible shaking in the character. The retargeting pipeline averaged each joint's rotation over a 20-frame quaternion queue. Separating the initial Hips offset from the Pelvis delta also helped preserve the character's placed root position instead of allowing it to jump.
Recognizing an action is not the same as triggering it
If every classification immediately triggers an event, one gesture can fire repeatedly or a weak prediction can change the story. I connected action recognition to scene-level configuration for the model, labels, confidence threshold, and cooldown.
The final prototype's report describes 16 action branches connected to seven scene branches and six endings. Thresholds gate low-confidence results, while cooldowns prevent the same action from retriggering in rapid succession.
Outcome and takeaway
The team integrated the prototype and documented preliminary exhibition tests for Beyond the Bloom. Further tuning for the final installation remained part of the report's next steps. The project connects my research interest in human behavior with a real-time system people can enter and influence.
The lesson I take from this work is to design the path from input to visible response as a whole. Model behavior, character animation, and narrative feedback need to be considered together; improving one component does not automatically make the interaction clear.