Three Hypotheses, and Nobody Looked

Every remote collaboration system I had built up to this point shared cues that people send on purpose. A reconstructed room. Gaze rays and hand meshes. A voice placed in 3D. Each one takes something you are already doing deliberately and moves it across the network.

Sitting next to somebody gives you a second channel that none of those cover. You can hear their breathing change. You notice when they go still. You catch the half-second pause before they answer, and you adjust without deciding to. None of it is chosen and all of it is information, and a video call throws essentially all of it away.

Headsets are unusual in that they already sit against the body. If you are willing to add a wrist or finger sensor, you can read heart rate, skin conductance and a rough attention signal in real time. So the argument was simple: put those in the shared space and collaborators will understand each other better.

This post is that argument and, eventually, the answer to it. The two are separated by two years, which is roughly how long it takes to find out whether you were right about anything in research.

What I built

PSCVR ran in Unity 2021 across two HTC Vive Pro Eye headsets on a local network. Ubiq handled the social VR layer, chosen over the alternatives because it was open source and readable rather than because it was the fastest. A Shimmer3 sensor gave galvanic skin response and heart rate. A Looxid attachment clipped to each headset supplied an attention value from EEG.

The task was a virtual engine with more than 50 separable parts, mounted on a stand in a replica workshop with the right tools laid out around it, built on the volumetric playback work from earlier in my PhD. Instructions appear, you find the tool, you take the part off.

Two details from the build that I still think were right.

The first is calibration. A GSR reading means nothing as an absolute number, so before each session participants would sit with a black screen for a minute and then run one-back and two-back tests. Those are standard working memory tasks with a known difficulty gap, which gives you a personal low, medium and high to scale against instead of a threshold borrowed from somebody else's body.

The second is that questionnaires were answered inside VR, between conditions, without removing the headset. Taking a headset off breaks EEG electrode contact and disturbs the GSR sensor, and then your next condition is measuring a slightly different person.

The part that was missing

There was no study.

The paper proposes four conditions: heart rate alone as the control, then heart rate plus cognitive load, heart rate plus attention, and all three together. It proposes a set of measures. It lists use cases in teaching, research tooling and counselling. And it states three hypotheses:

• Knowing your partner's physiological cues improves performance.
• Knowing your partner's physiological cues improves co-presence.
• Knowing your partner's physiological cues improves immersion.

That is where it ended. It went to the adjunct track at ISMAR 2022 as a system and a plan, which is a legitimate thing to publish, and I want to be straightforward about what it means: I had built the apparatus and written down what I expected, and I was arguing for a position I had not tested.

I believed all three hypotheses. They felt close to obvious. If you can see that your partner is overloaded, of course you will work better together.

Two years later, the study

By 2024 the system had grown up. Same shape, more sensors, all of it streaming into Unity over Lab Streaming Layer.

Three cues, three visual treatments. Heart rate is an animated heart that beats with your pulse and shows the number. Cognitive load is a brain icon on a low, medium and high traffic-light scale, derived from GSR. Attention is a moving circle showing a percentage, from the EEG.

The task was the engine, now with twelve jobs graded easy, medium and hard, from removing flywheel bolts up to the crankshaft bearing. One person holds the instructions, the other does the work, and they swap halfway with the difficulty balanced so nobody gets the easy half twice.

Here is the whole thing running, with the cues live:

Twenty-eight participants in fourteen pairs, four conditions, within subjects. Exactly the design the 2022 paper had proposed.

What happened

Task completion time: no significant difference across the four conditions. Workload on the NASA-TLX: no significant difference. The first hypothesis, the one I had been most confident about, simply did not hold.

Two things did move. Co-presence differed significantly between conditions, so the second hypothesis survived. And attention allocation differed strongly: when asked to rank the four conditions, 15 of 28 people put the attention cue first, and more than half preferred it overall. The third hypothesis never really got tested.

Then we looked at the eye tracking, which is the number that has stayed with me.

Participants looked at their partner's physiological cues for less than 10% of the session. For the largest group of them it was between 1% and 5%. Around 70% of the looking that did happen occurred in the first minute.

That is a habituation curve. The cues were novel, people checked them, and then they got on with the engine.

Why I think it failed

The paper offers several explanations and I believe all of them, but they are not equally interesting.

• The task was wrong, and that is on me. Engine assembly is procedural. Your partner's stress level does not change which bolt comes off next. We chose a physical task because it is realistic for training, but knowing someone is overloaded only matters if you can act on it. A negotiation or a design decision would have given the cues something to do.
• Heart rate was in every condition. That was a design error, and it was baked in back in 2022. Participants could read stress off the heartbeat alone, so the conditions were never as separated as the study design implied.
• The cues disagreed about time. Heart rate updated in real time. The Looxid attachment averages cognitive load over a 10-second epoch. So two numbers sat side by side in the same visual language, one describing now and one describing ten seconds ago, with nothing telling the user which was which. If you are going to fuse signals, they need a shared clock, and ours did not have one.
• People did not know what to do with the information. This is the finding that mattered most. Participants said outright that they struggled to interpret their partner's signals, and several said they would rather the system made the judgment for them. One wanted to see everything and choose what was relevant per task. Another said processing it all was simply too much.

The thing worth taking away

I had spent two years assuming the hard part was capture. Get the sensors on, get the signals clean, get them into the shared scene at low latency. We did all of that and it worked, and it changed nothing about how well two people assembled an engine.

The hard part is interpretation. A number on a wall is not a cue, it is homework. The participants who asked for the system to do the judging were describing the actual research problem, and every system I have worked on since has been some version of trying to answer them.

The one genuinely encouraging result is the attention cue. It was preferred, it moved co-presence, and I think that is because it is the only one of the three that answers a question people are already asking during collaboration, namely whether the other person is still with you. Heart rate and cognitive load answer questions nobody was asking yet.

Worth stating plainly: this work did not show that sharing physiological cues improves collaboration. It showed that we do not yet know how to present them so that anyone would use them.

What I would tell 2022 me is not that the idea was wrong, because the sensing worked fine. It is that I had spent all my design effort on capture and none on the question of what a person is supposed to do with a number that appears next to their colleague's head.

The system paper is PSCVR: Physiological Sensing in Collaborative Virtual Reality, ISMAR Adjunct 2022, with Yun Suen Pai, Huidong Bai and Mark Billinghurst. The study is A User Study on Sharing Physiological Cues in VR Assembly Tasks, IEEE VR 2024, with Ryo Hajika, Kunal Gupta, Tamil Selvan Gunasekaran, Yun Suen Pai, Huidong Bai, Suranga Nanayakkara and Mark Billinghurst. The same year as the first one I also wrote a doctoral consortium paper at IEEE VR laying out the whole thesis, which is a similarly confident document written before most of the evidence existed.