Speech in ultrasound and in silico
I ran one speech experiment through two systems: a functional ultrasound (fUSI) recording of auditory cortex, and Meta's TRIBE v2 brain foundation model, which predicts fMRI responses from audio, text and video. In the ultrasound data, I measured the blood-response timing directly. The original pipeline's −1.6 s onset shift went the wrong way: the best fit was +2.8 s, which held on held-out blocks, and the response was broader than the fMRI model assumes (center of mass 7.3 s vs 5.3 s). I then replayed the same block design through TRIBE with acoustic controls, and found that its audio pathway treats time-reversed speech as speech. Its text pathway restores the response only when a transcriber can recover the words (93% for vocoded speech). Along the way I found and fixed a transcript-placement bug in TRIBE's pipeline, and built a tested, reproducible codebase that recovers effects planted in synthetic data.
code and methods TRIBE v2 (Meta)
1, Onset-shift explorer
The measured fUSI signal in auditory cortex, with the fitted Glover model overlaid for a chosen onset shift. The original pipeline assumed −1.6 s; the data prefer +2.8 s.
2, Response shapes
Each response peak-normalized over 0–25 s, with its center of mass marked (computed over 0–40 s). The measured ultrasound response is slower and broader than the fMRI convention; TRIBE's prediction is faster.
3, What does the model treat as speech?
TRIBE's block-locked prediction for a control condition against the matched speech block. Retention is control response divided by speech response: 1 means the model treats the control like speech, 0 means it ignores it. These are model predictions, not brain measurements.
4, Does hearing the words help?
Change in TRIBE's retention when the text pathway is added, by control. Each bar is labeled with the fraction of words a transcriber recovered. Bars whose 95% interval excludes zero are solid; the rest are faded. Intact is the null check.
- This is one recording with one synthetic voice.
- The TRIBE results are model predictions, not brain measurements.
- In this data, a slow response and a late event clock cannot be told apart.