EXPLORE & DOWNLOAD VALIDATION DATA
20,084 LINKED EVENTS · 200 MIDI · 8 STYLES

SMPC 2026 · VALIDATED REAL-TIME RESEARCH ARCHITECTURE

A REAL-TIME MULTIMODAL ARCHITECTURE FOR MUSICAL EXPECTATION:

AUTOMATIC HARMONIC TOKENIZATION WITH BEHAVIORAL AND PHYSIOLOGICAL SYNCHRONIZATION

Oğuzhan Tuğral · Department of Music and Dance · University of Massachusetts Amherst

< 1 sevent-to-record processing target met
01

WHY IT MATTERS

THE GAP IS NOT MORE DATA.
IT IS EVENT-LEVEL INTEGRATION.

Harmonic analysis and multimodal music cognition have advanced along largely independent methodological trajectories.

FIELD EVIDENCE

190studies
113journals
48countries
98%of health-science studies used no harmonic or melodic analysis.≈70%of music-domain studies used no explicit harmonic or melodic framework.
Post-hoc alignment weakens temporal precision and separates musical structure from response.
OUR RESPONSELive MIDI becomes explainable harmonic tokens synchronized with EEG, cardiovascular, and behavioral streams—without a pre-labeled reference at inference time.
02

HOW musWM WORKS

LIVE HARMONY BECOMES AN EXPLAINABLE TOKEN

No pre-labeled reference is required at inference time.

LIVE
MIDInotes
MODULO
12pitch classes
SHORT-TERM
BUFFERtempo + meter
LOOKUP
TENSORbass · root · key
RN
TOKENfunctional
2¹¹ = 2,048bass-normalized, transposition-invariant pitch-class configurations
O(1)constant-time retrieval

The lookup path preserves inversional and functional distinctions using local key, bass, root, and chordCode.

ONE SHARED EVENT TIMESTAMP

EEGCARDIOBEHAVIORAUDIO

The event is the unit of multimodal linkage.

03

RN VALIDATION

DO THE TOKENS AGREE WITH TONAL THEORY?

musWM vs AugmentedNet across 200 polyphonic MIDI files spanning eight Western musical styles; exact labels and broad harmonic functions scored separately.

7,273ONE-TO-ONE TIME-ALIGNED RN EVENTS · same file + measure · ±0.25 beat · no event reused

FUNCTIONAL AGREEMENT

Exact 3,691 · 50.75%Equivalent 1,469 · 20.20%Different 2,113 · 29.05%
91.38%musWM theory fit
88.82%AugNet theory fit
FUNCTIONALLY MATCHED N = 5,160
THEORYBASSROOTSTRICT
musWM96.0999.0799.8896.09
AugNet95.6298.84N/A94.19
GENUINE DIFFERENCE N = 2,113
THEORYBASSROOTSTRICT
musWM86.9994.2798.2086.99
AugNet77.9089.21N/A38.14
musWM-ONLY EVENTS N = 7,046
THEORYBASSROOTSTRICT
musWM89.2494.7298.5889.24
AUGNET-ONLY EVENTS N = 4,012
THEORYBASSROOTSTRICT
AugNet85.8294.14N/A69.24

“Only” = unmatched within ±0.25 beat, not automatically wrong. AugmentedNet does not expose a root estimate; root is reported as N/A.

04

REAL-TIME VALIDATION

COVERAGE & RECORD INTEGRITY

200 / 200MIDI files parsed
20,084 / 20,084complete multimodal records
21,304 / 21,304shared musWM core parity

8 WESTERN MUSICAL STYLES

HARMONIC-EVENT PROCESSING LATENCY

MEAN0.1837 ms
P500.1069
P950.6773
P991.3116
MAX24.055

ALL EVENTS < 1 SECOND

Accelerated deterministic replay; processing latency, not wall-clock performance duration.

MAXIMUM SNAPSHOT AGE / ACCEPTANCE LIMIT

EEG
29 / 50 ms
BEHAVIOR
137 / 250 ms
CARDIO
851 / 1,000 ms

Snapshot age is the time between the latest modality sample and the harmonic-event timestamp; each stream must remain below its modality-specific limit.

WHAT THIS VALIDATES

  • ✓ event timing
  • ✓ cross-modal linkage
  • ✓ modality buffering
  • ✓ exportable event records
  • ✓ processing latency

Synthetic streams do not validate biological effects, hardware drift, packet loss, or physiological causality.

100% RECORD COMPLETENESS