SensorSynthFM:
FM Shaped by the Room
Research and process
The research trail is part of the project, not an appendix.
Start with the dated build history. It is the primary record for the current prototype, its physical evidence, its failures, and what happens next. Use the desk-research synthesis for the source trail behind those decisions.
The Idea
Most synth interfaces ask the player to manage a dense parameter editor. SensorSynthFM starts from a narrower bet: an iPad can become a small FM instrument that listens to touch, motion, surface vibration, and room energy, then turns those signals into timbre.
That only works if the mapping is legible. Passive sensing without explanation feels random. FM without explanation feels chaotic. Passive sensing plus FM without explanation feels broken.
The current build focuses on a single proof screen, not a finished product. It shows live sensor values, mapped FM values, toggles, sound controls, and diagnostic readouts on one iPad surface so the cause and effect can be judged directly.
The Current Thesis
Physical testing changed the shape of the work. Gyroscope and accelerometer mappings produced useful audible change. Dedicated diagnostics then moved the microphone from “seems active” on July 1 to a causal, stable two-minute route on July 15. The fixed routing model was still the weak part.
The modulation matrix made that routing visible, then physical use exposed the next defect. The selected-route slider sat inside the iPadOS bottom gesture zone. A local adaptive inspector now moves with handedness, hides diagnostics until requested, and gives zero a visible home, but it has not passed Mac, Xcode, simulator, or physical-iPad validation.
That is the thesis demo now: make a sensor-shaped FM sound feel intentional enough that a musician can explain what changed, hear why it changed, and recover when the mapping gets confusing. The interface has to survive the hand.
What Exists
AudioKit FM lane
The active prototype uses AudioKit with a single FM oscillator lane. The exposed controls are carrier frequency, modulation index, modulator ratio, and amplitude. That is enough for the thesis demo because continuous modulation can change the character of the sound without needing a full workstation.
Live sensor mapping
The current `SensorFMBridge` maps accelerometer magnitude to modulation index, microphone amplitude to amplitude, and gyroscope Y to modulator ratio. Those mappings are visible in `SensorFMTestView`, with smoothing and per-mapping enable controls.
Scene fingerprint direction
A candidate SceneFingerprint layer is implemented on a review branch. Its job is to turn raw sensor signals into descriptors such as room energy, surface impact, vibration, brightness, and bass pressure. Xcode tests and a generic iOS build passed, but the musical meaning of those descriptors still needs physical evaluation. Code existence is not the same as a validated interaction.
What Changed After Hardware
Listening on a real iPad ranked the gyroscope first and the accelerometer second. The microphone moved from subtle and untrusted to causal and stable only after dedicated diagnostics. That was better evidence than another architecture diagram.
A touch-first matrix replaced the hard-coded routing model with nine sources, four targets, and 36 bipolar route cells. James confirmed signing, installation, and launch on the physical device. The same pass exposed a route control too close to the system gesture zone. The hardware accepted the matrix and rejected the editor.
The next step is narrow: validate the adaptive route inspector on the Mac, in Xcode and the simulator, then under a hand on the iPad. Source, destination, amount, zero, and authority have to remain legible without narration.
Documentation and Research
The project has a dedicated documentation layer because the code, research, and design decisions were starting to drift apart. That was not a documentation problem. It was a truth problem.
Read the wiki article here: Why SensorSynthFM Needed a Wiki Before It Needed More Features.
The desk-research synthesis is here: SensorSynthFM desk research.
The complete dated build history, including archived Kanban screenshots and open hardware gates, is here: SensorSynthFM build history.
The code lives on GitHub. The public documentation layer is still in progress.
How It Is Being Built
I do not have a traditional software development background. The build uses AI collaborators for implementation, review, and research synthesis while I own the musical judgment, UX direction, and evidence gates.
Clarence is the working system around that collaboration. It keeps the task board, research notes, source maps, and review gates visible so the project does not turn into a pile of impressive but disconnected prototypes.
The friction matters. AI can write code quickly. It cannot decide whether a mapping feels musical, whether a public claim is earned, or whether a confusing control should be cut instead of explained harder.
What Is Still Unresolved
The public main branch still carries the July 1 physical-activation baseline. The physically reviewed microphone, SceneFingerprint, safe-area, and modulation-matrix work is ahead of it, and the adaptive route inspector is farther ahead on an unmerged local branch. That branch has passed source-level checks only.
Mac and Xcode builds, simulator runs, and physical-iPad validation for the route inspector remain open. Microphone calibration and the best matrix amount are not established. The looper, sequencer, four-operator engine, presets, MPE, AUv3, and Ableton integration remain parked until the stationary environmental-modulation proof is coherent.