SensorSynthFM:
FM Shaped by the Room
Physical build · September 20, 2026
The instrument can finally be played.
The current review branch starts, moves, holds, merges, and selectively releases notes directly on the iPad. The loop below came from the physical build. It is not a browser mockup.
Text description
Three touch points appear on the iPad surface. Orange reticles mark the active fingers, while green edge markers show held notes continuing after release.
Research and process
The research trail is part of the project, not an appendix.
Start with the dated build history. It is the primary record for the current prototype, its physical evidence, its failures, and what happens next. Use the desk-research synthesis for the source trail behind those decisions.
The Idea
Most synth interfaces ask the player to manage a dense parameter editor. SensorSynthFM starts from a narrower bet: an iPad can become a small FM instrument that listens to touch, motion, surface vibration, and room energy, then turns those signals into timbre.
That only works if the mapping is legible. Passive sensing and FM without explanation make the instrument feel random.
The current branch centers a direct-playing Performance surface while keeping the modulation Matrix available. Touch enters notes in Quantized or Freehand mode while passive sensing remains subordinate to note and pitch control. It is a bounded instrument proof, not a finished product.
The Current Thesis
Sensor modulation gave the project a point of view, but it did not make the prototype an instrument. The missing core was direct playing. The September 20 branch puts note entry back at the center: touch starts a voice, vertical movement changes pitch, multitouch makes chords, and Hold lets those chords remain editable after the fingers leave the glass.
Physical use changed the feedback while I was playing. A 38-point touch halo disappeared beneath a fingertip.
The thesis demo now has a playable center. The next question is whether another musician can understand it without me standing nearby.
What Exists
AudioKit FM lane
The current note-entry branch uses a fixed ten-slot AudioKit voice bank. Each touch owns its pitch while global FM controls and passive sensor input continue to shape timbre. September 7 owner practice demonstrated five stable sounding notes and ten simultaneous contacts. That does not qualify ten stable voices under combined sensor, FM, audio, and touch load. Maximum stable polyphony remains an open hardware benchmark.
Native note entry and Hold
Quantized and Freehand modes share the same landscape surface. Active reticles follow the finger, labels show note and frequency, Freehand adds signed cents, and fixed edge markers show unattended held notes. Hold supports retriggering, retargeting, occupied-pitch merge, selective removal, and a full release path.
Live sensor mapping
The current mapping sends accelerometer magnitude to modulation index, microphone amplitude to amplitude, and gyroscope Y to modulator ratio. The sensor test view shows those mappings with smoothing and per-mapping enable controls.
Scene description direction
A candidate scene-description layer is implemented in a version under review. Its job is to turn raw sensor signals into descriptors such as room energy, surface impact, vibration, brightness, and bass pressure. Xcode tests and a generic iOS build passed, but the musical meaning of those descriptors still needs physical evaluation. Code existence is not the same as a validated interaction.
What Changed After Hardware
Listening on a real iPad ranked the gyroscope first and the accelerometer second. The microphone moved from subtle and untrusted to causal and stable only after dedicated diagnostics. That was better evidence than another architecture diagram.
A touch-first matrix replaced the hard-coded routing model with nine sources, four targets, and 36 bipolar route cells. I confirmed signing, installation, and launch on the physical device. The first route editor sat too close to the system gesture zone and failed on hardware. A repaired candidate later passed a bounded owner walkthrough for route activation, removal, and recovery. Outside-user discoverability and automatic destination focus remain open.
The latest pass moved the project from routing an oscillator to playing an instrument. The review branch passed 57 Xcode tests after two test-contract failures were repaired. I then verified the Hold lifecycle and touch feedback on the iPad. That evidence is still maker-operated. The next pass belongs to people who did not build it.
Documentation and Research
The project has a dedicated documentation layer because the code, research, and design decisions were starting to drift apart.
Read the wiki article here: Why SensorSynthFM Needed a Wiki Before It Needed More Features.
The desk-research synthesis is here: SensorSynthFM desk research.
The complete dated build history, including archived Kanban screenshots and open hardware gates, is here: SensorSynthFM build history.
The code lives on GitHub. The public documentation layer is still in progress. The related Ableton evaluation suite shows the expert-tool research thread behind this creative-software work.
How It Is Being Built
I do not have a traditional software development background. The build uses AI collaborators for implementation, review, and research synthesis while I own the musical judgment, UX direction, and evidence gates.
Clarence is the working system around that collaboration. It keeps the task board, research notes, source maps, and review gates visible so the project does not turn into a pile of impressive but disconnected prototypes.
What Is Still Unresolved
Now
The September 20 build is not in the public app yet. September 7 owner practice demonstrated five stable sounding notes and ten simultaneous contacts. Sensor response, selected-zero activation, route removal, and Matrix round-trip also passed within that owner boundary. Maximum stable polyphony under combined load, automatic destination focus, left-handed-use evidence, and outside-user comprehension remain open.
Later
The next evidence gate is five to seven consented, moderated sessions with people who did not design the interface, including a left-handed participant where possible. Chord-name learning feedback is recorded as a candidate requirement, not current scope. The looper, sequence editor, patch generation, presets, MPE, AUv3, and Ableton integration remain parked outside the verified capstone slice. Participant evidence will guide later scope; it does not automatically commit those expansions.