SensorSynthFM:
FM Shaped by the Room
Research and process
The research trail is part of the project, not an appendix.
Start with the desk-research synthesis, then use the process log if you want the build trail behind the current prototype.
The Idea
Most synth interfaces ask the player to manage a dense parameter editor. SensorSynthFM starts from a narrower bet: an iPad can become a small FM instrument that listens to touch, motion, surface vibration, and room energy, then turns those signals into timbre.
That only works if the mapping is legible. Passive sensing without explanation feels random. FM without explanation feels chaotic. Passive sensing plus FM without explanation feels broken.
The current build focuses on a single proof screen, not a finished product. It shows live sensor values, mapped FM values, toggles, sound controls, and diagnostic readouts on one iPad surface so the cause and effect can be judged directly.
The Current Thesis
The first physical iPad pass changed the shape of the work. Gyroscope-to-ratio and accelerometer-to-modulation-index both produced audible, useful change. The microphone path did not prompt clearly and did not respond enough to trust. The fixed routing model was the weak part.
The design problem is now control authority. If a sensor is driving a slider, the player has to know that before the slider moves. If the player grabs a value manually, the screen has to show whether the player owns it, whether the sensor owns it, or whether control is returning to the room.
That is the thesis demo now: make a sensor-shaped FM sound feel intentional enough that a musician can explain what changed, hear why it changed, and recover when the mapping gets confusing.
What Exists
AudioKit FM lane
The active prototype uses AudioKit with a single FM oscillator lane. The exposed controls are carrier frequency, modulation index, modulator ratio, and amplitude. That is enough for the thesis demo because continuous modulation can change the character of the sound without needing a full workstation.
Live sensor mapping
The current `SensorFMBridge` maps accelerometer magnitude to modulation index, microphone amplitude to amplitude, and gyroscope Y to modulator ratio. Those mappings are visible in `SensorFMTestView`, with smoothing and per-mapping enable controls.
Scene fingerprint direction
A candidate SceneFingerprint layer is implemented for review. Its job is to turn raw sensor signals into more musical descriptors such as room energy, surface impact, vibration, brightness, and bass pressure. That layer still needs Mac, Xcode, and physical iPad validation before it becomes a public claim.
What Changed After Hardware
The first useful hardware finding was simple: sound worked. Accelerometer and gyro mappings worked. The microphone was not yet trustworthy. That is better evidence than another architecture diagram.
The second finding was more important. A sensor-driven slider that snaps back to its mapped value does not read as intelligence. It reads as a broken control. The next UI work is not decoration. It is accessibility, state legibility, and recovery.
The modulation matrix is still the right long-term direction, but it is not the next thesis-demo feature. The next step is source, destination, value, and authority, shown clearly enough that the instrument can be evaluated without narration.
Documentation and Research
The project has a dedicated documentation layer because the code, research, and design decisions were starting to drift apart. That was not a documentation problem. It was a truth problem.
Read the wiki article here: Why SensorSynthFM Needed a Wiki Before It Needed More Features.
The desk-research synthesis is here: SensorSynthFM desk research.
The build process is documented here: SensorSynthFM build process.
The code lives on GitHub. The public documentation layer is still in progress.
How It Is Being Built
I do not have a traditional software development background. The build uses AI collaborators for implementation, review, and research synthesis while I own the musical judgment, UX direction, and evidence gates.
Clarence is the working system around that collaboration. It keeps the task board, research notes, source maps, and review gates visible so the project does not turn into a pile of impressive but disconnected prototypes.
The friction matters. AI can write code quickly. It cannot decide whether a mapping feels musical, whether a public claim is earned, or whether a confusing control should be cut instead of explained harder.
What Is Still Unresolved
Three implementation gates are still open: microphone diagnostics, safe-area and slider-authority UI, and the SceneFingerprint review. Each one needs Mac, Xcode, and physical iPad validation before this becomes a stronger public case study.
No full modulation matrix exists yet. No looper or sequencer exists yet. The current work is smaller and harder: make the sensor-to-FM proof understandable enough that another person can test it.