Direct note entry and Hold are working on the iPad.
The September 20 build passed 57 Xcode tests with zero failures, and then my own test on the iPad. The clip shows a held chord, with reticles around the active touches and fixed markers for the held notes.
Recorded on the iPad, September 20.Text description
Three touch points appear on the iPad surface. Orange reticles mark the active fingers, while green edge markers show held notes continuing after release.
SensorSynthFM is an iPad FM instrument shaped by touch, device motion, microphone level, and other signals from the room. On September 5, 2026 I decided that playing notes directly came first. In the September 20 build I can start, move, hold, merge, and release notes on an iPad resting on a table, and the sensors still shape the sound. The full dated history is linked at the bottom.
The code is written with AI assistance. I set the direction and ran every test myself, on one 11-inch iPad Pro. Nobody else has tested the instrument yet.
What Works on the iPad
With Hold on, notes sustain after I lift my fingers. I can retrigger a held note without doubling it, slide a note to a new pitch, merge it into a pitch that is already held, remove one note, and turn Hold off while a finger still controls a note. Quantized and Freehand feedback stayed readable after the active reticle grew to 88 points, the label moved away from the playing arm, and the released-note ring was matched to a 350-millisecond fade.
The July 22 build passed 22 automated Xcode tests and a generic iOS build. I then signed, installed, and ran it on my 11-inch iPad Pro. When the seven-day development signing profile expired, a fresh Xcode run signed and installed the same build and it launched again.
On August 3 I ran an eight-step cognitive walkthrough on the iPad, stepping through the instrument as a first-time user would. Seven steps passed:
Play was visible.
Sound started, and the playing state made sense.
The active source and FM destination were clear.
Manual control and sensor modulation were distinguishable from each other.
Microphone response was visible and audible.
Setting a route to zero neutralized it, and restoring the amount resumed modulation.
I could explain the premise of the instrument in one sentence.
I tested each sensor mapping on the iPad. Accelerometer magnitude routed to modulation index at +40 percent produced a clear change when the device moved; table taps and stationary vibration did not produce a useful result. Gyroscope Y routed to modulator ratio at +41 percent changed timbre and apparent pitch predictably under slow tilting. With microphone amplitude routed to amplitude at +63 percent, a quiet tone stayed stable, speech raised loudness proportionally, and claps produced louder transients with a slight apparent delay.
There is also a handedness switch that moves auxiliary controls for left- and right-handed use. The switch works. No left-handed player has tried it.
What Failed on August 3
The eighth August 3 walkthrough step failed. Accelerometer modulation changed the sound with no detectable visual response. Gyroscope response was visible but too subtle to trust. That build did not adequately show what the sensors were doing to the sound.
The same walkthrough exposed a failure in routing. Selecting a route cell with a value of zero did not show the route as active. The route only became apparent after the amount moved away from zero. The interface did not explain how to create a route or how to remove one.
What Changed Because It Failed
Earlier tests on the iPad led to their own corrections. The iPad status area and gesture regions covered controls in an early layout. The first route editor collided with the iPadOS bottom gesture zone. Microphone behavior was too subtle to trust. Fixed routing hid the range of the instrument. Sensor writes made manual controls look broken. A side panel made the primary amount control too narrow, and the layout hid the ninth source entirely. Each of those failures led to a design change that is still in the September 20 build.
The August 3 walkthrough added these requirements:
Selected zero routes stay visibly selected.
The interface explains that moving the amount activates a route.
Every active route gets a visible X or remove control. Double-tap may remain a shortcut, but it cannot be the only removal method.
Sensor response is shown clearly, so the ear and the eye agree.
The September 6 repair fixed sensor response, selecting a route at zero, visible route removal, and recovery. I could follow the microphone, accelerometer, and gyroscope response and complete the route flow on the iPad. Automatic scrolling to the destination being edited is not built.
What did not change: familiar synthesis controls, amounts that run positive or negative, a visible zero state, large touch targets, and clear source-to-destination routes.
Note entry added another rule: touch feedback has to stay visible around the finger. A marker can look correct in a simulator and still disappear beneath the hand.
Requirements Under Test
I track requirements with the method in Jeremy Dick, Elizabeth Hull, and Ken Jackson, Requirements Engineering, Fourth Edition. It separates what a stakeholder needs from what the system does. Each requirement gets a source, a reason, a qualification test, and traceability from the code to the test result. A requirement passes when a person can do what it describes, whatever the code tests say. Official book page.
Seven of the requirements are listed below, with results from the August 3 eight-step walkthrough and the September 20 note-entry test.
Every enabled sensor route must make source, amount, destination, live response, and neutral zero state audible and visible.
Why it exists
Sensor modulation is the instrument's central promise. Audible change without visible cause reads as random or broken.
Test
On the iPad, isolate a route, create a source change, and identify source, destination, direction, and visible response before explanation.
Result
I could follow the microphone, accelerometer, and gyroscope response after the September 6 repair.
SSFM-SYS-002Passed, limited evidence
A selected route at zero must remain visibly selected and explain that changing the amount activates modulation.
Why it exists
A route set to zero is still selected and can still be edited.
Test
Starting at zero, identify the selected route and the activation action without prompting.
Result
After the September 6 repair, I could see the selected route at zero and how to activate it.
SSFM-SYS-003Passed, limited evidence
Every active route must provide a visible, accessibly named removal control and a straightforward recovery path.
Why it exists
A shortcut gesture may help, but it cannot be the only way to remove a route.
Test
Remove and restore a route without being told a hidden gesture.
Result
After the September 6 repair, I activated, removed, and restored a route.
SSFM-SYS-004Accepted, not yet qualified
In landscape, the modulation matrix must keep all nine current sources and eligible routes reachable, with readable source and destination labels and route cells at or above the 56-point minimum touch size.
Why it exists
Scrolling may be necessary as the matrix grows, but it cannot hide context, steal focus, or make the instrument harder to touch.
Test
On the 11-inch iPad in landscape, use scrolling as needed to verify every eligible route, readable source and destination context, and usable route cells, then confirm the most recently edited destination comes into view without sensor updates moving controls beneath an active finger.
Result
In August the ninth source required horizontal scrolling. Scrolling is permitted, but bringing the edited destination into view is not built and not every touch target has been checked.
SSFM-SYS-005Passed, limited evidence
Manual base values and live sensor contribution must remain distinct, and returning a route to zero must neutralize sensor contribution without destabilizing sound.
Why it exists
The musician needs to stay in control and be able to recover.
Test
Move a route through positive, zero, and negative amounts while comparing base and live values and checking sound stability.
Result
It passed my August 3 walkthrough.
SSFM-SYS-006Passed, limited evidence
Microphone modulation must remain visibly and audibly causal while quiet input stays stable and louder speech or transients increase response without runaway pumping for two minutes.
Why it exists
Environmental input should shape sound without taking over the instrument.
Test
Record quiet baseline, speech, and clap response on the iPad for two minutes, and judge causality, stability, pumping, delay, and musical usefulness.
Result
It passed one two-minute test. Exact calibration and transient delay remain open.
SSFM-SYS-007Passed, limited evidence
Before note-entry code is accepted, the note-entry specification must define pitch mapping, touch-to-sound timing, polyphony, multi-touch, pressure or area, glide, scale and range behavior, handedness, feedback, accidental activation, recovery, accessibility, and what is out of scope.
Why it exists
SensorSynthFM must become a playable instrument, and each part of note entry needs a deliberate decision.
Test
Review every named item, its source, edge cases, rejected alternatives, and iPad acceptance test before coding.
Result
On September 20, Hold, sliding a note to a new pitch, merge, selective removal, high-at-top pitch mapping, and Quantized and Freehand feedback passed on the iPad. Left-handed play by a left-handed player and ten notes at once with the sensors running are untested.
Six of the seven passed in my own testing. One is not yet tested, because automatic scrolling to the edited destination is not built and the matrix has not been fully checked.
Design iteration · September 5, 2026
September 5: note entry first
A twenty-question, AI-assisted design interview returned the project to its missing core: playing notes directly on the tabletop iPad. The next step was starting, holding, changing, and releasing notes reliably, then playing phrases and chords while the existing environmental modulation stayed available. The full capture, sequencing, or patch-generation system would come later.
The handedness switch did not come from this interview. Earlier requirements already specified a mirrored control area for the off hand and distinguished control touches from note touches. The interview kept that design and the target of ten independently sounding notes. That target had not been tested on the iPad.
Ableton's established interaction patterns stayed the reference wherever they apply. For capture, that means keeping recent playing, using the existing musical context when there is one, and letting the player change the selected phrase and loop boundaries. Applying that to continuous environmental signals still needed design and testing on the iPad.
Longer-term ideas were kept for later: patches seeded deliberately from the environment, patch profiles held to musical ranges, optional regeneration with parameter locks and Undo, live environmental modulation, and editable captured modulation. None of them had to exist before the instrument was playable.
The interview recorded what I wanted to build first. The next test was playing the note surface on the iPad and watching where it broke.
The interview also corrected the matrix requirement. Scrolling is permitted, as long as every eligible route stays reachable with readable source and destination labels and usable touch targets. The destination being edited should come into view without sensor updates moving controls beneath an active finger. That last behavior was not built.
Sessions With Other Players
The next step is five to seven moderated sessions of 30 to 45 minutes with people who did not design the interface, including at least one left-handed participant where possible. Recruitment was underway in November 2026. Each session needs informed consent, a fixed task script, observation notes, and a list of changes made in response. The sessions will test whether another musician can discover note entry, pitch movement, Hold, chord editing, route controls, sensor influence, handedness, and recovery without coaching.
I may also visit an Apple Store to watch how people approach display iPads. That visit is optional and separate from the sessions, and I would not observe formally without a settled method and any store permission required.
Chord-name feedback for learners is a possible later requirement.
Links
Full build history, the dated log of every build, failure, and correction
Desk research, the background work on FM synthesis and touch instruments