Active build·February to November 2026·Updated November 15, 2026
SensorSynthFM build history
Testing on the iPadSensor-to-FM mappingTouch-first interaction design
Tested on the iPad · September 20
Note entry works on the iPad.
In the September 20 build, notes start, move, hold, retrigger, merge, and release on the iPad. I adjusted the reticles, labels, held-note markers, and release fade after watching the screen under my own hand.
From the September 20 build. Three active touches form a held chord; the orange reticles show current contact while green edge markers keep released notes under Hold.Text description
Three touch points appear on the iPad surface. Orange reticles mark the active fingers, while green edge markers show held notes continuing after release.
What This Page Is
This is the dated build history for SensorSynthFM, an FM synthesizer driven by an iPad's onboard sensors.
It draws on the Git history, my research and design notes, my tests on the iPad, and the project task board. AI collaborators help with implementation, review, and research synthesis. I make the musical and design decisions and do the testing.
Where It Stood on November 15
Works
Three pages work on the iPad
The November 15 build has three pages: Performance for playing notes, Matrix for routing sensors and touch to the sound, and Instrument for envelope, filter, echo, and reverb. It reaches the iPad through TestFlight, the Apple service for distributing test builds. Note entry and Hold work as they did on September 20. Sliding a finger left or right now changes the tone of that note.
Untested
Nobody else has played it
Every test so far is my own, on one 11-inch iPad Pro. Moving four or five fingers at once can cut off every note, because iPadOS takes that motion for a system gesture. Guided Access, which locks the iPad to one app, prevents it. Still untested: ten notes sounding at once with the sensors running, and play by a left-handed player. Automatic scrolling to the destination being edited is not built.
Next
Next: sessions with other people
The next step is five to seven moderated sessions with consenting people who did not design the interface. Recruitment is underway and no session has run. The build stays as it is until then. I may also visit an Apple Store to watch how people approach display iPads. That visit is optional and separate from the sessions. Looping, sequencing, patch generation, presets, MPE, AUv3, and Ableton integration are not part of the capstone. A loop recorder modeled on Capture in Ableton Live is the first addition planned after the sessions.
Design note · July 17, 2026
The matrix is going to grow.
The matrix had nine sources and four targets when this note was written. It has to take on more as the engine grows: additional operators, envelopes, LFOs, macros, effects, and destinations. It cannot do that by shrinking touch targets.
After the July 22 test on the iPad, I adopted this study as the working layout. As of July 22, its matrix-first hierarchy, grouped route inspector, and compact right rail were not yet built in the app. The plan kept the existing behavior and the large touch controls.
Interactive design study. Select routes, drag the bipolar amount, use the one-percent nudges, reset through the signed value, switch handedness, and compare the layout at the time with the revision. This is a browser study, separate from the app.
These are screenshots of the project task board, reduced to milestone titles, status, and work category. The latest snapshot is from September 20. At that point the board held 2 ready, 15 todo, 3 needs-decision, 67 done, and 25 scheduled cards.
First public board
First board
Open full-size screenshot ↗This board set up the capstone work, the goal of running the app on an iPad, the source review, and the first public process page. The screenshot cuts off the end of the Done column.
Layout study
Layout study published
Open full-size screenshot ↗The layout study was published while the route-control changes on the iPad were still open.
Route controls
Route controls pass on the iPad
Open full-size screenshot ↗The July 18 board shows the accepted route controls, keeps the layout decision open, and adds the task of defining note entry for the Performance page.
Open full-size screenshot ↗The September 5 board records the decision to build note entry first. Environmental modulation stayed available. Capture, sequencing, patch generation, and the ten-note test on the iPad were still open.
Last board snapshot
Note entry and Hold work on the iPad
Open full-size screenshot ↗The September 20 board records note entry and Hold working on the iPad. The sensor display was repaired on September 6 in my own testing. Sessions with other people, the ten-note test, and left-handed play were still open. The board has not tracked the build since October. Later work is in the build history below.
Build History by Date
February 2026
FoundationVerified
Xcode project created
I created the Xcode project. The first commit set up the application shell.
Commit 3fd6cf8
Xcode project created
InteractionVerified
First interface mockups
SwiftUI mockups and project documentation gave the instrument its first visible interaction model. Nothing had run on a device yet.
Commit c01d1d6
SwiftUI mockups
Project documentation
SensorsVerified
Sensor input added
The README set out the working project description. The app began asking for microphone permission, and a memory leak in the FFT audio-analysis code was fixed. Sensor input now ran as code.
Commits 7715ce4 and 2d21827
Microphone permission
FFT leak fixed
AudioVerified
Audio engine made sound
A sine-wave test showed that the audio engine could run, before any FM was added.
Commit a39ee90
Sine-wave test
March 2026
DesignDecision
Design documents expanded
Project notes and design documents were expanded before the synthesis engine changed. They began to describe how the instrument should behave.
Commit 463a355
Design documents
Project notes
AudioVerified
FM synthesis replaced the sine wave
The sine wave was replaced by an FM oscillator from AudioKit, an open-source audio library. Carrier frequency, modulation index, modulator ratio, and amplitude became the first synthesis controls.
Commit 33009ff
AudioKit FM oscillator
Four exposed parameters
IntegrationVerified
Sensors connected to the FM engine
Accelerometer magnitude, microphone amplitude, and gyroscope Y were connected to FM parameters in one test view. It was the first place to judge whether a physical signal caused an audible change.
Commit 9be8498
Sensors wired to FM
Integrated test view
April 2026
DocumentationPublished
Wiki started
Public project documentation was added and a project wiki was started. The code, README, design documents, and longer plans had begun describing different versions of the instrument.
Commit 02fd220
Public project docs
Wiki started
AuditDecision
Decision: run the app on an iPad before adding features
The main repository and a duplicate starter repository were audited again. I decided to get the existing integrated view running on an iPad before adding features.
Repository audit
Duplicate starter identified
Checklist for the iPad run
InteractionDecision
Larger interaction design documented and set aside
Touch-native looping, a sequence and edit grammar, rhythmic capture, and a three-page architecture were documented. I set them aside until the app ran on a device.
Looping direction
Three-page architecture
iPad run still first
June 2026
RestartDecision
Work restarted from the device
The restart plan narrowed the work to getting the app running on an iPad. GitHub stayed the home of the code, and Google Drive stayed an archive.
iPad run first
Main repository restored
Drive kept as archive
CapstoneVerified
Research linked to the task board
The desk-research inventory and the open build questions were linked to the project task board. The board was set up to track research, implementation, review, and work set aside for later.
Public source inventory
Build questions linked
Capstone board set up
July 2026
HardwareVerified
Integrated app ran on an iPad
The app was set to open on the sensor test view. I ran the integrated build on an iPad and confirmed sound output plus audible accelerometer and gyroscope mappings. The microphone seemed active, but the response was too subtle to trust.
Commit 6a7683f
Sound passed
Motion mappings passed
PublicationPublished
Desk research and process page published
The desk-research page and the first process page were published with a screenshot of the 33-card task board. The board listed capstone work, open questions, source review, ideas set aside, and a privacy rule for usability video.
33-card board
Desk research published
Process page published
ResearchPublished
Source list published
Documentation in the repository was corrected to match the code, and the plan for the scene-description work was clarified. A source atlas built from 319 reviewed public URLs was published.
Documentation corrected
319 public URLs
Source atlas published
EvaluationDecision
Listening test reordered the mappings
Listening on the iPad, I ranked the gyroscope first and the accelerometer second. The microphone was working, but it needed a different interpretation. Accessibility requirements, an interaction model, a cognitive walkthrough, a moderated usability script, and a way to trace findings to decisions were then documented.
Gyroscope ranked first
Accessibility requirements
Evaluation plan
ImplementationOpen
Three repairs entered review
Microphone diagnostics, repairs to the safe-area layout and to sensor updates overriding manual controls, and scene-description work moved into review. The public project page was corrected at the same time so it no longer presented fixed routing as the finished direction.
Microphone diagnostics
Manual-control repair
Scene-description review
ScopeDecision
Demo narrowed to a stationary iPad
The focus narrowed to one two-minute environmental-modulation demo with the iPad resting on a table. Moving the whole device stopped being the main performance gesture. The room and the table surface modulate the sound.
Two-minute demo
Stationary iPad
Environmental modulation
DiagnosticsVerified
Microphone and scene description tested on the iPad
I assembled microphone diagnostics and a scene-description test in a version under review. Xcode tests and a generic iOS build passed. I cleared signing and confirmed that the microphone responded on the iPad.
Xcode tests passed
Generic iOS build passed
Microphone responded on iPad
InteractionVerified
Fixed mapping replaced by a matrix
A touch-first modulation matrix was implemented with nine sources, four targets, 36 bipolar route cells, editable base values, and migrated defaults. Seventeen Xcode tests and the generic iOS build passed. I signed, installed, and launched the build on the iPad. How each route responds to touch remained under review.
9 sources by 4 targets
17 tests passed
Launched on iPad
HardwareVerified
Microphone passed; route slider failed
I ran a two-minute stationary microphone test in a room measuring roughly 60 dB. The route clearly drove the sound, was stable, and pumped only slightly, although exact calibration remained open. The same test exposed a blocking defect: the route slider sat in the iPadOS bottom gesture zone.
Two-minute test on iPad
Microphone route drove the sound
System-gesture conflict
ImplementationVerified
First route-editor repair entered review
A route inspector was built that sat on the right by default, mirrored for left-handed use, moved diagnostics behind a disclosure control, and added a center-zero control with one-percent nudges and a zero detent. Source checks passed. The next test on the iPad rejected its fixed side rail.
Route inspector implemented
Source checks passed
iPad test next
HardwareOpen
Side inspector rejected on the iPad
The repaired build ran on the 11-inch iPad with the selected route fixed at the right edge. The 300-point rail made the primary bipolar slider too narrow for confident touch control and left most of the right column unused. The next layout had to put a wider slider directly below the matrix and reserve the right rail for secondary elements.
Rejected layout. The route amount control occupied a narrow right-side inspector, but testing on the iPad showed that the primary slider needs a wider strip below the matrix. The iPad status bar is redacted.
iPad screenshot
Side inspector rejected
Full-width slider required
InteractionDecision
Wider editor ran on the iPad; grid still unfinished
The selected-route editor moved below the matrix. Twenty Xcode tests, a generic iOS build, and simulator geometry checks passed before I signed, installed, and launched the build on the 11-inch iPad without errors. On the iPad the revision was better, but it still underused the available space. The dedicated ZERO control wasted room, the plus/minus nudge order had to reverse, and the grid should move toward an Ableton-like layout over time.
Commit c5d3472
20 Xcode tests passed
Launched on iPad; layout still open
EvidenceDecision
Screenshot method corrected
A check confirmed that Xcode 26.3 still saw the iPad as paired. The working method was to take each screenshot by hand at full resolution in Xcode, under Window > Devices and Simulators, then review it for privacy before publication. Apple's newer Device Hub remote-control instructions did not apply to the installed version of Xcode. Automated screenshot capture was proposed but not built.
Xcode 26.3 pairing rechecked
Manual screenshot method corrected
Automated capture not built
DesignPublished
Matrix layout study published
The rejected layout became an interactive study comparing the layout at the time with a revision. The revision kept all nine sources visible, gave the selected route a wide bipolar strip below the grid, reversed the nudge order, made the signed amount the reset target, and reserved the right rail for secondary context. The matrix also had to be able to grow past nine sources and four targets without shrinking touch targets or burying route editing.
Interactive layout study
All nine sources visible
Matrix growth still an open design problem
HardwareVerified
Route controls passed on the iPad
The dedicated ZERO control was removed, the default nudge order was reversed, and the signed amount became the reset target. Twenty Xcode tests, a generic iOS build, installation, and launch passed before I confirmed all three controls worked on the iPad. The larger layout from the web study was still not built in the app. The task board kept that decision open and listed the next steps in order: full sensor-to-sound integration, defining note entry, a walkthrough, checking which mappings hold up, and the capstone summary.
Commit 635ab09
20 Xcode tests passed
Three control checks passed on iPad
July 18 board archived
HardwareVerified
Integrated sensor build ran on the iPad
An integration fix replaced separate microphone reads with one consistent reading and moved sensor startup into the app's own startup. Twenty-two Xcode tests and the generic iOS build passed before Xcode signed, installed, and launched it on the 11-inch iPad. I confirmed that sound, accelerometer and gyroscope response, and microphone response all worked as before. The screenshot below shows the layout.
Accepted build, unfinished layout. The ninth source needs a horizontal swipe, part of the context rail falls below the fold, and the selected-route controls span more space than they need. The iPad status bar is redacted.
Commit 48b6b3d
22 Xcode tests passed
Signed, installed, and launched on iPad
Sound, motion, and microphone rechecked
DesignDecision
Decision: adopt part of the web study layout
I compared the iPad screenshot with the 1210 by 834 public study at the same scale. The app screen kept large, reliable touch controls, but it clipped the ninth source, left context below the fold, and separated the signed reset value from its nudge controls. I decided to adopt part of the study. Existing modulation behavior stayed. The matrix-first hierarchy, grouped route inspector, compact right rail, reset explanation, and diagnostics behind a disclosure control moved into the next layout brief. That layout had not been built or tested on the iPad.
Screenshot compared with the study at the same scale
Partial adoption chosen
App layout still open
August 2026
EvaluationVerified
Walkthrough passed seven of eight steps
The eight-step cognitive walkthrough on the iPad passed seven steps. Play was visible; sound and playing state made sense; the active source and destination were clear; manual control and sensor modulation were distinguishable; microphone response was visible and audible; zero neutralization and recovery worked; and the instrument premise could be stated in one sentence.
Seven of eight walkthrough steps passed
Walkthrough on the iPad
Zero neutralization and recovery passed
DesignOpen
Sensor display and route discovery failed
Accelerometer movement changed sound with no detectable visual response, and gyroscope visual response was too subtle to trust. Route discovery also failed because a selected zero route did not look active and creation and removal were unexplained. The changes required were a visible selected-zero state, an explanation of how to activate a route, a visible remove control, a clear display of sensor response, all nine sources visible without shrinking touch targets, and testing by other people.
Accelerometer sound change without detectable visual response
Gyroscope visual response too subtle
Selected zero route did not look active
ProcessDecision
Requirements tracking began
I began tracking requirements by ID, each with its own test and a status of passed, failed, or not yet tested. Seven are published on the current-build page.
Seven requirements published
Tasks on the board name their requirement and test
Passed, failed, or not yet tested
September 2026
DesignDecision
September 5: note entry first
A twenty-question, AI-assisted design interview returned the project to its missing core: playing notes directly on the tabletop iPad. The next step was starting, holding, changing, and releasing notes reliably, then playing phrases and chords while existing environmental modulation stayed available. The target of ten independently sounding notes and the handedness design had not been tested on the iPad. The matrix rule changed to permit scrolling when every eligible route stays reachable with readable labels and usable touch targets.
AI-assisted design interview
Note entry chosen as the first step
Ten-note test on the iPad still open
BuildVerified
September 20: note entry and Hold work on the iPad
The note-entry build passed 57 Xcode tests and a focused test on the iPad. Notes started, moved, retriggered, merged, stayed under Hold, and released without doubled pitches or stuck voices. Playing on the iPad changed the feedback: the active reticle grew to 88 points, labels moved away from the playing arm, and released notes kept an orange dashed ring through a 350-millisecond fade. Ten notes sounding at once had not been tested.
Frame from the September 20 note-entry test on the iPad. Three active touches are visible on the surface while three held notes remain marked at the pitch edge.
57 Xcode tests passed
Hold passed on iPad
Quantized and Freehand feedback reviewed on iPad
Not yet tested by anyone else
October 2026
BuildVerified
Instrument page added
A third page, Instrument, joined Performance and Matrix. It holds sliders for the fade in and fade out of each note (attack, decay, sustain, and release) and a filter that works on each note separately. Slider touch areas were corrected so the whole control responds. I tried the envelope and filter on the iPad.
Instrument page
Envelope and filter for each note
Tried on the iPad by me only
ScopeDecision
Session build limited to three pages
I set the build for the sessions as the three pages that already worked on the iPad: Performance, Instrument, and Matrix. The sequencer, capture, presets beyond one starting sound, undo, and saving were put off until after the capstone. The build gained a fixed starting sound, a RELEASE ALL button that stops every note, and a RESET START button that restores the starting sound.
Three pages in the session build
Sequencer and presets deferred
RELEASE ALL and RESET START added
HardwareVerified
Corrections after a rehearsal on the iPad
I rehearsed on the iPad. The rehearsal led to corrections in how playing state is shared between pages, in manual tuning, and in how notes release. A hidden audition control was removed, and the tuning reset was made easier to see.
Rehearsal on the iPad
Hidden audition control removed
Tuning reset made visible
November 2026
InteractionVerified
Left-right finger position became a modulation source
TOUCH X, the left-right position of each finger, joined the matrix as a tenth source. It works on each note separately, so two fingers can shape their own notes differently. Double-tapping a route cell now clears it. Audio routing controls were added for recording the sessions. Builds began reaching the iPad through TestFlight, the Apple service for distributing test builds.
Tenth matrix source
Double-tap clears a route
Builds delivered through TestFlight
HardwareOpen
Four or five moving fingers cut off every note
Playing with four or five fingers and moving them cut off every note. iPadOS takes that motion for a system gesture and cancels the touches, and it did so with the multitasking gesture setting turned off. I found no supported way for the app to prevent it. With Guided Access on, which locks the iPad to one app, the cutoff stopped, so the sessions will run that way.
Cutoff happened on the iPad
iPadOS system gesture
Guided Access prevents it
BuildVerified
November 15: navigation, a new filter, echo and reverb
Three builds went to the iPad in one evening. The first kept PERFORMANCE, MATRIX, and INSTRUMENT in the side rail on every page and removed duplicate buttons from the top of the Matrix page. The second replaced the low-pass filter with a ladder filter, a design with a stronger resonant peak, because resonance on the old filter had no audible effect. It also changed the starting sound so that sliding a finger left or right opens the filter and brightens the tone. The third added echo and reverb shared by every note, with three sliders on the Instrument page. I played each build on the iPad and kept all three changes. One automated interface test fails intermittently, as it did on the October 12 build.
Three builds through TestFlight
Played on the iPad
One interface test fails intermittently
Not yet tested by anyone else
ScopeDecision
Build held for the sessions
I decided to make no further major changes before the sessions. The first addition planned afterward is a loop recorder modeled on Capture in Ableton Live. It listens all the time, and one press turns the last phrase played into a loop.