Skip to main content
← SensorSynthFM
Active buildFebruary to November 2026Updated November 15, 2026

SensorSynthFM build history

Testing on the iPadSensor-to-FM mappingTouch-first interaction design

Tested on the iPad · September 20

Note entry works on the iPad.

In the September 20 build, notes start, move, hold, retrigger, merge, and release on the iPad. I adjusted the reticles, labels, held-note markers, and release fade after watching the screen under my own hand.

From the September 20 build. Three active touches form a held chord; the orange reticles show current contact while green edge markers keep released notes under Hold.
Text description

Three touch points appear on the iPad surface. Orange reticles mark the active fingers, while green edge markers show held notes continuing after release.

What This Page Is

This is the dated build history for SensorSynthFM, an FM synthesizer driven by an iPad's onboard sensors.

It draws on the Git history, my research and design notes, my tests on the iPad, and the project task board. AI collaborators help with implementation, review, and research synthesis. I make the musical and design decisions and do the testing.

Where It Stood on November 15

Works

Three pages work on the iPad

The November 15 build has three pages: Performance for playing notes, Matrix for routing sensors and touch to the sound, and Instrument for envelope, filter, echo, and reverb. It reaches the iPad through TestFlight, the Apple service for distributing test builds. Note entry and Hold work as they did on September 20. Sliding a finger left or right now changes the tone of that note.

Untested

Nobody else has played it

Every test so far is my own, on one 11-inch iPad Pro. Moving four or five fingers at once can cut off every note, because iPadOS takes that motion for a system gesture. Guided Access, which locks the iPad to one app, prevents it. Still untested: ten notes sounding at once with the sensors running, and play by a left-handed player. Automatic scrolling to the destination being edited is not built.

Next

Next: sessions with other people

The next step is five to seven moderated sessions with consenting people who did not design the interface. Recruitment is underway and no session has run. The build stays as it is until then. I may also visit an Apple Store to watch how people approach display iPads. That visit is optional and separate from the sessions. Looping, sequencing, patch generation, presets, MPE, AUv3, and Ableton integration are not part of the capstone. A loop recorder modeled on Capture in Ableton Live is the first addition planned after the sessions.

Design note · July 17, 2026

The matrix is going to grow.

The matrix had nine sources and four targets when this note was written. It has to take on more as the engine grows: additional operators, envelopes, LFOs, macros, effects, and destinations. It cannot do that by shrinking touch targets.

After the July 22 test on the iPad, I adopted this study as the working layout. As of July 22, its matrix-first hierarchy, grouped route inspector, and compact right rail were not yet built in the app. The plan kept the existing behavior and the large touch controls.

Interactive design study. Select routes, drag the bipolar amount, use the one-percent nudges, reset through the signed value, switch handedness, and compare the layout at the time with the revision. This is a browser study, separate from the app.

Open the full interactive study ↗

Task Board Snapshots, by Date

These are screenshots of the project task board, reduced to milestone titles, status, and work category. The latest snapshot is from September 20. At that point the board held 2 ready, 15 todo, 3 needs-decision, 67 done, and 25 scheduled cards.

First public board

First board

SensorSynthFM task board, July 1, 2026Open full-size screenshot ↗
This board set up the capstone work, the goal of running the app on an iPad, the source review, and the first public process page. The screenshot cuts off the end of the Done column.

Layout study

Layout study published

SensorSynthFM task board, July 17, 2026Open full-size screenshot ↗
The layout study was published while the route-control changes on the iPad were still open.

Route controls

Route controls pass on the iPad

SensorSynthFM task board, July 18, 2026Open full-size screenshot ↗
The July 18 board shows the accepted route controls, keeps the layout decision open, and adds the task of defining note entry for the Performance page.

Requirements method

Requirements tracking begins

SensorSynthFM task board, August 10, 2026Open full-size screenshot ↗
The board on August 10, when requirements tracking began.

Note-entry decision

Note entry comes first

SensorSynthFM task board, September 5, 2026Open full-size screenshot ↗
The September 5 board records the decision to build note entry first. Environmental modulation stayed available. Capture, sequencing, patch generation, and the ten-note test on the iPad were still open.

Last board snapshot

Note entry and Hold work on the iPad

SensorSynthFM task board, September 20, 2026Open full-size screenshot ↗
The September 20 board records note entry and Hold working on the iPad. The sensor display was repaired on September 6 in my own testing. Sessions with other people, the ten-note test, and left-handed play were still open. The board has not tracked the build since October. Later work is in the build history below.

Build History by Date

February 2026

FoundationVerified

Xcode project created

I created the Xcode project. The first commit set up the application shell.

  • Commit 3fd6cf8
  • Xcode project created

InteractionVerified

First interface mockups

SwiftUI mockups and project documentation gave the instrument its first visible interaction model. Nothing had run on a device yet.

  • Commit c01d1d6
  • SwiftUI mockups
  • Project documentation

SensorsVerified

Sensor input added

The README set out the working project description. The app began asking for microphone permission, and a memory leak in the FFT audio-analysis code was fixed. Sensor input now ran as code.

  • Commits 7715ce4 and 2d21827
  • Microphone permission
  • FFT leak fixed

AudioVerified

Audio engine made sound

A sine-wave test showed that the audio engine could run, before any FM was added.

  • Commit a39ee90
  • Sine-wave test

March 2026

DesignDecision

Design documents expanded

Project notes and design documents were expanded before the synthesis engine changed. They began to describe how the instrument should behave.

  • Commit 463a355
  • Design documents
  • Project notes

AudioVerified

FM synthesis replaced the sine wave

The sine wave was replaced by an FM oscillator from AudioKit, an open-source audio library. Carrier frequency, modulation index, modulator ratio, and amplitude became the first synthesis controls.

  • Commit 33009ff
  • AudioKit FM oscillator
  • Four exposed parameters

IntegrationVerified

Sensors connected to the FM engine

Accelerometer magnitude, microphone amplitude, and gyroscope Y were connected to FM parameters in one test view. It was the first place to judge whether a physical signal caused an audible change.

  • Commit 9be8498
  • Sensors wired to FM
  • Integrated test view

April 2026

DocumentationPublished

Wiki started

Public project documentation was added and a project wiki was started. The code, README, design documents, and longer plans had begun describing different versions of the instrument.

  • Commit 02fd220
  • Public project docs
  • Wiki started

AuditDecision

Decision: run the app on an iPad before adding features

The main repository and a duplicate starter repository were audited again. I decided to get the existing integrated view running on an iPad before adding features.

  • Repository audit
  • Duplicate starter identified
  • Checklist for the iPad run

InteractionDecision

Larger interaction design documented and set aside

Touch-native looping, a sequence and edit grammar, rhythmic capture, and a three-page architecture were documented. I set them aside until the app ran on a device.

  • Looping direction
  • Three-page architecture
  • iPad run still first

June 2026

RestartDecision

Work restarted from the device

The restart plan narrowed the work to getting the app running on an iPad. GitHub stayed the home of the code, and Google Drive stayed an archive.

  • iPad run first
  • Main repository restored
  • Drive kept as archive

CapstoneVerified

Research linked to the task board

The desk-research inventory and the open build questions were linked to the project task board. The board was set up to track research, implementation, review, and work set aside for later.

  • Public source inventory
  • Build questions linked
  • Capstone board set up

July 2026

HardwareVerified

Integrated app ran on an iPad

The app was set to open on the sensor test view. I ran the integrated build on an iPad and confirmed sound output plus audible accelerometer and gyroscope mappings. The microphone seemed active, but the response was too subtle to trust.

  • Commit 6a7683f
  • Sound passed
  • Motion mappings passed

PublicationPublished

Desk research and process page published

The desk-research page and the first process page were published with a screenshot of the 33-card task board. The board listed capstone work, open questions, source review, ideas set aside, and a privacy rule for usability video.

  • 33-card board
  • Desk research published
  • Process page published

ResearchPublished

Source list published

Documentation in the repository was corrected to match the code, and the plan for the scene-description work was clarified. A source atlas built from 319 reviewed public URLs was published.

  • Documentation corrected
  • 319 public URLs
  • Source atlas published

EvaluationDecision

Listening test reordered the mappings

Listening on the iPad, I ranked the gyroscope first and the accelerometer second. The microphone was working, but it needed a different interpretation. Accessibility requirements, an interaction model, a cognitive walkthrough, a moderated usability script, and a way to trace findings to decisions were then documented.

  • Gyroscope ranked first
  • Accessibility requirements
  • Evaluation plan

ImplementationOpen

Three repairs entered review

Microphone diagnostics, repairs to the safe-area layout and to sensor updates overriding manual controls, and scene-description work moved into review. The public project page was corrected at the same time so it no longer presented fixed routing as the finished direction.

  • Microphone diagnostics
  • Manual-control repair
  • Scene-description review

ScopeDecision

Demo narrowed to a stationary iPad

The focus narrowed to one two-minute environmental-modulation demo with the iPad resting on a table. Moving the whole device stopped being the main performance gesture. The room and the table surface modulate the sound.

  • Two-minute demo
  • Stationary iPad
  • Environmental modulation

DiagnosticsVerified

Microphone and scene description tested on the iPad

I assembled microphone diagnostics and a scene-description test in a version under review. Xcode tests and a generic iOS build passed. I cleared signing and confirmed that the microphone responded on the iPad.

  • Xcode tests passed
  • Generic iOS build passed
  • Microphone responded on iPad

InteractionVerified

Fixed mapping replaced by a matrix

A touch-first modulation matrix was implemented with nine sources, four targets, 36 bipolar route cells, editable base values, and migrated defaults. Seventeen Xcode tests and the generic iOS build passed. I signed, installed, and launched the build on the iPad. How each route responds to touch remained under review.

  • 9 sources by 4 targets
  • 17 tests passed
  • Launched on iPad

HardwareVerified

Microphone passed; route slider failed

I ran a two-minute stationary microphone test in a room measuring roughly 60 dB. The route clearly drove the sound, was stable, and pumped only slightly, although exact calibration remained open. The same test exposed a blocking defect: the route slider sat in the iPadOS bottom gesture zone.

  • Two-minute test on iPad
  • Microphone route drove the sound
  • System-gesture conflict

ImplementationVerified

First route-editor repair entered review

A route inspector was built that sat on the right by default, mirrored for left-handed use, moved diagnostics behind a disclosure control, and added a center-zero control with one-percent nudges and a zero detent. Source checks passed. The next test on the iPad rejected its fixed side rail.

  • Route inspector implemented
  • Source checks passed
  • iPad test next

HardwareOpen

Side inspector rejected on the iPad

The repaired build ran on the 11-inch iPad with the selected route fixed at the right edge. The 300-point rail made the primary bipolar slider too narrow for confident touch control and left most of the right column unused. The next layout had to put a wider slider directly below the matrix and reserve the right rail for secondary elements.

iPad running SensorSynthFM in landscape with context controls at left, the modulation matrix in the center, and a narrow selected-route inspector at right.
Rejected layout. The route amount control occupied a narrow right-side inspector, but testing on the iPad showed that the primary slider needs a wider strip below the matrix. The iPad status bar is redacted.
  • iPad screenshot
  • Side inspector rejected
  • Full-width slider required

InteractionDecision

Wider editor ran on the iPad; grid still unfinished

The selected-route editor moved below the matrix. Twenty Xcode tests, a generic iOS build, and simulator geometry checks passed before I signed, installed, and launched the build on the 11-inch iPad without errors. On the iPad the revision was better, but it still underused the available space. The dedicated ZERO control wasted room, the plus/minus nudge order had to reverse, and the grid should move toward an Ableton-like layout over time.

  • Commit c5d3472
  • 20 Xcode tests passed
  • Launched on iPad; layout still open

EvidenceDecision

Screenshot method corrected

A check confirmed that Xcode 26.3 still saw the iPad as paired. The working method was to take each screenshot by hand at full resolution in Xcode, under Window > Devices and Simulators, then review it for privacy before publication. Apple's newer Device Hub remote-control instructions did not apply to the installed version of Xcode. Automated screenshot capture was proposed but not built.

  • Xcode 26.3 pairing rechecked
  • Manual screenshot method corrected
  • Automated capture not built

DesignPublished

Matrix layout study published

The rejected layout became an interactive study comparing the layout at the time with a revision. The revision kept all nine sources visible, gave the selected route a wide bipolar strip below the grid, reversed the nudge order, made the signed amount the reset target, and reserved the right rail for secondary context. The matrix also had to be able to grow past nine sources and four targets without shrinking touch targets or burying route editing.

  • Interactive layout study
  • All nine sources visible
  • Matrix growth still an open design problem

HardwareVerified

Route controls passed on the iPad

The dedicated ZERO control was removed, the default nudge order was reversed, and the signed amount became the reset target. Twenty Xcode tests, a generic iOS build, installation, and launch passed before I confirmed all three controls worked on the iPad. The larger layout from the web study was still not built in the app. The task board kept that decision open and listed the next steps in order: full sensor-to-sound integration, defining note entry, a walkthrough, checking which mappings hold up, and the capstone summary.

  • Commit 635ab09
  • 20 Xcode tests passed
  • Three control checks passed on iPad
  • July 18 board archived

HardwareVerified

Integrated sensor build ran on the iPad

An integration fix replaced separate microphone reads with one consistent reading and moved sensor startup into the app's own startup. Twenty-two Xcode tests and the generic iOS build passed before Xcode signed, installed, and launched it on the 11-inch iPad. I confirmed that sound, accelerometer and gyroscope response, and microphone response all worked as before. The screenshot below shows the layout.

iPad running SensorSynthFM in landscape with PLAYING active, a nine-source modulation matrix, ACCEL MAG to MOD INDEX selected at plus 40 percent, and the rightmost source beyond the visible matrix.
Accepted build, unfinished layout. The ninth source needs a horizontal swipe, part of the context rail falls below the fold, and the selected-route controls span more space than they need. The iPad status bar is redacted.
  • Commit 48b6b3d
  • 22 Xcode tests passed
  • Signed, installed, and launched on iPad
  • Sound, motion, and microphone rechecked

DesignDecision

Decision: adopt part of the web study layout

I compared the iPad screenshot with the 1210 by 834 public study at the same scale. The app screen kept large, reliable touch controls, but it clipped the ninth source, left context below the fold, and separated the signed reset value from its nudge controls. I decided to adopt part of the study. Existing modulation behavior stayed. The matrix-first hierarchy, grouped route inspector, compact right rail, reset explanation, and diagnostics behind a disclosure control moved into the next layout brief. That layout had not been built or tested on the iPad.

  • Screenshot compared with the study at the same scale
  • Partial adoption chosen
  • App layout still open

August 2026

EvaluationVerified

Walkthrough passed seven of eight steps

The eight-step cognitive walkthrough on the iPad passed seven steps. Play was visible; sound and playing state made sense; the active source and destination were clear; manual control and sensor modulation were distinguishable; microphone response was visible and audible; zero neutralization and recovery worked; and the instrument premise could be stated in one sentence.

  • Seven of eight walkthrough steps passed
  • Walkthrough on the iPad
  • Zero neutralization and recovery passed

DesignOpen

Sensor display and route discovery failed

Accelerometer movement changed sound with no detectable visual response, and gyroscope visual response was too subtle to trust. Route discovery also failed because a selected zero route did not look active and creation and removal were unexplained. The changes required were a visible selected-zero state, an explanation of how to activate a route, a visible remove control, a clear display of sensor response, all nine sources visible without shrinking touch targets, and testing by other people.

  • Accelerometer sound change without detectable visual response
  • Gyroscope visual response too subtle
  • Selected zero route did not look active

ProcessDecision

Requirements tracking began

I began tracking requirements by ID, each with its own test and a status of passed, failed, or not yet tested. Seven are published on the current-build page.

  • Seven requirements published
  • Tasks on the board name their requirement and test
  • Passed, failed, or not yet tested

September 2026

DesignDecision

September 5: note entry first

A twenty-question, AI-assisted design interview returned the project to its missing core: playing notes directly on the tabletop iPad. The next step was starting, holding, changing, and releasing notes reliably, then playing phrases and chords while existing environmental modulation stayed available. The target of ten independently sounding notes and the handedness design had not been tested on the iPad. The matrix rule changed to permit scrolling when every eligible route stays reachable with readable labels and usable touch targets.

  • AI-assisted design interview
  • Note entry chosen as the first step
  • Ten-note test on the iPad still open

BuildVerified

September 20: note entry and Hold work on the iPad

The note-entry build passed 57 Xcode tests and a focused test on the iPad. Notes started, moved, retriggered, merged, stayed under Hold, and released without doubled pitches or stuck voices. Playing on the iPad changed the feedback: the active reticle grew to 88 points, labels moved away from the playing arm, and released notes kept an orange dashed ring through a 350-millisecond fade. Ten notes sounding at once had not been tested.

SensorSynthFM running on an iPad with three active note reticles, Hold enabled, and three held-note edge markers
Frame from the September 20 note-entry test on the iPad. Three active touches are visible on the surface while three held notes remain marked at the pitch edge.
  • 57 Xcode tests passed
  • Hold passed on iPad
  • Quantized and Freehand feedback reviewed on iPad
  • Not yet tested by anyone else

October 2026

BuildVerified

Instrument page added

A third page, Instrument, joined Performance and Matrix. It holds sliders for the fade in and fade out of each note (attack, decay, sustain, and release) and a filter that works on each note separately. Slider touch areas were corrected so the whole control responds. I tried the envelope and filter on the iPad.

  • Instrument page
  • Envelope and filter for each note
  • Tried on the iPad by me only

ScopeDecision

Session build limited to three pages

I set the build for the sessions as the three pages that already worked on the iPad: Performance, Instrument, and Matrix. The sequencer, capture, presets beyond one starting sound, undo, and saving were put off until after the capstone. The build gained a fixed starting sound, a RELEASE ALL button that stops every note, and a RESET START button that restores the starting sound.

  • Three pages in the session build
  • Sequencer and presets deferred
  • RELEASE ALL and RESET START added

HardwareVerified

Corrections after a rehearsal on the iPad

I rehearsed on the iPad. The rehearsal led to corrections in how playing state is shared between pages, in manual tuning, and in how notes release. A hidden audition control was removed, and the tuning reset was made easier to see.

  • Rehearsal on the iPad
  • Hidden audition control removed
  • Tuning reset made visible

November 2026

InteractionVerified

Left-right finger position became a modulation source

TOUCH X, the left-right position of each finger, joined the matrix as a tenth source. It works on each note separately, so two fingers can shape their own notes differently. Double-tapping a route cell now clears it. Audio routing controls were added for recording the sessions. Builds began reaching the iPad through TestFlight, the Apple service for distributing test builds.

  • Tenth matrix source
  • Double-tap clears a route
  • Builds delivered through TestFlight

HardwareOpen

Four or five moving fingers cut off every note

Playing with four or five fingers and moving them cut off every note. iPadOS takes that motion for a system gesture and cancels the touches, and it did so with the multitasking gesture setting turned off. I found no supported way for the app to prevent it. With Guided Access on, which locks the iPad to one app, the cutoff stopped, so the sessions will run that way.

  • Cutoff happened on the iPad
  • iPadOS system gesture
  • Guided Access prevents it

BuildVerified

November 15: navigation, a new filter, echo and reverb

Three builds went to the iPad in one evening. The first kept PERFORMANCE, MATRIX, and INSTRUMENT in the side rail on every page and removed duplicate buttons from the top of the Matrix page. The second replaced the low-pass filter with a ladder filter, a design with a stronger resonant peak, because resonance on the old filter had no audible effect. It also changed the starting sound so that sliding a finger left or right opens the filter and brightens the tone. The third added echo and reverb shared by every note, with three sliders on the Instrument page. I played each build on the iPad and kept all three changes. One automated interface test fails intermittently, as it did on the October 12 build.

  • Three builds through TestFlight
  • Played on the iPad
  • One interface test fails intermittently
  • Not yet tested by anyone else

ScopeDecision

Build held for the sessions

I decided to make no further major changes before the sessions. The first addition planned afterward is a loop recorder modeled on Capture in Ableton Live. It listens all the time, and one press turns the last phrase played into a loop.

  • No major changes before the sessions
  • Loop recorder planned afterward

Project Links