Skip to main content
← Building
Product Design · iOS Development2026In Progress

SensorSynthFM: FM Shaped by the Room

Kent State MS UX CapstoneiPad FM InstrumentPlayable Note EntryBuild of November 15, 2026

Recorded on the iPad · September 20, 2026

The instrument can be played.

Since the September 20 build, I can start, move, hold, merge, and selectively release notes directly on the iPad. The loop below was recorded on the iPad that day.

Three active touches become a held chord. The orange reticles remain visible around stationary fingers; green edge markers show notes that continue after release. Screen recording from the iPad.
Text description

Three touch points appear on the iPad surface. Orange reticles mark the active fingers, while green edge markers show held notes continuing after release.

Research and process

Build history and sources

The build history lists each build by date, including the failures. The desk research lists the sources.

Research update · November 2026

Participant screener

Recruitment is underway for early, moderated usability sessions with the working iPad prototype. I built a short questionnaire in Kent State’s Qualtrics to understand interest, describe each volunteer’s background, and arrange follow-up.

What the screener asks

  • Eligibility and interest: adult status and interest in trying an unfamiliar iPad musical instrument.
  • Background: music-making activities, optional recent synthesizer use, and prior exposure to the project.
  • Practical arrangements: possible session areas, availability, and contact permission.
  • Recording choices: willingness to be recorded, with separate choices about public excerpts and keeping the private recording.

Previous music-making or synthesizer experience is not required. Beginners are welcome. I review responses individually and arrange sessions directly; completing the screener does not book a session.

Audio and video recording is required for participation. Permission to publish excerpts and permission to retain a private recording are separate choices, confirmed through session consent.

The sessions will examine how participants approach the instrument and where they encounter confusion. Findings and resulting design changes will be added after the sessions.

Demonstration only. The mockup does not collect responses. The live recruitment questionnaire is distributed separately.

Actual Qualtrics preview

Desktop and mobile views of the published participant questionnaire, captured in Qualtrics preview. These images document the form and its questions, not participant responses or study findings.

Any visible selections are demonstration inputs used to navigate the preview.

Study information and privacy

Study titleView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the study title and the start of the introduction.
Study titleOpen full-size screenshot
Study informationView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the study information and the start of its mobile text.
Study informationOpen full-size screenshot
Study information continuedView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the continuation of the study information on mobile.
Study information continuedOpen full-size screenshot
Privacy noticeView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the privacy notice and the start of its mobile text.
Privacy noticeOpen full-size screenshot
Privacy notice continuedView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the continuation of the privacy notice on mobile.
Privacy notice continuedOpen full-size screenshot

Adult eligibility

Adult eligibilityView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 1 about adult eligibility with all three choices.
Adult eligibilityOpen full-size screenshot

Music background and interest

Music activitiesView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 2 about music activities with all eight choices.
Music activitiesOpen full-size screenshot
Something else selectionView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the demonstration selection of Something else in question 2.
Something else selectionOpen full-size screenshot
Optional other detailView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the optional Something else detail with a blank text field.
Optional other detailOpen full-size screenshot
Interest levelView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 3 about interest with all four choices.
Interest levelOpen full-size screenshot
Optional interest commentView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the optional interest comment with a blank text field.
Optional interest commentOpen full-size screenshot

Optional background and prior exposure

Optional synthesizer backgroundView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 4 about optional synthesizer background and its blank text field.
Optional synthesizer backgroundOpen full-size screenshot
Prior exposureView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 5 about prior exposure with all six choices.
Prior exposureOpen full-size screenshot

Session areas and availability

Session areasView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 6 about session areas including Pittsburgh and None of these.
Session areasOpen full-size screenshot
AvailabilityView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 7 about availability with a blank text field.
AvailabilityOpen full-size screenshot

Required recording

Required recordingView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 8 about required recording with both choices.
Required recordingOpen full-size screenshot

Public-use and retention choices

Public-use permissionView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 9 about public use, its privacy notice and all three choices.
Public-use permissionOpen full-size screenshot
Private recording retentionView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 10 about retaining the private recording with both choices.
Private recording retentionOpen full-size screenshot

Contact and follow-up permission

Email contactView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing question 11 with a blank email field.
Email contactOpen full-size screenshot
Optional preferred nameView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the optional preferred-name field left blank.
Optional preferred nameOpen full-size screenshot
Optional access or scheduling arrangementsView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the optional access or scheduling field and its no-medical-history advice.
Optional access or scheduling arrangementsOpen full-size screenshot
Recruitment sourceView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the optional recruitment-source question with all seven choices.
Recruitment sourceOpen full-size screenshot
Follow-up contact permissionView screenshot +Close screenshot −
Qualtrics desktop and mobile preview showing the unchecked follow-up contact agreement and its follow-up-only notice.
Follow-up contact permissionOpen full-size screenshot

Qualtrics screenshot

Selected design target · October 2026

Primary persona: the independent techno producer

A 45-year-old man who produces techno, has a handful of releases, occasionally plays shows, and is interested in technology. He has access to an iPad and takes music seriously, but earns no meaningful income from it.

His practice is semi-professional in experience and activity. He has practical music-making experience but need not be an FM-synthesis expert. No particular iPad model, DAW, occupation, or hardware collection is assumed.

What he wants from the instrument

  • Find distinctive sounds and turn experimentation into material for his own music.
  • Begin playing quickly without extensive setup or a theory lesson.
  • Explore touch and environmental influence while retaining intentional control of the sound.
  • Understand what caused a change and recover a useful musical state.
  • Find enough expressive depth to return after the novelty wears off.

How this guides the design

This persona keeps quick note entry, understandable sound changes, and recoverable control central to the design. SensorSynthFM adds another expressive instrument to his practice; it does not replace a complete production setup.

Basis and limits

I selected this working persona after reviewing the customer-market research. Age 45 is illustrative. The techno, release, performance, and technology traits define the chosen target; their combination has not been measured in the audience.

The goals above are design hypotheses to check in participant sessions. The recruiting pool is broader than this persona, including complete beginners. Volunteers do not need to match his age, gender, genre, release history, or equipment.

AI assistance supported research synthesis and drafting. I selected the persona and recruiting approach.

The Idea

Most synth interfaces ask the player to manage a dense parameter editor. SensorSynthFM starts from a narrower bet: an iPad can become a small FM instrument that listens to touch, motion, surface vibration, and room energy, then turns those signals into timbre.

That only works if the player can see what each sensor is doing. Without that, the instrument feels random.

The main screen is for playing notes. The modulation matrix, the grid that routes each sensor to a sound parameter, remains available. Touch enters notes in Quantized or Freehand mode. The sensors shape the tone but do not take over note or pitch control.

The Thesis

Sensor modulation gave the project a point of view, but the prototype could not be played directly. The September 20 build puts note entry at the center: touch starts a voice, vertical movement changes pitch, multitouch makes chords, and Hold lets those chords remain editable after the fingers leave the glass.

Playing on the iPad changed the feedback design. A 38-point touch halo disappeared beneath a fingertip.

The next question is whether another musician can understand it without my help.

What Exists

FM engine

The build uses a fixed bank of ten voices built on AudioKit, an open-source audio library. Each touch owns its pitch while global FM controls and sensor input continue to shape timbre. Each note has its own envelope and filter, and all notes share echo and reverb. On September 7 I held five notes steadily and registered ten touches at once.

Note entry and Hold

Quantized and Freehand modes share the same landscape surface. Active reticles follow the finger, labels show note and frequency, Freehand adds signed cents, and fixed edge markers show held notes after the finger lifts. With Hold on, I can retrigger a note, slide it to a new pitch, merge it into a pitch that is already held, remove one note, or release them all.

Live sensor mapping

The matrix routes ten sources to nine destinations. In the starting sound, device motion and left-right finger position drive modulation index, finger position also opens the filter, microphone level drives loudness, and tilt drives modulator ratio. Every route can be changed or cleared on the Matrix page.

Instrument page

Sliders set the fade in and fade out of each note, a low-pass filter with resonance, and the amount of echo and reverb. RESET START restores the starting sound, and RELEASE ALL stops every note.

Scene description

A scene-description layer is written and under review. It turns raw sensor signals into descriptors such as room energy, surface impact, vibration, brightness, and bass pressure. Xcode tests and a generic iOS build passed on July 13. I have not yet tested on the iPad whether those descriptors are musically useful.

What Changed After Testing on the iPad

Listening on the iPad, I ranked the gyroscope first and the accelerometer second. The microphone response was too subtle to trust until I added diagnostics. After that it was stable and clearly tied to the sound.

A touch-first matrix replaced the hard-coded routing with nine sources, four targets, and 36 route cells, each adjustable in either direction. I signed, installed, and launched it on the iPad. The first route editor sat too close to the iPadOS gesture zone at the bottom of the screen and failed. After the September 6 repair, I could turn a route on, remove it, and restore it.

The September 20 build passed 57 Xcode tests after two failing tests were fixed. I then checked Hold and the touch feedback on the iPad.

In November, moving four or five fingers at once cut off every note. iPadOS takes that motion for a system gesture. Guided Access, which locks the iPad to one app, prevents it, and the sessions will run that way. Resonance on the first low-pass filter had no audible effect, so a ladder filter, a design with a stronger resonant peak, replaced it on November 15.

Documentation and Research

I started a project wiki because the code, research, and design notes had drifted apart.

Read the wiki article here: Why SensorSynthFM Needed a Wiki Before It Needed More Features.

The desk-research synthesis is here: SensorSynthFM desk research.

The dated build history, with task-board screenshots, is here: SensorSynthFM build history.

The code lives on GitHub. The public repository trails the build on the iPad. The related Ableton evaluation suite examines expert music software.

How It Is Being Built

I do not have a traditional software development background. The build uses AI collaborators for implementation, review, and research synthesis while I make the musical and design decisions and do the testing.

Clarence, my AI system, runs that collaboration. It keeps the task board, research notes, and source lists in one place.

What Is Still Unresolved

As of November 15, 2026

The build reaches my iPad through TestFlight, the Apple service for distributing test builds. It is not available to the public. Still untested: ten notes sounding at once with the sensors running, and left-handed play by a left-handed player. Automatic scrolling to the destination being edited is not built.

Next

Next come five to seven moderated sessions with consenting people who did not design the interface, including a left-handed participant where possible. Recruitment is underway and no session has run. Chord-name feedback for learners is a possible later requirement. The looper, sequence editor, patch generation, presets, MPE, AUv3, and Ableton integration are not part of the capstone. What happens in the sessions will guide what comes after.

Themes in this project

Embodied interaction designFM synthesis interaction designAudioKit / SwiftUI prototypingSensor-to-FM mappingVisible instrument stateAccessibility and cognitive loadCognitive walkthrough planningHuman-AI collaborative developmentDocumentation-driven product work