NEF-SPA Phono2Gesture Pathway
The Phono2Gesture Pathway models how a phonological syllable sequence gives rise to time-varying gesture activations. Syllable oscillators provide a relative time frame, and separate recurrent ramp integrators generate the gesture trajectories. The original four-slot example also retains its inhibitory routing controller.
This is a sensorimotor component of the broader NEF-SPA speech-processing model. The Lexical Model addresses access between concepts, lemmas and phonological forms. The present pathway starts with a supplied syllable sequence; the two demonstrations are currently separate implementations.
Phono2Gesture Pathway — Interactive Simulation →
“From syllable representation to articulation.” Select one of eight German words, a stressed or unstressed syllable, or the original bap → gum → dip → bap sequence. All 30 selections replay precomputed neural recordings with play, pause, slow motion and a shared timeline. An expandable Integrated Control Parameters panel sits directly below the phase circles, so phase and articulatory activation can be observed together. Abstract agonist–antagonist groups and visual articulatory movements follow the same timeline. The public page performs no new neural computation; recomputation is available only in the local Python version.
Python source code: GitHub repository: nef-spa-phono2gesture. The repository includes the model source, contour references, recorded simulation, tests and instructions for running the model locally.
1. Words, syllables and timing
The repertoire contains Tasse, Tasche, Kanne, Katze, Teller, Topf, Stuhl and Schrank. Teller has two selectable realizations: tE → lA (short ER) and tE → lEA (long ER). The 20 individually selectable syllables include earlier CV and CCV reductions as well as the complete lexical syllables and bap, gum and dip. Separate selection rows distinguish stressed and unstressed syllables; @ denotes schwa. The original four-syllable sequence bap → gum → dip → bap remains available.
Each syllable occurrence has its own two-dimensional neural oscillator, including both bap occurrences. Its decoded x/y state defines phase within the cycle. The unit circle marks each gesture’s start and end in degrees, with matching colours in the Gesture Activation traces. The oscillator frequency is 1.5 Hz; slower playback changes viewing speed only.
The first oral onset gesture starts at 54°. Vowels in stressed syllables span 105.4°–257.5°; single vowels in unstressed open syllables span 105.4°–240.6°. lEA has two overlapping vowel gestures on one oscillator. Onset and coda clusters, associated glottis gestures and nasal velum gestures follow the full fixed rule set documented in the player’s expandable Timing rules.
Syllable-to-syllable coordination
The first oscillator receives an external start signal. Each subsequent oscillator is triggered during 162°–180° of its immediate predecessor, with command amplitude 1.5. The next oscillator continues after that start signal ends. Its own gesture windows then govern its gestures, allowing overlap with the preceding syllable. Neural decoding and filtering can delay the visible response relative to the prescribed phase boundaries. This transition rule currently applies equally to stressed and unstressed syllables and to all implemented syllable structures.
2. Routing through inhibition and disinhibition
The original four-slot sequence retains the SPA controller: the active syllable representation releases the appropriate signal pathways from inhibition. The selected oscillator phase then determines which gesture signal passes through. Section 02 shows graded inhibitory neural activity for one representative channel per slot.
The externally selected lexical words and individual syllables use direct connections from phase decoders to their gesture ramps, without competing routing pathways. Lexical word order is stored and checked against the mental lexicon’s syllabification output; a neural trigger ensemble passes the start signal to the next oscillator. The demonstrations remain separate implementations, with no automatic online connection to the mental lexicon.
3. Separate gestures and integrated control parameters
Each gesture occurrence has its own recurrent ramp integrator, including repeated gestures and different consonant positions within a syllable. The four-syllable reference now has 16 separate gesture ramps. Gesture Activation and the optional Gesture Ramp Integrators panel preserve these instances. Consonant gesture names describe their articulator and, where needed, their position C1, C2, C3 or C4.
Several gestures can act on the same control parameter: for example, different lip-constriction gestures contribute to Lip closure. Conversely, one gesture can drive several parameters: Sch controls tongue-tip height, tongue-tip frontness and lip rounding synchronously. The expandable Integrated Control Parameters panel directly below the oscillators shows the resulting shared parameter activities, combining the effects of all relevant gestures. It starts collapsed.
4. Abstract agonist–antagonist muscle groups
Each articulatory parameter is represented by two one-dimensional ensembles, with 300 spiking neurons per ensemble. The reference sequence has eight parameter pairs; the extended word and syllable programs have nine, adding tongue-tip frontness. Positive and negative group activities are displayed separately. Their difference is the signed signal shown in Integrated Control Parameters, before the articulation model’s gains and offsets.
Targets come from the Speech Articulation Trainer gesture lexicon. Simultaneously active gestures contribute an activation-weighted mixture, with neutral targets filling unused activation weight and normalization above total weight one. Signs are separated before mixing. The muscle stage is a separate Nengo simulation driven by the recorded gesture traces. Physical muscle forces and coactivation-dependent stiffness are not modeled.
5. Visual articulatory movement
A visual-only Python port of the Speech Articulation Trainer contour model converts decoded parameters into tongue, lip, jaw and velum contours. Glottal aperture is shown separately in a superior view. Every frame uses the corresponding neural sample without additional temporal smoothing or retiming.
The expanded repertoire includes E, O, schwa and vocalic ER, Sch and other fricative targets, a velar R approximant, and final L. The display is a coarse articulatory prototype; affricates use coordinated oral gestures, and no frication sound or acoustic output is generated.
At the geometry input, positive gains are 1.1 for tongue-tip height, 1.2 for tongue-dorsum height and glottal aperture, and 1.0 for the remaining parameters. Velum height uses offset +0.1 and negative gain 1.3; other offsets are zero and negative gains are 1.0. The neural displays retain the unscaled values. Visible fluctuations, overlaps and imperfect closures remain part of the recorded model behaviour.
Implementation and scope
All 30 recordings were computed in Nengo with a 1-ms integration step and 5-ms display samples over 2.5 seconds. Neuron counts vary by program and are shown in the player’s Model, playback and scope panel. The four-slot reference contains 309,820 neurons before its 4,800-neuron opponent stage; extended programs use 5,400 neurons in that stage.
Neural model ensembles use seed 11 and opponent groups seed 23. The historical SPA reference also constructs an unsaved random vocabulary, so these seeds alone do not guarantee bit-identical regeneration of that reference. The published recordings are preserved with checksums. The website only plays saved data; the downloadable Python version supports local recomputation of the selected program.
The reference is based on 20_prod_4syll_map2phon_allGestures.ipynb from the 2022_Syllables collection. Its four-sequence-slot controller is retained, while gesture ramps are separated by occurrence and current timing rules are applied. The added lexical programs also use one oscillator per syllable occurrence. This remains a proof of concept without acoustic synthesis or articulator biomechanics.
Scientific background
Kröger BJ (2023). The NEF-SPA Approach as a Framework for Developing a Neurobiologically Inspired Spiking Neural Network Model for Speech Production. Journal of Integrative Neuroscience 22:124. Read the publication. The oscillator, routing and gesture-activation stages are illustrated in the model’s motor-planning account; the four-syllable example is shown in Figure 5.
Development: Model reproduction and interactive documentation under the scientific direction of Bernd J. Kröger, with assistance from ChatGPT.