How to Add Private Speech-to-Text to a Website with Whisper and WebAssembly

Adding a microphone button to a website looks simple: capture the user's voice, recognize the words, and insert them into a text field. In practice, making this work reliably across desktop and mobile browsers is considerably more difficult, especially when all speech recognition must happen locally without uploading microphone audio to a server.

The challenge is not merely choosing a speech recognition model. A working implementation must correctly handle microphone permissions, audio capture, resampling, browser workers, WebAssembly, ONNX model initialization, tokenization, caching, and integration with the application's text input.

This article explains how to build a practical, privacy-preserving speech-to-text feature using OpenAI Whisper, Transformers.js, ONNX Runtime Web, AudioWorklet, and a dedicated Web Worker. It also covers several real integration failures that can leave a microphone button stuck in a loading state even when the model files have downloaded successfully.

The goal is straightforward: a visitor clicks a microphone button, speaks naturally, and sees the recognized text appear in an existing input field. The visitor can review or edit the result before submitting it. The microphone recording itself never needs to leave the browser.

What fully client-side speech recognition means

There are two fundamentally different approaches to adding voice input to a website.

The first uses a speech recognition service. The browser captures audio, sends it to a remote service, and receives recognized text. This can simplify implementation, but it introduces network dependence, privacy considerations, provider availability, and potentially usage-based costs.

The second approach performs inference directly inside the browser. The website downloads a speech recognition model and the runtime needed to execute it. The browser captures microphone audio, converts the audio into numerical samples, runs the model locally, and produces the transcript without a speech-recognition server.

This article focuses entirely on the second approach.

It is important not to confuse ordinary browser speech recognition with guaranteed-local recognition. The Web Speech API can use a server-backed recognition engine. Some newer implementations provide a processLocally option, but support differs between browsers and operating systems. If strict local processing is required, generic SpeechRecognition must not be used as an automatic fallback.

Instead, the application controls the complete inference pipeline.

Microphone
    |
    v
AudioWorklet
    |
    v
Float32 PCM audio
    |
    v
Dedicated Web Worker
    |
    v
16 kHz resampling
    |
    v
Whisper + ONNX Runtime Web
    |
    v
Tokenizer and text decoding
    |
    v
React input state

The website still serves JavaScript, WebAssembly, configuration files, and model weights. These are ordinary static downloads. The distinction is that microphone audio is processed locally rather than uploaded for recognition. If the user later submits the resulting text through an ordinary form, that is a separate application action.

Choosing a practical model: Whisper Tiny or Base

Whisper is a multilingual speech recognition model family. Its ability to recognize many languages using one model makes it especially useful for multilingual websites.

For short voice messages, two models provide a useful balance between recognition quality and browser resource consumption: Whisper Tiny and Whisper Base.

These sizes refer to the selected encoder and decoder ONNX files in the examined exports. They do not include every supporting file and should not be confused with peak browser memory consumption.

Whisper Tiny is attractive because it downloads faster, normally requires fewer computing resources, and is more suitable for uncertain mobile environments. Whisper Base generally provides better recognition quality, particularly for some languages where smaller models make more mistakes.

The model should not be selected using physical RAM alone. The browser may have significantly less memory available than the device specification suggests. Existing page memory, WebAssembly allocations, temporary tensors, and operating system memory pressure all matter.

A conservative approach is to start with Tiny on mobile devices whose capabilities are unknown, reserve Base for qualified device classes, and fall back from Base to Tiny if model initialization fails.

A failed Base instance must be released before Tiny is loaded. Keeping both models resident defeats the purpose of the fallback.

Why WebAssembly is a sensible production default

ONNX Runtime Web supports executing neural networks inside browsers. For this workload, WebAssembly provides a practical CPU inference path across major desktop and mobile browsers.

WebGPU can accelerate supported models on suitable hardware, but its availability is not equivalent to a working speech recognition pipeline. A browser exposing navigator.gpu does not guarantee that ONNX Runtime supports the platform, that the particular Whisper graph initializes correctly, or that repeated inference avoids resource problems.

A conservative implementation can therefore use WASM as the production backend and evaluate WebGPU separately after compatibility, accuracy, memory, and repeated-session tests.

For the initial implementation, single-threaded WASM also avoids making the entire website depend on cross-origin isolation and SharedArrayBuffer.

env.backends.onnx.wasm.numThreads = 1;
env.backends.onnx.wasm.proxy = false;

Running inference inside a dedicated worker keeps the expensive computation away from the main JavaScript thread. The input field, virtual keyboard, scrolling, and application interface remain responsive while the model processes audio.

Keep speech recognition completely out of the initial page load

A microphone feature should not impose a substantial network and memory cost on visitors who never use it.

Do not eagerly import Transformers.js into the initial application bundle. Do not initialize ONNX Runtime when the page opens, and do not automatically download model weights during idle time.

Instead, load the recognition stack only after the user explicitly clicks the microphone button.

A useful first-use sequence is:

1. Receive the microphone button click.
2. Request microphone permission through getUserMedia().
3. If permission is denied, display an error without downloading the model.
4. If permission is granted, release the temporary microphone stream while the model is prepared.
5. Dynamically initialize the dedicated worker and load the selected local model.
6. Wait until the worker confirms that the recognition pipeline is ready.
7. Start recording automatically if the user has not canceled the operation.
8. Display an understandable listening state.

The permission request must be associated with the user's interaction. The application should also handle canceled operations and browser-specific capture restrictions instead of assuming that recording can always be restarted silently.

Model preparation should have a visible state such as Preparing voice input. A spinner without a meaningful transition can make a successful model download look like a broken microphone.

Once initialized, the worker can remain available briefly for another dictation. An idle timeout around one minute is one reasonable starting point. After that, terminate the worker and release its in-memory model. Browser-cached model files can remain available for subsequent visits.

Capture raw audio with AudioWorklet

Whisper requires numerical audio samples. Although MediaRecorder is excellent for recording audio files, using it for local speech recognition may introduce an unnecessary encoding and decoding round trip.

The simpler processing path is:

getUserMedia()
    |
    v
AudioContext
    |
    v
AudioWorklet
    |
    v
Float32 PCM buffers
    |
    v
Dedicated Worker

AudioWorklet runs audio-processing code outside the application's main JavaScript thread. It can capture PCM samples, reduce multichannel input to mono, calculate simple volume measurements, and transfer accumulated buffers to the worker.

A simplified microphone initialization looks like this:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    channelCount: { ideal: 1 },
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true,
  },
});

const context = new AudioContext();

await context.audioWorklet.addModule(
  '/assets/speech/pcm-worklet.js'
);

const source = context.createMediaStreamSource(stream);
const processor = new AudioWorkletNode(
  context,
  'pcm-capture'
);

source.connect(processor);

// Connect the worklet through an inaudible output path
// if required to keep the audio processing graph active.

The microphone constraints are preferences, not guarantees. Browsers may apply different echo cancellation, noise suppression, and automatic gain control algorithms.

One important detail is that the microphone sample rate is not necessarily 16 kHz. Many audio contexts use 44.1 kHz or 48 kHz. Whisper expects 16 kHz input, so the captured samples need proper resampling.

Simply discarding every second or third sample is not a generally correct solution. A suitable low-pass resampler, such as a windowed-sinc implementation, reduces aliasing and preserves useful speech frequencies when lowering the sample rate.

Keep the resampling work inside the dedicated worker rather than the real-time AudioWorklet callback.

Transfer audio efficiently between browser threads

AudioWorklet can produce many small chunks of audio. Forwarding every audio-rendering quantum as a separate message may create unnecessary overhead.

A practical strategy is to accumulate short batches of PCM and transfer each batch as an ArrayBuffer.

worker.postMessage(
  {
    type: 'audio-chunk',
    buffer,
  },
  [buffer]
);

The transfer list is important: it transfers ownership of the ArrayBuffer instead of making an unnecessary full copy.

The worker can append these chunks to an audio accumulator and combine them when the recording stops.

It is also useful to track the actual capture sample rate, the number of collected samples, and the recording duration. These metadata values help diagnose silent recordings without logging sensitive audio data.

Detect pauses without adding another neural network

Short text-message dictation does not necessarily require a neural voice activity detector.

An adaptive RMS detector can provide a lightweight approximation of speech activity. RMS measures signal amplitude, allowing the application to estimate when a previously active speaker has become quiet.

The interface should always support manual stopping with a second microphone click. Automatic silence detection is a convenience, not a requirement for successful recognition.

Reasonable initial values for short dictation include:

These are configuration starting points rather than universal optimal thresholds. Quiet speech, background music, and microphone noise processing can affect RMS measurements.

Prefer letting the user stop recording manually over automatically cutting off the final words of a sentence.

For ordinary short-form text input, one-shot transcription is usually simpler than repeatedly running Whisper for interim results. Record the utterance, stop, run the model once, and insert the final transcript.

The complete Whisper pipeline requires more than an ONNX model

This is one of the most important implementation details.

Whisper inference is not simply a call to an ONNX encoder followed by an ONNX decoder. A complete automatic speech recognition pipeline also requires audio feature extraction, tokenization, generation configuration, and decoding the predicted tokens into readable text.

The complete processing chain is:

Raw microphone PCM
        |
        v
Resampling to 16 kHz
        |
        v
Whisper feature extractor
        |
        v
ONNX encoder
        |
        v
ONNX decoder and token generation
        |
        v
Whisper tokenizer
        |
        v
Recognized text

A successful ONNX session initialization proves only that the model graph can be loaded. It does not prove that the entire recognition pipeline works.

In one tested Transformers.js integration, the model sessions could initialize but transcription still failed because the locally constructed pipeline did not have the required processor and tokenizer configured correctly.

The compatible implementation explicitly initialized these components using local model metadata.

The following is a reduced illustration of that version-specific approach, not a complete standalone worker:

import {
  WhisperTokenizer,
  WhisperFeatureExtractor,
  WhisperProcessor,
} from '@huggingface/transformers';

// These values come from local model configuration files.
const tokenizer = new WhisperTokenizer(
  tokenizerJson,
  tokenizerConfig
);

const featureExtractor = new WhisperFeatureExtractor(
  preprocessorConfig
);

const processor = new WhisperProcessor(
  {},
  {
    tokenizer,
    feature_extractor: featureExtractor,
  },
  ''
);

// In the affected version-specific integration:
recognizer.tokenizer = tokenizer;
recognizer.processor = processor;

The exact constructor signatures and integration method must be checked against the installed Transformers.js version. Some versions and model exports can construct these components automatically; manual creation should not be treated as a universal requirement.

With the complete pipeline available, short-form transcription can be invoked using an explicit language and task:

const result = await recognizer(audio16k, {
  language: 'en',
  task: 'transcribe',
  return_timestamps: false,
});

When the application already knows the user's language, passing it explicitly avoids unnecessary language detection and can improve predictability for short recordings.

Pin compatible Transformers.js and ONNX Runtime versions

A second important failure mode involves the relationship between Transformers.js and ONNX Runtime Web.

Transformers.js provides the higher-level recognition pipeline, while ONNX Runtime Web executes the neural network graphs. These packages must be compatible with each other and with the exact ONNX model exports being used.

In the investigated implementation, a development build of ONNX Runtime Web initialized the Whisper encoder but failed when creating a quantized decoder session. The error involved a missing quantization scale in a DequantizeLinear operation.

This was not a microphone permission problem, a download problem, or an audio format problem. The browser could receive the model files, but the runtime could not initialize one of the required model graphs.

Testing both encoder and decoder directly with a stable, pinned ONNX Runtime Web release resolved that specific incompatibility.

The compatibility check was especially valuable because it separated two different claims:

• The model file was successfully downloaded.
• The runtime successfully created a usable inference session from the model file.

Those are not equivalent.

For the tested model exports, a compatible pairing used Transformers.js 4.2.0 with ONNX Runtime Web 1.24.3. A newer development runtime failed on the quantized decoder graph. This is a version-specific observation, not evidence that older runtimes are universally better.

When overriding a transitive runtime dependency, pin the exact version in the package lockfile and document why the override exists. Upgrading the runtime should require a full browser inference regression test, not merely a successful package installation.

Check the worker bundle format before debugging Whisper

A JavaScript Web Worker can be created using different module formats.

One integration failure occurred because the application instantiated a worker as an ES module, while the generated worker bundle used the classic-worker mechanism importScripts().

These execution models are not interchangeable. A module worker cannot simply execute classic-worker-only code.

The practical lesson is to match the Worker constructor to the actual bundled output, rather than copying a constructor option from a generic tutorial.

For example, the following options are not automatically equivalent:

// Module worker: requires a compatible module bundle.
new Worker(workerUrl, { type: 'module' });

// Classic worker: requires a compatible classic bundle.
new Worker(workerUrl);

The correct choice depends on the bundler and generated JavaScript.

When diagnosing a worker initialization error, inspect the browser Console, the generated worker file, and the Network panel. Determine whether the failure happens before ONNX Runtime even begins to initialize.

Some bundled browser libraries also compute URLs relative to their own module location. When packaging them inside workers, test the resulting runtime asset paths instead of assuming that the original package-relative URLs still resolve correctly.

Serve WASM and JavaScript modules with correct MIME types

Even perfectly compatible JavaScript and ONNX model files can fail if the web server returns incorrect response headers.

A concrete failure involved a JavaScript module with the .mjs extension being served as application/octet-stream. The browser rejected the module because its response type was not appropriate for JavaScript module execution.

The expected response types are typically:

Check the actual production response, not merely the server configuration file.

curl -I https://example.com/runtime/speech/v2/runtime.mjs
curl -I https://example.com/runtime/speech/v2/runtime.wasm

The URLs above are illustrative. Use the actual versioned asset filenames from the selected runtime build.

A correct HTTP status alone is not sufficient. The browser may still reject a file because of MIME type, Content Security Policy, module execution mode, or an incorrect asset URL.

Remember that the same request can succeed when accessed directly in a browser tab but fail when loaded as an executable module or WebAssembly dependency.

Use immutable asset URLs without trapping users on broken runtime files

Whisper model files are large enough that browser caching is important.

Versioned immutable asset paths allow the browser to reuse unchanged model and runtime files without downloading them every time the user speaks.

A typical cache policy is:

Cache-Control: public, max-age=31536000, immutable

However, this creates a deployment requirement: once a URL is declared immutable, its contents should not be silently replaced.

If the runtime binary must change, publish it under a new versioned URL.

/models/speech/v1/
/runtime/speech/v1/
/runtime/speech/v2/

The purpose of changing the runtime path is not cosmetic. A browser may continue using a previously cached, incompatible WebAssembly binary if the application reuses an immutable URL after changing its contents.

It is also useful to keep the JavaScript glue and WebAssembly files from the same runtime distribution. Mixing runtime files across versions can cause initialization failures that are difficult to diagnose.

A build-time manifest containing the expected files, sizes, and cryptographic checksums can catch missing or mismatched assets before deployment.

Configure the recognition library to use first-party assets and disable remote model loading. That way, a missing local model produces an explicit error rather than triggering an unplanned download from an external model host.

Why the microphone can spin even though nothing works

A common symptom during development is a microphone button that requests permission, displays a loading animation, and eventually returns to an error state without inserting text.

The browser may even show that the microphone is being used.

This does not prove that the speech recognizer works. It proves only that microphone capture was permitted or active at some point.

Several independent failures can produce nearly identical symptoms:

1. Incorrect JavaScript module MIME type. The browser downloads a runtime module but refuses to execute it.
2. Worker execution format mismatch. The compiled worker uses classic-worker APIs while being started as a module worker, or vice versa.
3. ONNX decoder incompatibility. The runtime creates the encoder session but fails on a quantized decoder graph.
4. Incomplete recognition pipeline. The ONNX sessions exist, but the tokenizer or audio processor is not initialized correctly.

These problems must be diagnosed independently. Fixing one may expose the next.

The most important debugging principle is that HTTP 200 does not mean successful model initialization, and successful model initialization does not mean successful transcription.

Even successful transcription does not prove that the result reaches the application's actual text field.

Each boundary needs a meaningful diagnostic signal.

Add observability without collecting private speech

Useful diagnostic logging does not require recording what the user says.

The application can track technical states such as:

permission-granted
worker-created
runtime-loading
runtime-ready
encoder-ready
decoder-ready
recording-started
pcm-received
recording-stopped
inference-started
inference-completed
result-inserted

On failure, include the stage, a stable technical error code, and sanitized error metadata.

A model initialization failure should be distinguishable from microphone permission denial or an empty recording. A worker failure should be visible in development instead of being swallowed by a generic error handler.

Do not log microphone samples, recognized sentences, token sequences, or other speech content. Performance telemetry can use duration buckets, selected model tier, initialization time, inference time, and error categories.

This makes a failing microphone much easier to debug without weakening the privacy requirements.

Insert the transcript through React state

Speech recognition is only useful when the final transcript reaches the existing form field.

In a React application, that field is often controlled by component state. Directly assigning to input.value can conflict with the framework's rendering behavior.

Instead, the recognized text should pass through the same state update path used by ordinary typing.

The integration also needs to consider asynchronous behavior. Recognition may finish after the user has already typed additional characters or moved the cursor.

Using a stale input value captured at recording start can accidentally overwrite newer edits.

A simplified integration pattern is:

const insertion = insertTranscript({
  currentValue: latestValue,
  transcript: recognizedText,
  selection: currentSelection,
  maxLength: inputLimit,
});

setMessage(insertion.value);
restoreCaret(insertion.caret);

The insertion helper should preserve the current text, optionally replace the selected range, apply appropriate spacing, and respect the field's maximum length.

Unicode deserves attention because JavaScript string length counts UTF-16 code units. Cutting a string at an arbitrary position can split a surrogate pair.

Most importantly, recognizing speech should never automatically submit the form. The user must be able to review and correct the text before sending it.

Handle failures without breaking ordinary text input

The speech subsystem should be optional.

If microphone permission is denied, the browser cannot allocate the model, a worker crashes, or a recording produces no speech, the normal input field must continue working.

Represent the microphone lifecycle with explicit states such as:

idle
preparing
listening
recognizing
error

Use accessible labels and visible status feedback. A microphone that simply turns red or pink after loading tells the user almost nothing.

Cancellation should clean up audio tracks, the audio context, pending buffers, and the active recording session. The interface should also prevent stale results from an earlier canceled recording from overwriting a newer input state.

If a large model fails because of memory or initialization constraints, release it completely before attempting a smaller model. Limit retry attempts so the browser does not repeatedly download or initialize models that cannot run.

Do not silently switch to a cloud speech service when local recognition fails. If local-only processing is a strict requirement, a failed local recognizer must fail locally.

Test the complete browser pipeline

A successful TypeScript check or production build is not sufficient proof that voice input works.

Likewise, confirming that the browser can download ONNX files does not prove that both model sessions initialize successfully.

The most useful regression test should exercise the same worker, runtime files, model, processor, and tokenizer that production uses.

Feed a known short PCM recording into the real browser inference path and verify that a nonempty transcript is returned.

Then test the complete microphone interaction in an actual browser.

• Initial page load: verify that no speech models or runtime assets download before the microphone is used.
• Cold initialization: verify microphone permission, model requests, JavaScript and WASM MIME types, and a real worker-ready response.
• ONNX initialization: verify that both encoder and decoder sessions can be created.
• Audio capture: confirm that AudioWorklet produces nonempty PCM buffers.
• Resampling: confirm that the worker receives the real sample rate and produces valid 16 kHz audio.
• Manual stopping: verify that clicking the microphone a second time stops recording reliably.
• Automatic stopping: verify that natural pauses end recording without cutting off quiet speech.
• Transcription: verify that the complete processor, model, and tokenizer return recognizable text.
• Input integration: verify that the recognized text appears inside the actual controlled field without overwriting user edits.
• Failure recovery: test denied permissions, missing files, worker errors, failed model initialization, cancellation, and a smaller-model fallback.
• Privacy: inspect network requests and confirm that microphone audio is not uploaded to a server or analytics service.
• Mobile compatibility: repeat tests on real phones, including cold starts, warm starts, and repeated dictation sessions.

Recorded PCM tests are excellent for validating the inference stack, but they do not prove that microphone capture works. The final acceptance test must include the real microphone and actual user interface.

A practical architecture for reliable browser voice typing

For a website that needs short, multilingual, privacy-preserving voice input, the most practical architecture is deliberately simple:

React microphone button
        |
        v
Permission request
        |
        v
Lazy model initialization
        |
        v
AudioWorklet PCM capture
        |
        v
Adaptive RMS / manual stop
        |
        v
Dedicated Worker
        |
        v
16 kHz audio resampling
        |
        v
Whisper Tiny or Base Q8
        |
        v
Compatible ONNX Runtime WASM
        |
        v
Whisper processor + tokenizer
        |
        v
Final transcript
        |
        v
Controlled React input

The application should self-host versioned model and runtime files, keep inference off the main thread, avoid unnecessary model downloads, release resources after use, and never upload microphone audio for speech recognition.

More sophisticated options such as WebGPU acceleration, neural voice activity detection, continuous partial transcription, or additional model tiers can be considered later when measurements justify their complexity.

The most valuable lesson is not a particular library version or a particular model size. It is the importance of validating every boundary of the browser speech recognition pipeline.

A working microphone permission prompt does not mean audio is being transcribed. A downloaded model does not mean its decoder initialized. A successful ONNX session does not mean the tokenizer is available. And a successful transcript does not mean the application inserted the text correctly.

The feature is complete only when a visitor can press the microphone, speak naturally, stop recording, and see the recognized words appear in the editable field—without sending the recording to a speech recognition server.