Skip to main content

Class: OpenAIWhisperSpeechToTextProvider

Defined in: packages/agentos/src/io/hearing/providers/OpenAIWhisperSpeechToTextProvider.ts:240

Speech-to-text provider that uses the OpenAI transcription API.

API Contract​

  • Endpoint: POST {baseUrl}/audio/transcriptions
  • Authentication: Authorization: Bearer <apiKey>
  • Content-Type: multipart/form-data (FormData with file blob)
  • Response format: Controlled by the response_format field. When the caller names none, OpenAI's gpt- transcription models are asked for json, and every other model, whisper-1 and a whisper-compatible server's own included, for verbose_json (segments, language and duration).

Models​

  • gpt-transcribe (default): the model OpenAI recommends for recorded speech. It returns the text and the detected languages, and it takes language hints as a languages list.
  • whisper-1: removed from the API on 2027-02-26. It serves segment timestamps, srt and vtt, so a call that asks for one of those formats runs on it unless the call names a model or the configured model serves the format itself. A configured gpt- model does not.

Supported Response Formats​

  • verbose_json: full JSON with segments, duration and language (whisper-1)
  • json: JSON with the text (the newer models also return languages)
  • text: plain text, no JSON
  • srt: SubRip subtitles (whisper-1)
  • vtt: WebVTT subtitles (whisper-1)

When text, srt, or vtt format is used, the response is returned as plain text and segments are not available.

See​

OpenAIWhisperSpeechToTextProviderConfig for configuration options See normalizeSegments() for the segment normalization logic.

Example​

const provider = new OpenAIWhisperSpeechToTextProvider({
  apiKey: process.env.OPENAI_API_KEY!,
});
// Runs on gpt-transcribe and answers in json.
const result = await provider.transcribe(
  { data: audioBuffer, mimeType: 'audio/wav', fileName: 'recording.wav' },
  { language: 'en' },
);
// Segment timestamps come from whisper-1.
const timed = await provider.transcribe(
  { data: audioBuffer, mimeType: 'audio/wav', fileName: 'recording.wav' },
  { responseFormat: 'verbose_json' },
);

Implements​

Constructors​

Constructor​

new OpenAIWhisperSpeechToTextProvider(config): OpenAIWhisperSpeechToTextProvider

Defined in: packages/agentos/src/io/hearing/providers/OpenAIWhisperSpeechToTextProvider.ts:269

Parameters​

config​

OpenAIWhisperSpeechToTextProviderConfig

Returns​

OpenAIWhisperSpeechToTextProvider

Properties​

displayName​

readonly displayName: "OpenAI Whisper" = 'OpenAI Whisper'

Defined in: packages/agentos/src/io/hearing/providers/OpenAIWhisperSpeechToTextProvider.ts:245

Human-readable display name for UI and logging.

Implementation of​

SpeechToTextProvider.displayName

id​

readonly id: "openai-whisper" = 'openai-whisper'

Defined in: packages/agentos/src/io/hearing/providers/OpenAIWhisperSpeechToTextProvider.ts:242

Unique provider identifier used for registration and resolution.

Implementation of​

SpeechToTextProvider.id

supportsStreaming​

readonly supportsStreaming: false = false

Defined in: packages/agentos/src/io/hearing/providers/OpenAIWhisperSpeechToTextProvider.ts:248

Whisper API is batch-only; streaming requires a WebSocket adapter.

Implementation of​

SpeechToTextProvider.supportsStreaming

Methods​

getProviderName()​

getProviderName(): string

Defined in: packages/agentos/src/io/hearing/providers/OpenAIWhisperSpeechToTextProvider.ts:284

Returns the human-readable provider name.

Returns​

string

The display name string 'OpenAI Whisper'.

Example​

provider.getProviderName(); // 'OpenAI Whisper'

Implementation of​

SpeechToTextProvider.getProviderName

transcribe()​

transcribe(audio, options?): Promise<SpeechTranscriptionResult>

Defined in: packages/agentos/src/io/hearing/providers/OpenAIWhisperSpeechToTextProvider.ts:310

Transcribes an audio buffer using the OpenAI Whisper API.

The audio is sent as a multipart form upload with the file, model, and optional parameters (language, prompt, temperature, response_format).

Parameters​

audio​

SpeechAudioInput

Raw audio data and metadata. The data buffer is wrapped in a Blob and sent as a form file field. If fileName is not provided, a default name is generated from the format field.

options?​

SpeechTranscriptionOptions = {}

Optional transcription settings including language hint, context prompt, temperature for sampling, and response format.

Returns​

Promise<SpeechTranscriptionResult>

A promise resolving to the normalized transcription result.

Throws​

When the OpenAI API returns a non-2xx status code.

Example​

const result = await provider.transcribe(
  { data: mp3Buffer, mimeType: 'audio/mpeg', fileName: 'voice.mp3' },
  { language: 'fr', prompt: 'Discussion about AI' },
);

Implementation of​

SpeechToTextProvider.transcribe