Audio Generation

AgentOS provides provider-agnostic APIs for generating music and sound effects from text prompts. Two high-level functions cover the full audio generation pipeline:

Function	Purpose
generateMusic()	Full-length musical compositions from text prompts
generateSFX()	Short sound effects from text descriptions

Both APIs support automatic provider detection, fallback chains via FallbackAudioProxy, progress callbacks, and per-call provider preference overrides.

Providers

Music providers

Provider	Env Var	ID	Notes
Suno	SUNO_API_KEY	suno	Replicate predictions API, model suno-ai/suno (the key is a Replicate token)
Udio	UDIO_API_KEY	udio	Replicate predictions API, model udio/udio
Stable Audio	STABILITY_API_KEY	stable-audio	Stability AI v2beta API, model stable-audio-open-1.0; one synchronous request
Replicate	REPLICATE_API_TOKEN	replicate-audio	meta/musicgen by default
Fal	FAL_API_KEY	fal-audio	Queue API, fal-ai/stable-audio by default
MusicGen Local	(none)	musicgen-local	Xenova/musicgen-small through @huggingface/transformers

SFX providers

Provider	Env Var	ID	Notes
ElevenLabs	ELEVENLABS_API_KEY	elevenlabs-sfx	ElevenLabs Sound Generation API
Stable Audio	STABILITY_API_KEY	stable-audio	Same model as for music
Replicate	REPLICATE_API_TOKEN	replicate-audio	meta/audiogen by default
Fal	FAL_API_KEY	fal-audio	fal-ai/stable-audio by default
AudioGen Local	(none)	audiogen-local	Defaults to Xenova/audiogen-medium, which Hugging Face does not serve (see Local generation)

Provider resolution follows priority order (top of table = highest priority). A cloud provider joins the chain when its environment variable is set; the local provider needs none and always comes last. A named provider goes first, followed by the other configured providers. When the chain holds more than one provider, a FallbackAudioProxy wraps the chain for automatic failover.

generateMusic()

Generate a musical composition from a text prompt.

import { generateMusic } from '@framers/agentos';

const result = await generateMusic({
  prompt: 'Upbeat lo-fi hip hop beat with vinyl crackle and mellow piano',
  durationSec: 60,
});
console.log(result.audio[0].url);
console.log(`Provider: ${result.provider}, Model: ${result.model}`);

With provider preferences

const result = await generateMusic({
  prompt: 'Ambient electronic soundscape with reverb pads',
  provider: 'stable-audio',
  model: 'stable-audio-open-1.0',
  durationSec: 30,
  onProgress: (event) => {
    console.log(`[${event.status}] ${event.progress ?? '?'}% - ${event.message}`);
  },
});

GenerateMusicOptions

Option	Type	Description
prompt	string	Text prompt describing the desired composition (required)
provider	string	Provider ID ("suno", "udio", "stable-audio", etc.)
model	string	Model id. Suno, Udio, Stable Audio and Fal use it when they are the named or first provider, Replicate only when it is the one provider in the chain. musicgen-local loads Xenova/musicgen-small whatever it says; when musicgen-local is the only provider in the chain, result.model still names the model given, and in a longer chain it names Xenova/musicgen-small
durationSec	number	Desired output duration in seconds
negativePrompt	string	Musical elements to avoid; passed to the provider, which ignores it
outputFormat	AudioOutputFormat	"mp3" / "wav" / "flac" / "ogg" / "aac"; passed to the provider, which ignores it, so the clip comes in the provider's own format
seed	number	Seed for reproducible generation; passed to the provider, which ignores it
timeoutMs	number	Maximum wait time in milliseconds
n	number	Number of clips to request; passed to the provider, which ignores it
onProgress	(event) => void	Progress callback with AudioProgressEvent
providerPreferences	MediaProviderPreference	Reorder or filter the fallback chain
apiKey	string	Override the API key of the first provider
providerOptions	Record<string, unknown>	Provider-specific options; passed to the provider, which ignores them

Every built-in provider reads three fields of a request: the prompt, the model and durationSec (the providers under src/io/media/audio/providers/). negativePrompt, outputFormat, seed, n and providerOptions reach the provider and change nothing, so a call returns the clips and the format the provider's API produces.

generateSFX()

Generate a short sound effect from a text description.

import { generateSFX } from '@framers/agentos';

const result = await generateSFX({
  prompt: 'Thunder crack followed by heavy rain on a tin roof',
  durationSec: 5,
});
console.log(result.audio[0].url);

GenerateSFXOptions

Option	Type	Description
prompt	string	Text prompt describing the desired sound effect (required)
provider	string	Provider ID ("elevenlabs-sfx", "stable-audio", etc.)
model	string	Model id. Stable Audio and Fal use it when they are the named or first provider, Replicate only when it is the one provider in the chain; ElevenLabs ignores it, and audiogen-local loads Xenova/audiogen-medium whatever it says (see Local generation)
durationSec	number	Desired output duration (SFX: typically 1-15s)
outputFormat	AudioOutputFormat	"mp3" / "wav" / "flac" / "ogg" / "aac"; passed to the provider, which ignores it, so the clip comes in the provider's own format
seed	number	Seed for reproducible generation; passed to the provider, which ignores it
timeoutMs	number	Maximum wait time in milliseconds
n	number	Number of clips to request; passed to the provider, which ignores it
onProgress	(event) => void	Progress callback with AudioProgressEvent
providerPreferences	MediaProviderPreference	Reorder or filter the fallback chain
apiKey	string	Override the API key of the first provider
providerOptions	Record<string, unknown>	Provider-specific options; passed to the provider, which ignores them

Result types

Both generateMusic() and generateSFX() return a similar result envelope:

interface GenerateMusicResult {
  model: string;     // e.g. "suno-ai/suno"
  provider: string;  // e.g. "suno"
  created: number;   // Unix timestamp (seconds)
  audio: GeneratedAudio[];
  usage?: AudioProviderUsage;
}

Each GeneratedAudio object contains:

interface GeneratedAudio {
  url?: string;         // Public download URL
  base64?: string;      // Base64-encoded audio data
  mimeType?: string;    // e.g. "audio/mpeg"
  durationSec?: number; // Clip duration
  sampleRate?: number;  // e.g. 44100
}

AudioProgressEvent

interface AudioProgressEvent {
  status: 'queued' | 'processing' | 'downloading' | 'complete' | 'failed';
  progress?: number;            // 0-100
  estimatedRemainingMs?: number;
  message?: string;
}

generateMusic() and generateSFX() report queued (0) once they have chosen a provider, processing (25) once it is created, and complete (100) with the result. An error at any point reports failed, so a call that fails while choosing its provider (a named provider without its API key, or no provider configured) reports failed alone. The providers report no progress of their own.

Local generation

Both musicgen-local and audiogen-local run on the local machine through @huggingface/transformers, which must be installed, and download their model on first use. No API key is required. Without providerPreferences they come last in the chain, and with no cloud key set they are the whole chain. A preferred list keeps only the available providers it names, in its order, so it can put a local provider first or leave it out, weights can pick it as the first provider, and blocked removes it. musicgen-local loads Xenova/musicgen-small. audiogen-local defaults to Xenova/audiogen-medium, a repository Hugging Face does not serve (Xenova publishes no AudioGen export), so every call that reaches audiogen-local fails, and the model option does not change the model it loads. A provider with a key earlier in the chain keeps a call from reaching it when that provider succeeds; when that provider fails, the chain falls through to audiogen-local, which fails too, and the call rejects with an AggregateError holding every provider's error.

Observability

All audio API calls emit OpenTelemetry spans (agentos.api.generate_music, agentos.api.generate_sfx) and record usage metrics to the durable usage ledger when configured.