Skip to main content

Nvidia Riva (Speech-to-Text)

LiteLLM supports NVIDIA Riva for speech-to-text via /audio/transcriptions. Works with both the NVCF-hosted Riva endpoint (e.g. Parakeet on build.nvidia.com) and self-hosted Riva deployments.

PropertyDetails
DescriptionRiva is NVIDIA's GPU-accelerated speech AI. LiteLLM streams the audio to Riva over gRPC and returns OpenAI-compatible transcripts.
Provider Route on LiteLLMnvidia_riva/
Provider DocRiva ASR docs ↗
TransportgRPC (not HTTP)
Supported OpenAI Endpoints/audio/transcriptions
Optional install

nvidia_riva requires the gRPC client and audio decoding libraries. Install them with:

pip install 'litellm[stt-nvidia-riva]'

This pulls in nvidia-riva-client, soundfile, audioread, and numpy. They are imported lazily so the rest of LiteLLM keeps working without them.

Quick Start​

from litellm import transcription
import os

os.environ["NVIDIA_RIVA_API_KEY"] = "nvapi-..."   # your nvapi key

audio_file = open("/path/to/audio.mp3", "rb")

response = transcription(
    model="nvidia_riva/nvidia/parakeet-ctc-1_1b-asr",
    file=audio_file,
    api_base="grpc.nvcf.nvidia.com:443",
    nvcf_function_id="1598d209-5e27-4d3c-8079-4751568b1081",  # NVCF function id
)

print(response.text)

LiteLLM resamples the audio to 16 kHz mono LINEAR_PCM (Riva's required wire format) before streaming, so you can send mp3 / wav / flac / ogg directly. No need to preprocess.

Deployment modes​

Riva runs in two very different shapes. The presence of nvcf_function_id is the signal LiteLLM uses to default use_ssl, but you can always override it.

NVCF (NVIDIA-hosted)​

model_list:
  - model_name: parakeet-asr
    litellm_params:
      model: nvidia_riva/nvidia/parakeet-ctc-1_1b-asr
      api_base: grpc.nvcf.nvidia.com:443
      api_key: os.environ/NVIDIA_RIVA_API_KEY     # nvapi-...
      nvcf_function_id: 1598d209-5e27-4d3c-8079-4751568b1081

When nvcf_function_id is set, LiteLLM:

  • enables TLS (use_ssl=True)
  • attaches the function-id gRPC metadata
  • attaches authorization: Bearer <api_key>

Self-hosted (no TLS)​

model_list:
  - model_name: parakeet-asr
    litellm_params:
      model: nvidia_riva/nvidia/parakeet-ctc-1_1b-asr
      api_base: localhost:50051

Self-hosted behind an ingress with TLS​

model_list:
  - model_name: parakeet-asr
    litellm_params:
      model: nvidia_riva/nvidia/parakeet-ctc-1_1b-asr
      api_base: riva.internal.company.com:443
      use_ssl: true

Offline-only models (Parakeet TDT, Whisper)​

Some Riva NIMs only load an offline ASR engine, so a streaming request fails with INVALID_ARGUMENT: Unavailable model requested ... type=online. Set riva_offline: true on the deployment and LiteLLM sends the whole resampled buffer through the unary Recognize RPC instead of StreamingRecognize. Available from v1.108.0.

Word timestamps work the same way as in streaming mode: send response_format=verbose_json together with timestamp_granularities[]=word, since verbose_json alone returns no words. Some offline NIMs, such as Whisper Large v3, return the transcript without word timings.

Parakeet TDT leaves some words out of words unless verbatim_transcripts: true is set on the deployment, while text still has every word. Set it when you need a timing for every word, for example to seek to a word in a player.

model_list:
  - model_name: parakeet-tdt
    litellm_params:
      model: nvidia_riva/nvidia/parakeet-0.6b-tdt
      api_base: localhost:50051
      riva_offline: true
      language: multi

LiteLLM Proxy Usage​

1. Add the model to your config​

model_list:
  - model_name: parakeet-asr
    litellm_params:
      model: nvidia_riva/nvidia/parakeet-ctc-1_1b-asr
      api_base: grpc.nvcf.nvidia.com:443
      api_key: os.environ/NVIDIA_RIVA_API_KEY
      nvcf_function_id: 1598d209-5e27-4d3c-8079-4751568b1081
    model_info:
      mode: audio_transcription

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

2. Start the proxy​

litellm --config /path/to/config.yaml

# RUNNING on http://0.0.0.0:4000

3. Send a request​

curl --location 'http://0.0.0.0:4000/v1/audio/transcriptions' \
  --header "Authorization: Bearer $LITELLM_API_KEY" \
  --form 'file=@"/path/to/speech.mp3"' \
  --form 'model="parakeet-asr"'

Supported parameters​

OpenAI parameters that map cleanly to Riva:

OpenAI paramBehavior
languageMapped to Riva language_code. Bare codes like en are normalized to en-US. BCP-47 codes like de-DE pass through.
response_formatjson (default) returns { "text": "..." }. verbose_json adds duration and words (timestamps in seconds).
timestamp_granularitiesPass ["word"] to enable word-level timestamps (timestamp_granularities[]=word in a multipart form request). Without it, verbose_json returns no words.

Riva-specific parameters you can set in litellm_params (or pass directly to transcription(...)):

ParamDefaultPurpose
nvcf_function_idunsetNVCF function id. When set, defaults use_ssl=True and attaches NVCF metadata.
use_sslTrue if nvcf_function_id is set, else FalseForce TLS on or off. Useful for self-hosted Riva behind a TLS ingress.
riva_offlineFalseUse the unary Recognize RPC instead of gRPC streaming. Required for NIMs that only ship an offline engine (e.g. parakeet-0.6b-tdt, whisper-large-v3).
verbatim_transcriptsFalseRiva's verbatim_transcripts flag. Set it with Parakeet TDT in offline mode to get a timing for every word.
riva_model_name"" (auto-select)Override the internal Riva model name. Leaving it empty lets Riva pick based on language_code + sample_rate_hertz. Recommended unless you know exactly what you want.
enable_automatic_punctuationTrueStandard Riva flag.
endpointing_configunsetPass a dict that mirrors Riva's EndpointingConfig (start_threshold, stop_threshold, stop_history, stop_history_eou, ...).
chunking_strategyunsetOpenAI-style VAD config ({"type": "server_vad", "threshold": 0.5, "silence_duration_ms": 700, "prefix_padding_ms": 250}). LiteLLM translates it to Riva's EndpointingConfig.

Why is riva_model_name empty by default?​

Internal Riva deployment names like parakeet-1.1b-en-US-asr-streaming-silero-vad-sortformer are NVIDIA's deployment identifiers. They change across NIM versions, regions, and self-hosted builds. Leaving model="" in RecognitionConfig lets Riva auto-select the right one based on language_code and sample_rate_hertz, which is what you almost always want. Only set riva_model_name if you have a specific deployed model you need to pin.

Audio formats​

LiteLLM decodes inbound audio with soundfile (wav / flac / ogg) and falls back to audioread for mp3 / m4a / mp4 / webm. Audio is then resampled to 16 kHz mono LINEAR_PCM before streaming to Riva.

If decoding fails (e.g. exotic codecs, DRM, or audioread not installed), LiteLLM raises a clear error asking you to convert upstream:

ffmpeg -i input.mp3 -ac 1 -ar 16000 -sample_fmt s16 output.wav

Environment variables​

VariablePurpose
NVIDIA_RIVA_API_KEYAPI key sent as authorization: Bearer .... NVCF expects nvapi-....
NVIDIA_RIVA_API_BASEDefault host:port for the gRPC endpoint. Same effect as setting api_base in litellm_params.
NVIDIA_NIM_API_KEYUsed as a fallback for NVIDIA_RIVA_API_KEY since most users reuse the same nvapi-... key across NVCF services.

Notes & limitations​

  • Transport is gRPC streaming by default, so even short files are sent as a stream. Set riva_offline: true for NIMs that only expose the offline engine, self-hosted or NVCF-hosted. In offline mode the whole file goes in one gRPC message, so very long files can hit the server's maximum message size, and LiteLLM returns a 413. The deployment's timeout applies to the offline call, and a call that runs past it returns a 408.
  • Diarization (diarization_config) and srt / vtt response formats aren't wired up yet. Open an issue if you need them.
  • Cost calc: Riva doesn't return token usage. LiteLLM stores the audio duration on _hidden_params["audio_transcription_duration"] so cost can be derived externally.
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails, built for production.
Learn more →