Skip to main content

Baseten

LiteLLM supports any Text-Gen-Interface models on Baseten.

Here's a tutorial on deploying a huggingface TGI model (Llama2, CodeLlama, WizardCoder, Falcon, etc.) on Baseten

API KEYS​

import os 
os.environ["BASETEN_API_KEY"] = ""

Baseten Models​

Baseten provides infrastructure to deploy and serve ML models https://www.baseten.co/. Use liteLLM to easily call models deployed on Baseten.

Example Baseten Usage - Note: liteLLM supports all models deployed on Baseten

Usage: Pass model=baseten/<Model ID>

Model NameFunction CallRequired OS Variables
Falcon 7Bcompletion(model='baseten/qvv0xeq', messages=messages)os.environ['BASETEN_API_KEY']
Wizard LMcompletion(model='baseten/q841o8w', messages=messages)os.environ['BASETEN_API_KEY']
MPT 7B Basecompletion(model='baseten/31dxrj3', messages=messages)os.environ['BASETEN_API_KEY']