LLM Developer Guide

This guide explains the technical flow of how Large Language Model (LLM) configurations are propagated to the LLM Gateway in Deepdesk.

LLM Config Propagation Flow

1. Admin saves LLM config

   • When an administrator creates or updates an LLM config in the Admin UI, the configuration is saved to a Google Secret Manager secret named <account>_llm_config.

2. Secret sync to Kubernetes

   • A background sync process runs every 10 minutes, copying the latest secret value from Google Secret Manager to a Kubernetes secret in the Deepdesk cluster.

3. Environment variable injection

   • The Kubernetes secret is mounted as an environment variable named CUSTOMER_LLM_CONFIG in the LLM Gateway deployment.

4. LLM Gateway config loading

   • On startup (and on config reload), the LLM Gateway loads both:
     • The Deepdesk-provisioned config from the PROVISIONED_OPENAI_CONFIG environment variable
     • The customer-provided config from the CUSTOMER_LLM_CONFIG environment variable
   • When a request is received, the gateway inspects the requested model and routes the request to the correct provider (Deepdesk or customer endpoint) based on the configuration.

Diagram

Notes

• The secret name is unique per account, ensuring tenant isolation.
• Only customer-managed endpoints are written to the customer config; Deepdesk-managed endpoints are handled separately.
• The LLM Gateway always loads both configs and dynamically selects the routing target at runtime.

Example customer config

Below is an example of a CUSTOMER_LLM_CONFIG environment variable value:

{
  "default": {
    "provider": "azure",
    "version": "2023-05-15",
    "supported_models": [
      "gpt-4",
      "gpt-4o-mini",
      "text-embedding-ada-002"
    ],
    "deployment_prefix": "deepdesk",
    "base_url": "https://d69c2439-9d4f-45dd-9cb2-ba2a9e191361.az-eu.api.rabo.cloud",
    "api_key": null,
    "oauth_token_url": "https://login.microsoftonline.com/6e93a626-8aca-4dc1-9191-ce291b4b75a1/oauth2/v2.0/token",
    "oauth_client_id": "3cedf522-ee01-4751-9c5f-f919a8ed816d",
    "oauth_client_secret": "****",
    "scopes": [
      "410cf92c-a71c-418e-9d2a-35d5ad54ef34/.default"
    ]
  }
}

Provisioned config

The provisioned config in PROVISIONED_OPENAI_CONFIG has a similar structure, but contains the Deepdesk-managed endpoints. The config is managed by Terraform, and is stored in globally shared secrets, one for each region, with the format azure_openai_<region>. For example, our deepdesk-ch-production cluster in the europe-west6 region (Switzerland) uses the config from azure_openai_europe-west6.

Example provisioned config:

{
  "endpoints": {
    "switzerlandnorth": {
      "api_key": "****",
      "base_url": "https://dd-openai-global-switzerlandnorth-001.openai.azure.com/"
    }
  },
  "models": {
    "gpt-4o": [
      "switzerlandnorth"
    ],
    "gpt-4o-mini": [
      "switzerlandnorth"
    ],
    "text-embedding-ada-002": [
      "switzerlandnorth"
    ]
  }
}