Advanced LLM Configuration

To set up the models used by AI Assistant you can use the environment variables or a configuration file / VS Code setting.

Environment variables-based configuration

The fastest way to set-up the models is through specific environment variables.

Note

When the models configuration file exists, the environment variables-based model configuration is skipped.

Predefined connectors

AI Assistant offers built-in support for popular LLM models via predefined connectors. Some of these connectors can also be adapted to work with other LLM providers that utilize the same API scheme or protocol by modifying the base URL or endpoint.

OpenAI

• OPENAI_API_KEY - required

• OPENAI_BASE_URL - optional

• OPENAI_MODELS - optional, comma-separated values

AzureOpenAI

• OPENAI_API_VERSION - required

• AZURE_OPENAI_API_KEY - required

• AZURE_OPENAI_ENDPOINT or OPENAI_BASE_URL - required

• AZURE_OPENAI_MODELS - optional, comma-separated values

Anthropic

• ANTHROPIC_API_KEY - required

• ANTHROPIC_BASE_URL - optional

• ANTHROPIC_MODELS - optional, comma-separated values

Google AI

• GOOGLE_AI_API_KEY - required

• GOOGLE_AI_MODELS - optional, comma-separated values

GitHub Copilot

• The process of setting up GitHub Copilot in VS Code involves the following steps:

  • Install the GitHub Copilot Chat extension.

  • Sign-in with your GitHub account.

  • When asked, allow DVT to use the Copilot features.

• COPILOT_MODELS - optional, comma-separated values

• GitHub Copilot are not available in Eclipse

Ollama

• OLLAMA_HOST - required

• OLLAMA_MODELS - optional, comma-separated values

Models selection

To adjust the selection of models, utilize the <...>_MODELS environment variable. This variable should contain a comma-separated list of model names.

If you specify exact model names, their availability won’t be verified. Although non-existent models will appear in the list, attempting to use them will result in an error.

You can also use wildcard-based patterns (* and ?) rather than specifying exact model names. Only models that match these patterns will be included in the list.

Note

• Patterns work only with model factories / providers / connectors capable of listing the available models.

• When using the OpenAI connector with non-OpenAI compatible servers by setting OPENAI_BASE_URL, listing the available models might not be available. In this case avoid using wildcards and specify the exact model names.

File-based configuration

AI Assistant offers the flexibility to fine-tune its configuration for more advanced use cases. For instance:

• Setting additional options for the model factory / provider / connector (timeout, max_retries, …).

• Changing the LLM parameters (temperature, top_p, …).

• Using short lived api keys / tokens.

• Pre-configuring an immutable set of available models for all users.

Several formats are supported for file-based configuration:

• JSON with Comments models.jsonc or VS Code settings.json -> DVT.languageModels

• JSON models.json

• JavaScript models.js

• TypeScript models.ts

When you need a dynamic configuration (ex. for short-lived API keys or tokens), it’s best to use the JavaScript or TypeScript format. Otherwise, the JSON-based formats are generally preferred.

Regardless of the format you choose, the configuration has the following properties:

• version - required, number (latest version is 2)

• configs - required, array of dynamic configurations

Each dynamic configuration has the following properties:

• provider - required, string

• models - required, comma-separated values string, array of strings, or async function returning an array of strings (JavaScript / TypeScript)

• providerOptions - optional, object or async function returning an object (JavaScript / TypeScript)

• modelOptions - optional, object or async function returning an object (JavaScript / TypeScript)

Note

modelOptions and providerOptions are specific to each model factory / provider / connector, see supported options.

AI Assistant looks for the model configs in these locations:

• $DVT_HOME/config/ai/models.{json,jsonc,js,ts}

• $DVT_AI_DIR/ai/models.{json,jsonc,js,ts}

• $HOME/.dvt/ai/models.{json,jsonc,js,ts}

• <project>/.dvt/ai/models.{json,jsonc,js,ts}

• VS Code Settings -> DVT.AI.languageModels (JSON)

Note

While the models configuration files are loaded and executed only once unless modified, dynamic configurations are cached for only a brief period (10-60 seconds). Ensure that functions used by dynamic configurations do not have lengthy execution times. It’s a good practice to cache their values and compute them only when necessary, such as when short-lived tokens have expired.

In certain setups, it is preferred to have an officially approved configuration for all users, preventing any changes by them. This can be accomplished by adding a file named final within the $DVT_HOME/config/ai/ directory. This approach is effective only in read-only shared installations, typically those based on DVT distros, as users should not be able to modify the installation files.

Warning

This is not considered a security feature because users with adequate permissions can install DVT locally and apply their own configurations.

Predefined connectors

AI Assistant offers built-in support for popular LLM models via predefined connectors. Some of these connectors can also be adapted to work with other LLM providers that utilize the same API scheme or protocol by modifying the base URL or endpoint.

Use the following values to set the models through file-base dynamic configurations. To find more information about the supported options, their meaning and default values, please refer to the official documentation of each API provider.

OpenAI

• provider: openai

• providerOptions:

  • apiKey

  • baseURL - https://api.openai.com/v1

  • organization

  • project

  • timeout - 60000 ms

  • maxRetries - 2

  • defaultHeaders

• modelOptions:

  • max_completion_tokens

  • temperature

  • top_p

  • frequency_penalty

  • presence_penalty

  • reasoning_effort

AzureOpenAI

• provider: azure-openai

• providerOptions:

  • apiKey

  • apiVersion

  • endpoint

  • baseURL

  • deployment

  • organization

  • project

  • timeout - 60000 ms

  • maxRetries - 2

  • defaultHeaders

• modelOptions:

  • max_completion_tokens

  • temperature

  • top_p

  • frequency_penalty

  • presence_penalty

  • reasoning_effort

Anthropic

• provider: anthropic

• providerOptions:

  • apiKey

  • authToken

  • baseURL - https://api.anthropic.com

  • timeout - 60000 ms

  • maxRetries - 2

  • defaultHeaders

• modelOptions:

  • max_tokens - 4096

  • temperature

  • top_k

  • top_p

Google AI

• provider: google

• providerOptions:

  • apiKey

GitHub Copilot

• provider: copilot

Ollama

• provider: ollama

• providerOptions:

  • host

  • headers

• modelOptions:

  • temperature

  • top_k

  • top_p

  • repeat_penalty

  • presence_penalty

  • frequency_penalty

Formats

JSON / JSON with Comments

{
    "version": 2,
    "configs": [
        {
            "provider": "openai",
            "models": "gpt-4*",
            "modelOptions": {
                "temperature": 0.3
            },
            "providerOptions": {
                "apiKey": "..."
            }
        }
    ]
}

JavaScript

export default {
    version: 2,
    configs: [
        {
            provider: "openai",
            models: "gpt-4*",
            modelOptions: {
                temperature: 0.3
            },
            providerOptions: {
                apiKey: "..."
            }
        }
    ]
}

TypeScript

import { ModelsConfig } from "./@api/v2"

export default {
    version: 2,
    configs: [
        {
            provider: "openai",
            models: "gpt-4*",
            modelOptions: {
                temperature: 0.3
            },
            providerOptions: {
                apiKey: "..."
            }
        }
    ]
} satisfies ModelsConfig