.. _Advanced LLM Configuration:

Advanced LLM Configuration
==========================

To set up the models used by AI Assistant you can use the **environment variables** or a **configuration file / VS Code setting**.


Environment variables-based configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The fastest way to set-up the models is through specific environment variables. Configuration based on these variables is skipped when :ref:`file-based configurations <file-based-configuration>` are used.

Predefined connectors
---------------------

AI Assistant offers built-in support for popular LLM models via predefined connectors. Some of these connectors can also be adapted to work with other LLM providers that utilize the same API scheme or protocol by modifying the base URL or endpoint.


OpenAI
......

- ``OPENAI_API_KEY`` - **required**
- ``OPENAI_BASE_URL`` - optional
- ``OPENAI_ORGANIZATION`` - optional
- ``OPENAI_PROJECT`` - optional
- ``OPENAI_MODELS`` - optional, comma-separated values

AzureOpenAI
...........

- ``AZURE_OPENAI_API_KEY`` - **required**
- ``AZURE_OPENAI_RESOURCE_NAME`` or ``AZURE_OPENAI_BASE_URL`` - **required**
- ``AZURE_OPENAI_DEPLOYMENT_NAME`` - **required**
- ``AZURE_OPENAI_API_VERSION`` - optional

OpenAI-Compatible
.................

- ``OPENAI_COMPATIBLE_BASE_URL`` - **required**
- ``OPENAI_COMPATIBLE_MODELS`` - **required**, comma-separated values
- ``OPENAI_COMPATIBLE_DISPLAY_NAME`` - optional, a custom name for the provider, displayed in the UI for easier identification

Anthropic
.........

- ``ANTHROPIC_API_KEY`` - **required**
- ``ANTHROPIC_BASE_URL`` - optional
- ``ANTHROPIC_MODELS`` - optional, comma-separated values

Google AI
.........
- ``GOOGLE_AI_API_KEY`` - **required**
- ``GOOGLE_AI_MODELS`` - optional, comma-separated values

Google Vertex AI
................
- ``GOOGLE_VERTEX_API_KEY`` or (``GOOGLE_VERTEX_PROJECT``, ``GOOGLE_VERTEX_LOCATION``, and ``GOOGLE_VERTEX_SERVICE_ACCOUNT_KEY_FILE``) - **required**
- ``GOOGLE_VERTEX_MODELS`` - optional, comma-separated values

.. note::

    To configure Google Vertex AI provider it is enough to set the ``GOOGLE_VERTEX_API_KEY`` environment variable if you are using Vertex AI's `express mode <https://docs.cloud.google.com/vertex-ai/generative-ai/docs/start/express-mode/overview>`__ . If you are using a service account, you can set ``GOOGLE_VERTEX_PROJECT``, ``GOOGLE_VERTEX_LOCATION`` and ``GOOGLE_VERTEX_SERVICE_ACCOUNT_KEY_FILE`` (which should point to the absolute path of the key file for your service account).

Amazon Bedrock
..............

- ``AWS_REGION`` - **required**
- ``AWS_BEARER_TOKEN_BEDROCK`` or (``AWS_ACCESS_KEY_ID`` and  ``AWS_SECRET_ACCESS_KEY``) - **required**
- ``AWS_SESSION_TOKEN`` - optional
- ``BEDROCK_MODELS`` - optional, comma-separated values

.. note::

    To configure Amazon Bedrock provider it is enough to set the region and an `API Key <https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_bedrock.html>`__ using the ``AWS_BEARER_TOKEN_BEDROCK`` environment variable or an `access key for an IAM user <https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html>`__ with the right Bedrock permissions using the ``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY`` environment variables.

GitHub Copilot
..............

- The process of setting up GitHub Copilot in VS Code involves the following steps:
    - Install the GitHub Copilot Chat extension.   
    - Sign-in with your GitHub account.
    - When asked, allow DVT to use the Copilot features.
    - To configure which models are available, see `Configuring access to AI models in Copilot <https://docs.github.com/en/copilot/how-tos/ai-models/configuring-access-to-ai-models-in-copilot>`_.
- ``COPILOT_MODELS`` - optional, comma-separated values used to refine the list of available models, to access advanced models you still need to configure access to AI models in Copilot as mentioned above.
- GitHub Copilot are not available in Eclipse.

Ollama
......

- ``OLLAMA_HOST`` - **required**
- ``OLLAMA_MODELS`` - optional, comma-separated values

Models selection
----------------

To adjust the selection of models, utilize the ``<...>_MODELS`` environment variable. This variable should contain a comma-separated list of model names.

If you specify exact model names, their availability won't be verified. Although non-existent models will appear in the list, attempting to use them will result in an error.

You can also use wildcard-based patterns  (``*`` and ``?``) rather than specifying exact model names. Only models that match these patterns will be included in the list. 

.. note:: 
    
    - Patterns work only with model factories / providers / connectors capable of listing the available models.
    
    - When using the OpenAI connector with non-OpenAI compatible servers by setting ``OPENAI_BASE_URL``, listing the available models might not be available. In this case avoid using wildcards and specify the exact model names. 

.. _proxies-certificates-env-vars:

Proxies and certificates 
------------------------

Configuring proxies or specific certificate authority (CA) certificates might be necessary to connect to a LLM provider. AI Assistant takes into account the following environment variables:

- ``DVT_AI_HTTP_PROXY`` - proxy URL for HTTP requests, if not set, the value of ``http_proxy`` or ``HTTP_PROXY`` environment variable will be used.
- ``DVT_AI_HTTPS_PROXY`` - proxy URL for HTTPS requests, if not set, the value of ``https_proxy`` or ``HTTPS_PROXY`` environment variable will be used.
- ``DVT_AI_NO_PROXY`` - comma-separated list of domains that should bypass the specified proxies, excluding a domain is possible using ``*.`` or ``.`` prefixes (ex. ``.example.com`` or ``*.example.com``), if not set, the value of ``no_proxy`` or ``NO_PROXY`` environment variable will be used.
- ``DVT_AI_CA_CERTIFICATES`` - comma-separated list of PEM certificate files absolute paths, when set the system trusted CA certificates will be ignored.
- ``DVT_AI_IGNORE_CERTIFICATES`` - certificate validation will be skipped when ``true``

.. _file-based-configuration:

File-based configuration
~~~~~~~~~~~~~~~~~~~~~~~~

AI Assistant offers the flexibility to fine-tune its configuration for more advanced use cases. For instance: 

- Setting additional options for the model factory / provider / connector (timeout, max_retries, ...).
- Changing the LLM parameters (temperature, top_p, ...).
- Using short lived API keys / tokens.
- Pre-configuring an immutable set of available models for all users.


Several formats are supported for file-based configuration:

- JSON with Comments ``models.jsonc`` or ``VS Code settings.json -> DVT.languageModels``
- JSON ``models.json``
- JavaScript ``models.js``
- TypeScript ``models.ts``

When you need a dynamic configuration (ex. for short-lived API keys or tokens), it's best to use the JavaScript or TypeScript format. Otherwise, the JSON-based formats are generally preferred.

Regardless of the format you choose, the configuration has the following properties:

- ``version`` - **required**, number (latest version is **6**)
- ``configs`` - **required**, array of *dynamic configurations*

Each *dynamic configuration* has the following properties:

- ``provider`` - **required**, string
- ``models`` - **required**, comma-separated values string, array of strings, or async function returning an array of strings (JavaScript / TypeScript)
- ``providerOptions`` - **optional**, object or async function returning an object (JavaScript / TypeScript)
- ``modelOptions`` - **optional**, object or async function returning an object (JavaScript / TypeScript)

.. note:: 
    
    ``modelOptions`` and ``providerOptions`` are specific to each model factory / provider / connector, see :ref:`supported options <file-predefined-connectors>`.

AI Assistant looks for the model configs in these locations:

- **$DVT_HOME/config/ai/models.{json,jsonc,js,ts}**
- **$DVT_AI_DIR/models.{json,jsonc,js,ts}**
- **$HOME/.dvt/ai/models.{json,jsonc,js,ts}**
- **<project>/.dvt/ai/models.{json,jsonc,js,ts}**
- **VS Code Settings -> DVT.AI.languageModels (JSON)**

.. note:: 
    
    While the models configuration files are loaded and executed only once unless modified, *dynamic configurations* are cached for only a brief period (10-60 seconds). Ensure that functions used by dynamic configurations do not have lengthy execution times. It's a good practice to cache their values and compute them only when necessary, such as when short-lived tokens have expired.

In certain setups, it is preferred to have an officially approved configuration for all users, preventing any changes by them. This can be accomplished by adding a file named ``final`` within the ``$DVT_HOME/config/ai/`` directory. This approach is effective only in read-only shared installations, typically those based on DVT distros, as users should not be able to modify the installation files.

.. warning::
    
    This is not considered a security feature because users with adequate permissions can install DVT locally and apply their own configurations.


Formats
-------

JSON / JSON with Comments
.........................

.. code-block:: json

    {
        "version": 6,
        "configs": [
            {
                "provider": "openai",
                "models": "gpt-4*",
                "modelOptions": {
                    "temperature": 0.3
                },
                "providerOptions": {
                    "apiKey": "..."
                }
            }
        ]
    }


JavaScript
..........

.. code-block:: js

    export default {
        version: 6,
        configs: [
            {
                provider: "openai",
                models: "gpt-4*",
                modelOptions: {
                    temperature: 0.3
                },
                providerOptions: {
                    apiKey: "..."
                }
            }
        ]
    }
    

TypeScript
..........

.. code-block:: ts
    
    import { ModelsConfig } from "./@api/v6"

    export default {
        version: 6,
        configs: [
            {
                provider: "openai",
                models: "gpt-4*",
                modelOptions: {
                    temperature: 0.3
                },
                providerOptions: {
                    apiKey: "..."
                }
            }
        ]
    } satisfies ModelsConfig
    

.. _file-predefined-connectors:

Predefined connectors
---------------------

AI Assistant offers built-in support for popular LLM models via predefined connectors. Some of these connectors can also be adapted to work with other LLM providers that utilize the same API scheme or protocol by modifying the base URL or endpoint.

Use the following values to set the models through file-base *dynamic configurations*. To find more information about the supported options, their meaning and default values, please refer to the official documentation of each API provider.

OpenAI
......

- provider: **openai**
- providerOptions:

  - **apiKey**
  - baseURL
  - organization
  - project
  - headers
  - displayName - custom name for the provider, displayed in the UI for easier identification

- modelOptions:

  - maxOutputTokens
  - temperature
  - stopSequences
  - topP
  - topK
  - presencePenalty
  - frequencyPenalty
  - responseFormat
  - seed
  - headers

AzureOpenAI
...........

- provider: **azure-openai**
- providerOptions:

  - **apiKey**
  - **resourceName** or **baseURL**
  - apiVersion
  - headers
  - useDeploymentBasedUrls
  - displayName - custom name for the provider, displayed in the UI for easier identification

- modelOptions:

  - maxOutputTokens
  - temperature
  - stopSequences
  - topP
  - topK
  - presencePenalty
  - frequencyPenalty
  - responseFormat
  - seed
  - headers

OpenAI-Compatible
.................

- provider: **openai-compatible**
- providerOptions:

  - **baseURL**
  - apiKey
  - headers
  - displayName - custom name for the provider, displayed in the UI for easier identification

- modelOptions:

  - maxOutputTokens
  - temperature
  - stopSequences
  - topP
  - topK
  - presencePenalty
  - frequencyPenalty
  - responseFormat
  - seed
  - headers

Anthropic
.........

- provider: **anthropic**
- providerOptions:

  - **apiKey**
  - baseURL
  - headers
  - displayName - custom name for the provider, displayed in the UI for easier identification

- modelOptions:

  - maxOutputTokens
  - temperature
  - stopSequences
  - topP
  - topK
  - presencePenalty
  - frequencyPenalty
  - responseFormat
  - seed
  - headers

Google AI
.........

- provider: **google**

- providerOptions:

  - **apiKey**
  - baseURL
  - headers
  - displayName - custom name for the provider, displayed in the UI for easier identification

- modelOptions:

  - maxOutputTokens
  - temperature
  - stopSequences
  - topP
  - topK
  - presencePenalty
  - frequencyPenalty
  - responseFormat
  - seed
  - headers

Google Vertex AI
................

- provider: **google-vertex**

- providerOptions:

  - apiKey - used for express mode
  - location
  - project
  - baseURL
  - headers
  - googleAuthOptions - authentication options that may include:
    
    - authClient
    - keyFile
    - credentials
    - clientOptions
    - scopes
    - projectId
    - universeDomain
  - displayName - custom name for the provider, displayed in the UI for easier identification

- modelOptions:

  - maxOutputTokens
  - temperature
  - stopSequences
  - topP
  - topK
  - presencePenalty
  - frequencyPenalty
  - responseFormat
  - seed
  - headers

Amazon Bedrock
..............

- provider: **bedrock**

- providerOptions:
  
  - **region**
  - **apiKey** or (**accessKeyId** and **secretAccessKey**)
  - sessionToken
  - baseURL
  - headers
  - credentialProvider

- modelOptions:

  - maxOutputTokens
  - temperature
  - stopSequences
  - topP
  - topK
  - presencePenalty
  - frequencyPenalty
  - responseFormat
  - seed
  - headers

.. note::

    To configure Amazon Bedrock provider it is enough to provide an `API Key <https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_bedrock.html>`__ using the ``apiKey`` option or an `access key for an IAM user <https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html>`__ with the right Bedrock permissions using the ``accessKeyId`` and ``secretAccessKey`` options.

GitHub Copilot
..............

- provider: **copilot**

Ollama
......

- provider: **ollama**

- providerOptions:
  
  - **host**
  - headers
  
- modelOptions:
  
  - temperature
  - top_k
  - top_p
  - repeat_penalty
  - presence_penalty
  - frequency_penalty
  - stop
  - typical_p
  - seed

.. _proxies-certificates-predefined-connectors:

Proxies and certificates
------------------------

Configuring proxies or specific certificate authority (CA) certificates might be necessary to connect to a LLM provider. AI Assistant supports provider specific configuration by placing the relevant configuration inside the ``providerOptions`` property.
  
- ``httpsProxy`` - **optional**, string representing the proxy URL for HTTP requests.
- ``httpProxy`` - **optional**, string representing the proxy URL for HTTPS requests.
- ``noProxy`` - **optional**, comma-separated values string representing a list of domains that should bypass the specified proxies, excluding a domain is possible using ``*.`` or ``.`` prefixes (ex. ``.example.com`` or ``*.example.com``).
- ``caCertificates`` - **optional**, string or array of strings representing a list of PEM certificate files absolute paths, when set the system trusted CA certificates will be ignored.
- ``ignoreCertificates`` - **optional**, boolean, certificate validation will be skipped when ``true``

.. note::
    
    :ref:`Environment variables-based <proxies-certificates-env-vars>` configuration of proxies and certificates will be ignored when using ``providerOptions`` to set them.

.. code-block:: js
    
    export default {
        version: 6,
        configs: [{
            provider: "openai",
            models: "gpt-4o,gpt-5,gpt-5.1",
            providerOptions: {
                apiKey: "...",
                httpsProxy: "https://proxy.com:9000"
            }
        }]
    }


Examples
--------

Using an OpenAI compatible server with short term token
.......................................................

.. code-block:: js
    
    /*
    Example of model config for an OpenAI compatible server with short term token
    LLM access token / api key is obtained from an authentication server and it is refreshed after 60 minutes
    */

    /* LLM inference server base URL */
    const SERVER_BASE_URL = '...'

    /* Comma separated list of models available on the server. Wildcard-based patterns can be used as long as the server supports listing models (/models endpoint) */
    const SERVER_MODELS = '...'

    /* Authentication server URL */
    const AUTH_URL = '...'

    /* Authentication user */
    const AUTH_USER = '...'

    /* Custom name for the provider, displayed in the UI */
    const PROVIDER_DISPLAY_NAME = '...'

    /* Authentication password / secret */
    const AUTH_PASS = '...'

    /* JSON property from authentication server's response that holds the token / api key */
    const AUTH_RESPONSE_TOKEN_PROPERTY = 'access_token'

    /* Refresh the token / api key after it expires */
    const TOKEN_EXPIRATION_PERIOD =
        /* min */   60 *
        /* sec */   60 *
        /*  ms */ 1000;

    /* ------------------------------------------------------------------------ */

    const api = acquireDvtAiApi();

    let cachedToken;
    let tokenExpirationTimestamp = 0;
    async function getToken() {
        const now = Date.now();
        if (tokenExpirationTimestamp <= now) {
            if (cachedToken)
                api.log.info(`Cached token expired @ ${new Date(tokenExpirationTimestamp).toLocaleString()}`)
            tokenExpirationTimestamp = now + TOKEN_EXPIRATION_PERIOD;
            cachedToken = await requestToken();
        } else {
            api.log.info("Reusing cached token");
        }
        return cachedToken;
    }

    async function requestToken() {
        api.log.info(`Requesting new token...`);
        try {

            const response = await api.runtime.getFetchModule().fetch(AUTH_URL, {
                method: 'POST',
                headers: {
                    'Content-Type': 'application/json',
                },
                body: JSON.stringify({
                    username: AUTH_USER,
                    password: AUTH_PASS
                }),
            });

            if (!response.ok)
                throw new Error(`Error fetching token. HTTP response code ${response.status} : ${response.statusText}`);

            const data = (await response.json());

            return data[AUTH_RESPONSE_TOKEN_PROPERTY];
        } catch (e) {
            api.log.error(`Error fetching token. ${e.name}: ${e.message}`);
            throw new Error("Error fetching token.", { cause: e });
        } finally {
            api.log.info(`Requesting new token... DONE`);
        }
    }

    export default {
        version: 6,
        configs: [{
            provider: "openai-compatible",
            models: SERVER_MODELS,
            providerOptions: async () => {
                return {
                    apiKey: await getToken(),
                    baseURL: SERVER_BASE_URL,
                    displayName: PROVIDER_DISPLAY_NAME
                }
            }
        }]
    }

.. note::
    The ``api.runtime.getFetchModule().fetch()`` method takes into consideration the proxies and certificates settings configured using the specific :ref:`environment variables <proxies-certificates-env-vars>`.
 
    You can override these global proxy and certificate settings on a per-request basis by passing additional arguments to ``api.runtime.getFetchModule()``.
    
    These arguments follow the same format as those used by :ref:`predefined connectors <proxies-certificates-predefined-connectors>`.

    .. code-block:: js

        const fetch = api.runtime.getFetchModule({
            httpsProxy: "http://proxy.com:9000"
        }).fetch;

