How to Route Tab Autocomplete and Chat to Separate Models in Continue

Developers using Continue in VS Code or JetBrains often face an architectural dilemma: keystroke-level tab autocomplete requires near-instantaneous responses (under 300 milliseconds), while complex codebase refactoring and sidebar chat require deep reasoning models with expansive context windows. Forcing a heavyweight reasoning model like Claude 3.5 Sonnet or GPT-4o into tab completion introduces intolerable typing latency and ballooning API costs. Conversely, relying on a lightweight completion model for multi-file architectural planning leads to shallow code analysis.

The solution is role-based model routing. Continue decouples real-time inline completion from exploratory conversational reasoning. This guide demonstrates how to configure Continue to route tab autocomplete and chat requests to separate, specialized models across both the modern config.yaml standard and legacy config.json environments, optimize local and cloud endpoints, and prevent latency bottlenecks.

Architectural Separation: Autocomplete vs. Chat

Tab autocomplete and chat operate under fundamentally divergent performance constraints and model architectures:

By routing these tasks to separate engines, developers achieve local zero-cost, privacy-first typing assist while retaining frontier model intelligence in the sidebar.

Method 1: The Modern Standard (config.yaml)

In modern releases of Continue, configuration is driven by config.yaml located at ~/.continue/config.yaml. Task dispatching is governed by the roles array assigned to each model entry.

To isolate tab autocomplete from chat:

1. Define a high-throughput, low-parameter FIM model with roles: [autocomplete].
2. Define one or more frontier models with roles: [chat, edit].

Production Configuration Example

name: Continue Hybrid Routing
version: 0.0.1
schema: v1

models:
  # 1. Specialized Tab Autocomplete Model (Local via Ollama)
  - name: Qwen2.5 Coder 1.5B (Autocomplete)
    provider: ollama
    model: qwen2.5-coder:1.5b
    roles:
      - autocomplete
    autocompleteOptions:
      debounceDelay: 250
      maxPromptTokens: 1024

  # 2. Frontier Reasoning Model for Chat and Refactoring (Cloud via Anthropic)
  - name: Claude 3.5 Sonnet (Chat & Edit)
    provider: anthropic
    model: claude-3-5-sonnet-latest
    apiKey: YOUR_ANTHROPIC_API_KEY
    roles:
      - chat
      - edit

  # 3. Alternative Cloud Chat Model (OpenAI)
  - name: GPT-4o (Chat)
    provider: openai
    model: gpt-4o
    apiKey: YOUR_OPENAI_API_KEY
    roles:
      - chat

Key Parameters in config.yaml

• roles: Determines where Continue surfaces the model. When autocomplete is listed, Continue uses this model for inline ghost text instead of the sidebar active dropdown. When chat is specified, it populates the conversation model selector.
• autocompleteOptions.debounceDelay: The pause in milliseconds after a keystroke before Continue issues an inference call (recommended between 200ms and 350ms). Setting this properly prevents flooding local GPU queues while actively typing.
• autocompleteOptions.maxPromptTokens: Limits surrounding code context sent to the completion model. Lower token limits keep inference within peak performance thresholds for lightweight hardware.
• Disabling Thinking Mode: If utilizing reasoning models that support chain-of-thought toggles, explicitly add extraBodyProperties: { think: false } under the autocomplete model to prevent hidden token generation from blocking tab suggestions.

Method 2: Legacy Configuration (config.json)

If your team maintains an older deployment using ~/.continue/config.json, the routing architecture relies on a dedicated top-level object named tabAutocompleteModel distinct from the models array.

[!NOTE] config.json and the top-level tabAutocompleteModel key are deprecated in favor of config.yaml. However, Continue maintains backward compatibility during the transition period.

Legacy Configuration Example

{
  "models": [
    {
      "title": "Claude 3.5 Sonnet",
      "provider": "anthropic",
      "model": "claude-3-5-sonnet-latest",
      "apiKey": "YOUR_ANTHROPIC_API_KEY"
    },
    {
      "title": "GPT-4o",
      "provider": "openai",
      "model": "gpt-4o",
      "apiKey": "YOUR_OPENAI_API_KEY"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen2.5-Coder 1.5B",
    "provider": "ollama",
    "model": "qwen2.5-coder:1.5b"
  },
  "tabAutocompleteOptions": {
    "debounceDelay": 250,
    "maxPromptTokens": 1024
  }
}

In this schema:

• Entries inside "models" appear in the GUI chat selector.
• The "tabAutocompleteModel" block handles inline ghost text exclusively.
• Tuning parameters reside under "tabAutocompleteOptions".

Recommended Model Pairings by Scenario

Choosing the right pairing depends on available local compute, latency expectations, and budget.

Scenario A: Hybrid Local/Cloud (Optimal for Most Developers)

• Autocomplete: qwen2.5-coder:1.5b or qwen2.5-coder:7b via local Ollama. Delivers sub-150ms keystroke suggestions at zero API cost.
• Chat: Anthropic claude-3-5-sonnet or OpenAI gpt-4o. Provides deep architectural refactoring and multi-step reasoning in the sidebar.

Scenario B: Full Air-Gapped / Privacy-First Local Setup

• Autocomplete: qwen2.5-coder:1.5b (Q4_K_M or Q8_0) hosted via Ollama or vLLM on GPU.
• Chat: qwen2.5-coder:32b or deepseek-coder-v2:16b hosted locally. Requires 16GB–32GB unified memory or dedicated VRAM.

Scenario C: High-Accuracy Dedicated Cloud Autocomplete

• Autocomplete: Mistral codestral-latest via Mistral API. Purpose-built for 80+ programming languages with fill-in-the-middle support and generous context windows.
• Chat: OpenAI gpt-4o or Anthropic claude-3-5-sonnet.

Endpoint Optimization and Latency Tuning

Routing requests to separate models only solves half the problem; developers must also optimize the underlying inference endpoints to avoid resource contention.

1. Optimize Local Ollama for Concurrent Execution

When running a local autocomplete model while querying chat or edit models, default Ollama settings may queue requests, causing the editor to freeze autocomplete while chat is streaming.

• Set OLLAMA_NUM_PARALLEL=2 (or higher) in your system environment to permit simultaneous autocomplete and chat inferences.
• Set OLLAMA_KEEP_ALIVE=24h to prevent the operating system from unloading the autocomplete model weights from GPU memory during idle breaks. A cold-start reload can cause a 5-to-10 second lag spike on the first keystroke.

2. Configure Debounce and Context Sizing

In fast typing workflows, an aggressive debounce delay triggers speculative completions that are immediately discarded by subsequent keystrokes.

• Fast typists (80+ WPM): Configure debounceDelay: 300 to 350ms.
• Slower typists or precision coding: Configure debounceDelay: 200 to 250ms.
• Keep maxPromptTokens between 1024 and 2048. Inline completion rarely benefits from 16k tokens of codebase context; excess prefix processing increases TTFT linearly.

3. Eliminate IDE Extension Collisions

If inline suggestions fail to appear after configuring roles: [autocomplete]:

• Verify that Enable Tab Autocomplete is toggled on in Continue’s settings.
• Check for conflicting inline completion providers in your IDE (e.g., GitHub Copilot, Supermaven, or native VS Code completions). Running two inline completion engines simultaneously causes race conditions where one suppresses the other.

Summary and Verification

By explicitly assigning the autocomplete and chat roles in Continue’s config.yaml, developers establish a dual-engine workflow: millisecond-level local completions keep developers in flow state while frontier cloud models handle complex software engineering tasks.

To verify your configuration in VS Code or JetBrains:

1. Open the Continue extension panel (Ctrl+L / Cmd+L) and click the gear icon to open configuration.
2. Ensure your chat model appears in the dropdown menu at the bottom of the chat panel.
3. Open a code file, type a partial function signature, and pause for 300 milliseconds. A light-gray ghost text suggestion confirming the local model trigger should appear immediately.

References

• Autocomplete Role in Continue Models | Continue Docs
• Chat Role | Continue Docs
• Continue Autocomplete Setup and Configuration Guide | Continue Docs
• How to Understand Configuration | Continue Docs
• Migrating Config to YAML | Continue Docs
• config.json Reference (Deprecated) | Continue Docs
• Using Ollama with Continue: A Developer's Guide | Continue Docs