VereskLearn

Module 10 · Beyond retrieval

Lesson 10.3

Switching model providers

Switching providers is safe when every model call goes through one interface, and the eval set decides.

Lesson 62 of 622 min read

The problem

Why it matters

Without it

Model calls are spread across the code. Changing the provider means editing many files and hoping nothing breaks.

With it

One interface hides the provider. A new provider needs one adapter, and the same eval set compares it with the old one.

How it works

  • One function for every model call

    The rest of the code calls one function, complete(messages). It takes a list of messages and returns text. An interface is a fixed contract that every adapter must follow, so callers never need to know which provider answered.

    1. Calling code
    2. complete(messages)
    3. Adapter
    4. Provider
    Callers talk to one interface, and only the adapter talks to a provider.
    type Message = { role: string; content: string };type ModelClient = { complete: (m: Message[]) => string };const alpha: ModelClient = {  complete: () => "Refunds take 5 to 7 days.",};const ask = [{ role: "user", content: "Refund time?" }];console.log(alpha.complete(ask));

    Output

    Refunds take 5 to 7 days.

  • Each provider gets an adapter

    An adapter translates the interface into the request format of one provider. Providers differ in message roles, length limits, and error codes, and the adapter hides those differences. The beta adapter has the same signature, so it can replace alpha. In this lesson both adapters are fake, and each returns a fixed reply.

    alpha adapter

    • Same complete(messages) signature
    • Returns "Refunds take 5 to 7 days."

    beta adapter

    • Same complete(messages) signature
    • Returns "Refunds: 5-7 days."
    Two adapters with the same signature and different replies.

  • A setting picks the adapter

    A setting, such as PROVIDER set to "alpha", picks the adapter when the program starts. Calling code only reads the chosen client, so changing the setting changes no calling code. The example prints the default name, which is alpha.

    SettingAdapter used
    PROVIDER = "alpha"alpha
    PROVIDER = "beta"beta
    Each setting value and the adapter it selects.
    const providers = {  alpha: { name: "alpha" },  beta: { name: "beta" },};const PROVIDER = "alpha";console.log("Default ->", providers[PROVIDER].name);

    Output

    Default -> alpha

  • The same eval runs on each adapter

    Run the same golden set against every adapter, with the same messages. Compare the pass count and the cost of each run, such as 42 of 50 against 39 of 50. The check here is simple: the reply must contain the fact 5 to 7. Record the cost in every eval report, because prices change over time.

    Illustrative pass counts out of 50 for each provider.
    const replies = {  alpha: "Refunds take 5 to 7 days.",  beta: "Refunds: 5-7 days.",};for (const [name, reply] of Object.entries(replies)) {  console.log(name, reply.includes("5 to 7") ? "PASS" : "FAIL");}

    Output

    alpha PASS
beta FAIL

  • Rollback is one setting

    Change the default only when the score and the cost are acceptable. Keep the old adapter, so a rollback is one setting, such as PROVIDER back to "alpha". A cheaper provider that fails the eval is a worse choice, not a bargain.

    1. Switch PROVIDER to beta
    2. Eval fails
    3. PROVIDER back to alpha
    A rollback when the new provider fails the eval.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Writing provider-specific code outside the adapter. A switch then needs edits in many files.
  • Comparing providers with different prompts. Send the same messages, or the test measures the prompts.
  • Switching on price alone. A cheaper provider that fails the eval is a worse choice.

Use it when and skip it when

Is it the right tool?

Use it when

  • Your provider raises prices or has outages, and you need a second option.
  • You want to compare two providers on your own questions.

Skip it when

  • You use one provider and have no plan to change it.
  • The code calls a model in one place only, so a switch is already small.

Words to know

Terms in this lesson

Interface
A fixed set of function names and inputs that the rest of the code relies on.
Adapter
Code that translates one interface into the calls of one provider.
Provider
A company or service that runs models and charges for their use.
Rollback
Going back to the previous setting after a bad switch.
Full code