LLM Inference

An OpenAI-compatible chat completions endpoint for teams that want to run a language model against their own prompts through Medsender. One route, one model, and a key you create and revoke yourself.

Get started

  1. Create a developer account and verify your email.
  2. Open LLM Inference, accept the current LLM agreement, and complete the required BAA. Your signer can complete it through DocuSign; no Medsender account review is needed for LLM access.
  3. Create a key in LLM API Keys and save the secret shown once.
  4. Open Billing and add credits through Stripe. Billing setup is automatic. Key creation does not require payment first.
  5. Run the example below with your key. A successful response confirms your first completion.

Production faxing has a separate approval process. Fax approval is not required for LLM inference.

LLM keys make real inference requests and consume credits. Staging and production use separate hosts; there is no simulated or free test-key mode.

Inference requires available credits and an unfrozen billing organization. If a request is refused for insufficient credits, top up in Billing and retry after the balance updates.

Endpoint and authentication

POST /v1/chat/completions is the only route the gateway serves. Every other path is refused at the edge.

Hosts

EnvironmentEndpoint
Productionhttps://llm.medsender.com/v1/chat/completions
Staginghttps://llm-staging.medsender.com/v1/chat/completions

Send your key in the Authorization header using the Bearer scheme. Create keys in the portal at LLM Keys.

This is not a Medsender Developer API key

The two key types are separate. A Developer API key does not work on this endpoint, and an LLM key does not work on the fax, email, or other Developer API endpoints. An LLM key is shown once at creation and stored nowhere, so a lost key is replaced, not recovered.

bash
curl https://llm.medsender.com/v1/chat/completions \
  -H "Authorization: Bearer $MEDSENDER_LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"fast","messages":[{"role":"user","content":"Say hello."}],"max_tokens":32}'
curl https://llm.medsender.com/v1/chat/completions \
  -H "Authorization: Bearer $MEDSENDER_LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"fast","messages":[{"role":"user","content":"Say hello."}],"max_tokens":32}'

Making a request

A complete chat completion: a system message that sets the behavior, a user message that asks the question, and a cap on the generated length.

bash
curl https://llm.medsender.com/v1/chat/completions \
  -H "Authorization: Bearer $MEDSENDER_LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "fast",
  "messages": [
    {
      "role": "system",
      "content": "You are a clinical intake assistant. Answer in one sentence."
    },
    {
      "role": "user",
      "content": "Why would a primary care office send a referral to a cardiologist?"
    }
  ],
  "max_tokens": 256
}'
curl https://llm.medsender.com/v1/chat/completions \
  -H "Authorization: Bearer $MEDSENDER_LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "fast",
  "messages": [
    {
      "role": "system",
      "content": "You are a clinical intake assistant. Answer in one sentence."
    },
    {
      "role": "user",
      "content": "Why would a primary care office send a referral to a cardiologist?"
    }
  ],
  "max_tokens": 256
}'

Using the OpenAI SDK

The endpoint is OpenAI-compatible, so the official openai SDKs work once you override the base URL. Shown for Python and Node/TypeScript, the languages with an official SDK; in any other language, call the endpoint over plain HTTP as above.

Python

Python
# pip install openai
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://llm.medsender.com/v1",
    api_key=os.environ["MEDSENDER_LLM_API_KEY"],
)

completion = client.chat.completions.create(
    model="fast",
    messages=[
        {"role": "system", "content": "You are a clinical intake assistant. Answer in one sentence."},
        {"role": "user", "content": "Why would a primary care office send a referral to a cardiologist?"},
    ],
    max_tokens=256,
)
print(completion.choices[0].message.content)
# pip install openai
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://llm.medsender.com/v1",
    api_key=os.environ["MEDSENDER_LLM_API_KEY"],
)

completion = client.chat.completions.create(
    model="fast",
    messages=[
        {"role": "system", "content": "You are a clinical intake assistant. Answer in one sentence."},
        {"role": "user", "content": "Why would a primary care office send a referral to a cardiologist?"},
    ],
    max_tokens=256,
)
print(completion.choices[0].message.content)

Node and TypeScript

TypeScript
// npm install openai
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://llm.medsender.com/v1',
  apiKey: process.env.MEDSENDER_LLM_API_KEY,
});

const completion = await client.chat.completions.create({
  model: 'fast',
  messages: [
    { role: 'system', content: 'You are a clinical intake assistant. Answer in one sentence.' },
    { role: 'user', content: 'Why would a primary care office send a referral to a cardiologist?' },
  ],
  max_tokens: 256,
});
console.log(completion.choices[0].message.content);
// npm install openai
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://llm.medsender.com/v1',
  apiKey: process.env.MEDSENDER_LLM_API_KEY,
});

const completion = await client.chat.completions.create({
  model: 'fast',
  messages: [
    { role: 'system', content: 'You are a clinical intake assistant. Answer in one sentence.' },
    { role: 'user', content: 'Why would a primary care office send a referral to a cardiologist?' },
  ],
  max_tokens: 256,
});
console.log(completion.choices[0].message.content);

Request schema and limits

The body takes exactly three fields. Any other field is rejected rather than ignored.

FieldRequiredDescription
modelYesMust be "fast". It is the only accepted value.
messagesYesThe conversation, in order. Each entry has exactly role and content: role is system, user, or assistant, and content is a non-empty string.
max_tokensYesMaximum tokens to generate, up to 1,024.

A request is refused when it exceeds any of these:

  • 32 messages per request
  • 8,192 input tokens
  • max_tokens of 1,024
  • 1 MiB of request body (1,048,576 bytes)

Errors

Most failures return the OpenAI-compatible error envelope: an error object with message, type, param, and code. Branch on the status and, where the table names one, on code. Request-schema rejections are the exception and use a different shape — see below.

StatusCodeMeaningWhat to do
401—The key is missing, malformed, revoked, or expired.Check the Authorization header. If the key expired or was revoked, create a new one.
400—The request named a model other than fast, or included an unsupported field.Send "model": "fast" and only model, messages, and max_tokens. Read detail.error for the reason.
408—The upload stalled. The body must arrive within 15 seconds.Send the whole body without pausing, then retry.
413—The request body is over 1 MiB.Shorten or trim the messages.
429gateway_rate_exceededToo many requests this minute.Wait the number of seconds in the Retry-After header, which counts down to the next minute.
429gateway_concurrency_exceededToo many requests in flight at once.Retry shortly, and keep no more than four requests open.
503gateway_admission_unavailableThe gateway could not accept the work.Retry.
504gateway_deadline_exceededThe request passed the 15-second ceiling.Retry. A shorter prompt and a smaller max_tokens finish sooner.

Example error body

JSON
{
  "error": {
    "message": "Rate limit exceeded.",
    "type": "rate_limit_error",
    "param": null,
    "code": "gateway_rate_exceeded"
  }
}
{
  "error": {
    "message": "Rate limit exceeded.",
    "type": "rate_limit_error",
    "param": null,
    "code": "gateway_rate_exceeded"
  }
}

A 400 is shaped differently: the reason is a plain string under detail.error, not an error object.

JSON
{
  "detail": {
    "error": "model must be fast"
  }
}
{
  "detail": {
    "error": "model must be fast"
  }
}

Not supported

These are rejected, not ignored: a request that uses one fails instead of quietly falling back, so do not build against them.

  • Streaming (stream) and stream_options
  • Tools and function calling
  • response_format
  • The user field
  • The Responses API
  • Embeddings

Key lifecycle

Each key carries its own expiry, budget, and throughput. The limits are per key, so splitting traffic across keys is how you separate one workload from another.

  • Expires 30 days after it is created.
  • Spends against a $25 budget.
  • Allows 60 requests per minute, 100,000 tokens per minute, and 4 concurrent requests.
  • Counts toward a ceiling of 50 active keys per account.
  • Revoking takes effect immediately and is permanent. There is no un-revoke, so a revoked key is replaced by a new one.

Create, review, and revoke keys at LLM Keys.