# LLM Inference

An OpenAI-compatible chat completions endpoint for teams that want to run a language model against their own prompts through Medsender. One route, one model, and a key you create and revoke yourself.

## Get started

1. Create a developer account and verify your email.
2. Open [LLM Inference](/llm-api), accept the current LLM agreement, and complete the required BAA. Your signer can complete it through DocuSign; no Medsender account review is needed for LLM access.
3. Create a key in [LLM API Keys](/llm-keys) and save the secret shown once.
4. Open [Billing](/billing) and add credits through Stripe. Billing setup is automatic. Key creation does not require payment first.
5. Run the example below with your key. A successful response confirms your first completion.

Production faxing has a separate approval process. Fax approval is not required for LLM inference.

LLM keys make real inference requests and consume credits. Staging and production use separate hosts; there is no simulated or free test-key mode.

Inference requires available credits and an unfrozen billing organization. If a request is refused for insufficient credits, top up in Billing and retry after the balance updates.

## Endpoint and authentication

POST `/v1/chat/completions` is the only route the gateway serves. Every other path is refused at the edge.

Hosts

| Environment | Endpoint |
| --- | --- |
| Production | https://llm.medsender.com/v1/chat/completions |
| Staging | https://llm-staging.medsender.com/v1/chat/completions |

Send your key in the `Authorization` header using the Bearer scheme. Create keys in the portal at [LLM Keys](/llm-keys).

This is not a Medsender Developer API key

The two key types are separate. A Developer API key does not work on this endpoint, and an LLM key does not work on the fax, email, or other Developer API endpoints. An LLM key is shown once at creation and stored nowhere, so a lost key is replaced, not recovered.

#### cURL

```bash
curl https://llm.medsender.com/v1/chat/completions \
  -H "Authorization: Bearer $MEDSENDER_LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"fast","messages":[{"role":"user","content":"Say hello."}],"max_tokens":32}'
```

#### Node

```javascript
const LLM_BASE='https://llm.medsender.com/v1';
const API_KEY=process.env.MEDSENDER_LLM_API_KEY;

const res = await fetch(`${LLM_BASE}/chat/completions`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    model: 'fast',
    messages: [{ role: 'user', content: 'Say hello.' }],
    max_tokens: 32
  })
});
console.log(res.status, await res.json());
```

#### TypeScript

```typescript
const LLM_BASE: string='https://llm.medsender.com/v1';
const API_KEY: string=process.env.MEDSENDER_LLM_API_KEY ?? '';

const res = await fetch(`${LLM_BASE}/chat/completions`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    model: 'fast',
    messages: [{ role: 'user', content: 'Say hello.' }],
    max_tokens: 32
  })
});
console.log(res.status, await res.json());
```

#### Python

```python
import os, requests
LLM_BASE='https://llm.medsender.com/v1'
API_KEY=os.environ['MEDSENDER_LLM_API_KEY']

r=requests.post(
  f"{LLM_BASE}/chat/completions",
  headers={'Authorization':f'Bearer {API_KEY}','Content-Type':'application/json'},
  json={'model':'fast','messages':[{'role':'user','content':'Say hello.'}],'max_tokens':32},
)
print(r.status_code, r.json())
```

#### Ruby

```ruby
require 'net/http'
require 'json'
require 'uri'
uri = URI.parse('https://llm.medsender.com/v1/chat/completions')
req = Net::HTTP::Post.new(uri)
req['Authorization'] = "Bearer #{ENV.fetch('MEDSENDER_LLM_API_KEY')}"
req['Content-Type'] = 'application/json'
req.body = { model: 'fast', messages: [{ role: 'user', content: 'Say hello.' }], max_tokens: 32 }.to_json
res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |http| http.request(req) }
puts res.code, res.body
```

#### PHP

```php
<?php
$body = json_encode([
  'model' => 'fast',
  'messages' => [['role' => 'user', 'content' => 'Say hello.']],
  'max_tokens' => 32,
]);
$ch = curl_init();
curl_setopt_array($ch, [
  CURLOPT_URL => 'https://llm.medsender.com/v1/chat/completions',
  CURLOPT_POST => true,
  CURLOPT_POSTFIELDS => $body,
  CURLOPT_HTTPHEADER => [
    'Authorization: Bearer ' . getenv('MEDSENDER_LLM_API_KEY'),
    'Content-Type: application/json',
  ],
  CURLOPT_RETURNTRANSFER => true,
]);
$res = curl_exec($ch);
echo curl_getinfo($ch, CURLINFO_HTTP_CODE), "\n", $res;
```

#### Java

```java
import java.net.http.*;
import java.net.URI;
public class LlmAuthHeader {
  public static void main(String[] args) throws Exception {
    var body = """
      {"model":"fast","messages":[{"role":"user","content":"Say hello."}],"max_tokens":32}""";
    var client = HttpClient.newHttpClient();
    var req = HttpRequest.newBuilder()
      .uri(URI.create("https://llm.medsender.com/v1/chat/completions"))
      .header("Authorization","Bearer " + System.getenv("MEDSENDER_LLM_API_KEY"))
      .header("Content-Type","application/json")
      .POST(HttpRequest.BodyPublishers.ofString(body))
      .build();
    var res = client.send(req, HttpResponse.BodyHandlers.ofString());
    System.out.println(res.statusCode());
    System.out.println(res.body());
  }
}
```

#### C#

```csharp
using System;
using System.Net.Http;
using System.Text;
var url = "https://llm.medsender.com/v1/chat/completions";
var body = "{\"model\":\"fast\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hello.\"}],\"max_tokens\":32}";
using var client = new HttpClient();
client.DefaultRequestHeaders.Add("Authorization","Bearer " + Environment.GetEnvironmentVariable("MEDSENDER_LLM_API_KEY"));
var res = await client.PostAsync(url, new StringContent(body, Encoding.UTF8, "application/json"));
Console.WriteLine((int)res.StatusCode);
Console.WriteLine(await res.Content.ReadAsStringAsync());
```

## Making a request

A complete chat completion: a system message that sets the behavior, a user message that asks the question, and a cap on the generated length.

#### cURL

```bash
curl https://llm.medsender.com/v1/chat/completions \
  -H "Authorization: Bearer $MEDSENDER_LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "fast",
  "messages": [
    {
      "role": "system",
      "content": "You are a clinical intake assistant. Answer in one sentence."
    },
    {
      "role": "user",
      "content": "Why would a primary care office send a referral to a cardiologist?"
    }
  ],
  "max_tokens": 256
}'
```

#### Node

```javascript
const LLM_BASE='https://llm.medsender.com/v1';
const API_KEY=process.env.MEDSENDER_LLM_API_KEY;

const res = await fetch(`${LLM_BASE}/chat/completions`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    model: 'fast',
    messages: [
      { role: 'system', content: 'You are a clinical intake assistant. Answer in one sentence.' },
      { role: 'user', content: 'Why would a primary care office send a referral to a cardiologist?' }
    ],
    max_tokens: 256
  })
});

if (!res.ok) {
  const { error } = await res.json();
  throw new Error(`${res.status} ${error.code}: ${error.message}`);
}
console.log(await res.json());
```

#### TypeScript

```typescript
const LLM_BASE: string='https://llm.medsender.com/v1';
const API_KEY: string=process.env.MEDSENDER_LLM_API_KEY ?? '';

const res = await fetch(`${LLM_BASE}/chat/completions`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${API_KEY}`,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    model: 'fast',
    messages: [
      { role: 'system', content: 'You are a clinical intake assistant. Answer in one sentence.' },
      { role: 'user', content: 'Why would a primary care office send a referral to a cardiologist?' }
    ],
    max_tokens: 256
  })
});

if (!res.ok) {
  const { error } = await res.json();
  throw new Error(`${res.status} ${error.code}: ${error.message}`);
}
console.log(await res.json());
```

#### Python

```python
import os, requests
LLM_BASE='https://llm.medsender.com/v1'
API_KEY=os.environ['MEDSENDER_LLM_API_KEY']

r=requests.post(
  f"{LLM_BASE}/chat/completions",
  headers={'Authorization':f'Bearer {API_KEY}','Content-Type':'application/json'},
  json={
    'model':'fast',
    'messages':[
      {'role':'system','content':'You are a clinical intake assistant. Answer in one sentence.'},
      {'role':'user','content':'Why would a primary care office send a referral to a cardiologist?'},
    ],
    'max_tokens':256,
  },
)
r.raise_for_status()
print(r.json())
```

#### Ruby

```ruby
require 'net/http'
require 'json'
require 'uri'
uri = URI.parse('https://llm.medsender.com/v1/chat/completions')
req = Net::HTTP::Post.new(uri)
req['Authorization'] = "Bearer #{ENV.fetch('MEDSENDER_LLM_API_KEY')}"
req['Content-Type'] = 'application/json'
req.body = {
  model: 'fast',
  messages: [
    { role: 'system', content: 'You are a clinical intake assistant. Answer in one sentence.' },
    { role: 'user', content: 'Why would a primary care office send a referral to a cardiologist?' }
  ],
  max_tokens: 256
}.to_json
res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |http| http.request(req) }
raise "#{res.code}: #{res.body}" unless res.is_a?(Net::HTTPSuccess)
puts res.body
```

#### PHP

```php
<?php
$body = json_encode([
  'model' => 'fast',
  'messages' => [
    ['role' => 'system', 'content' => 'You are a clinical intake assistant. Answer in one sentence.'],
    ['role' => 'user', 'content' => 'Why would a primary care office send a referral to a cardiologist?'],
  ],
  'max_tokens' => 256,
]);
$ch = curl_init();
curl_setopt_array($ch, [
  CURLOPT_URL => 'https://llm.medsender.com/v1/chat/completions',
  CURLOPT_POST => true,
  CURLOPT_POSTFIELDS => $body,
  CURLOPT_HTTPHEADER => [
    'Authorization: Bearer ' . getenv('MEDSENDER_LLM_API_KEY'),
    'Content-Type: application/json',
  ],
  CURLOPT_RETURNTRANSFER => true,
]);
$res = curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ($status >= 400) {
  throw new RuntimeException("LLM gateway returned $status: $res");
}
echo $res;
```

#### Java

```java
import java.net.http.*;
import java.net.URI;
public class LlmChatCompletion {
  public static void main(String[] args) throws Exception {
    var body = """
      {
        "model": "fast",
        "messages": [
          {"role": "system", "content": "You are a clinical intake assistant. Answer in one sentence."},
          {"role": "user", "content": "Why would a primary care office send a referral to a cardiologist?"}
        ],
        "max_tokens": 256
      }""";
    var client = HttpClient.newHttpClient();
    var req = HttpRequest.newBuilder()
      .uri(URI.create("https://llm.medsender.com/v1/chat/completions"))
      .header("Authorization","Bearer " + System.getenv("MEDSENDER_LLM_API_KEY"))
      .header("Content-Type","application/json")
      .POST(HttpRequest.BodyPublishers.ofString(body))
      .build();
    var res = client.send(req, HttpResponse.BodyHandlers.ofString());
    if (res.statusCode() >= 400) {
      throw new RuntimeException("LLM gateway returned " + res.statusCode() + ": " + res.body());
    }
    System.out.println(res.body());
  }
}
```

#### C#

```csharp
using System;
using System.Net.Http;
using System.Text;
var url = "https://llm.medsender.com/v1/chat/completions";
var body = """
{
  "model": "fast",
  "messages": [
    {"role": "system", "content": "You are a clinical intake assistant. Answer in one sentence."},
    {"role": "user", "content": "Why would a primary care office send a referral to a cardiologist?"}
  ],
  "max_tokens": 256
}
""";
using var client = new HttpClient();
client.DefaultRequestHeaders.Add("Authorization","Bearer " + Environment.GetEnvironmentVariable("MEDSENDER_LLM_API_KEY"));
var res = await client.PostAsync(url, new StringContent(body, Encoding.UTF8, "application/json"));
var payload = await res.Content.ReadAsStringAsync();
if (!res.IsSuccessStatusCode) {
  throw new HttpRequestException($"LLM gateway returned {(int)res.StatusCode}: {payload}");
}
Console.WriteLine(payload);
```

## Using the OpenAI SDK

The endpoint is OpenAI-compatible, so the official `openai` SDKs work once you override the base URL. Shown for Python and Node/TypeScript, the languages with an official SDK; in any other language, call the endpoint over plain HTTP as above.

Python

```python
# pip install openai
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://llm.medsender.com/v1",
    api_key=os.environ["MEDSENDER_LLM_API_KEY"],
)

completion = client.chat.completions.create(
    model="fast",
    messages=[
        {"role": "system", "content": "You are a clinical intake assistant. Answer in one sentence."},
        {"role": "user", "content": "Why would a primary care office send a referral to a cardiologist?"},
    ],
    max_tokens=256,
)
print(completion.choices[0].message.content)
```

Node and TypeScript

```typescript
// npm install openai
import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://llm.medsender.com/v1',
  apiKey: process.env.MEDSENDER_LLM_API_KEY,
});

const completion = await client.chat.completions.create({
  model: 'fast',
  messages: [
    { role: 'system', content: 'You are a clinical intake assistant. Answer in one sentence.' },
    { role: 'user', content: 'Why would a primary care office send a referral to a cardiologist?' },
  ],
  max_tokens: 256,
});
console.log(completion.choices[0].message.content);
```

## Request schema and limits

The body takes exactly three fields. Any other field is rejected rather than ignored.

| Field | Required | Description |
| --- | --- | --- |
| model | Yes | Must be `"fast"`. It is the only accepted value. |
| messages | Yes | The conversation, in order. Each entry has exactly `role` and `content`: `role` is `system`, `user`, or `assistant`, and `content` is a non-empty string. |
| max_tokens | Yes | Maximum tokens to generate, up to 1,024. |

A request is refused when it exceeds any of these:

- 32 messages per request
- 8,192 input tokens
- `max_tokens` of 1,024
- 1 MiB of request body (1,048,576 bytes)

## Errors

Most failures return the OpenAI-compatible error envelope: an `error` object with `message`, `type`, `param`, and `code`. Branch on the status and, where the table names one, on `code`. Request-schema rejections are the exception and use a different shape — see below.

| Status | Code | Meaning | What to do |
| --- | --- | --- | --- |
| 401 | — | The key is missing, malformed, revoked, or expired. | Check the `Authorization` header. If the key expired or was revoked, create a new one. |
| 400 | — | The request named a model other than `fast`, or included an unsupported field. | Send `"model": "fast"` and only `model`, `messages`, and `max_tokens`. Read `detail.error` for the reason. |
| 408 | — | The upload stalled. The body must arrive within 15 seconds. | Send the whole body without pausing, then retry. |
| 413 | — | The request body is over 1 MiB. | Shorten or trim the messages. |
| 429 | gateway_rate_exceeded | Too many requests this minute. | Wait the number of seconds in the `Retry-After` header, which counts down to the next minute. |
| 429 | gateway_concurrency_exceeded | Too many requests in flight at once. | Retry shortly, and keep no more than four requests open. |
| 503 | gateway_admission_unavailable | The gateway could not accept the work. | Retry. |
| 504 | gateway_deadline_exceeded | The request passed the 15-second ceiling. | Retry. A shorter prompt and a smaller `max_tokens` finish sooner. |

Example error body

```json
{
  "error": {
    "message": "Rate limit exceeded.",
    "type": "rate_limit_error",
    "param": null,
    "code": "gateway_rate_exceeded"
  }
}
```

A `400` is shaped differently: the reason is a plain string under `detail.error`, not an `error` object.

```json
{
  "detail": {
    "error": "model must be fast"
  }
}
```

## Not supported

These are rejected, not ignored: a request that uses one fails instead of quietly falling back, so do not build against them.

- Streaming (`stream`) and `stream_options`
- Tools and function calling
- `response_format`
- The `user` field
- The Responses API
- Embeddings

## Key lifecycle

Each key carries its own expiry, budget, and throughput. The limits are per key, so splitting traffic across keys is how you separate one workload from another.

- Expires 30 days after it is created.
- Spends against a $25 budget.
- Allows 60 requests per minute, 100,000 tokens per minute, and 4 concurrent requests.
- Counts toward a ceiling of 50 active keys per account.
- Revoking takes effect immediately and is permanent. There is no un-revoke, so a revoked key is replaced by a new one.

Create, review, and revoke keys at [LLM Keys](/llm-keys).