Language · Pharma

Chaperone-SA-1.0: what the content says about each drug

Our commercial model reads pharma and medical content and labels every drug or therapy it names as Positive, Neutral or Negative. One request per document returns one JSON answer, with a label for each drug. It was trained on 49,339 curated pharma and medical documents.

Request access How we tested

For research and commercial-intelligence use. Not for diagnosis or treatment decisions.

Sentiment accuracy

91%

Right on 432 of 473 held-out test labels. The 95% range is 88–94%.

Across content types

87–95%

Accuracy on each of the five content types it was tested on

Negative sentiment

84% / 84%

Precision and recall. Only 45 Negatives in the test, so the 95% range is 71–92%.

Structured output

100%

Every requested drug got a valid label: 0 failures in 1,500 test answers over three full runs

What it reads

The model was trained and tested on five kinds of pharma and medical content. For each document you name the drugs to rate, and it answers with one label per drug.

Conference abstracts

New data as it is first presented

PubMed publications

Peer-reviewed research

ClinicalTrials.gov records

Registered trials and their results

Physicians' social posts

What clinicians say in public

Press and news

Coverage of drugs and therapies

One document + the drugs to rate One request JSON: each drug Positive · Neutral · Negative

Model size

~31 GB in FP8

Small enough to serve from a single datacenter GPU, privately, on your own infrastructure.

Hardware

1× L40S or larger

Runs on one NVIDIA L40S (48 GB) or a bigger card. No third-party API involved.

FP8 vs full precision

Same accuracy

91.5–91.8% on the 473 test labels in FP8, against 91.3% in full precision (bf16).

Against other models

On the same 473 test items, Chaperone-SA-1.0 is 8 to 15 points more accurate than the Qwen, MedGemma and Llama models we tested (76–84%). And it labels every drug in a document in one request, where they were asked about one drug at a time.

Model	Accuracy	Negative precision	Negative recall
Chaperone-SA-1.0 (ours)	91.3%	84%	84%
Qwen3.8-27B	83.5%	100%	13%
Qwen3-32B	83.3%	60%	84%
MedGemma-27BGoogle's medical model	82.9%	79%	58%
Llama-3.1-8B	76.3%	62%	11%

Every other model got long labelling instructions and one request per drug. Qwen3.8-27B's 100% Negative precision comes with 13% recall: it called only 6 items Negative.

Medical knowledge

Chaperone-SA-1.0's scores on standard medical benchmarks, mostly with thinking mode on. The sentiment task itself runs with thinking mode off, so these show what the model knows about medicine, not results of the sentiment task.

Benchmark	What it tests	Chaperone-SA-1.0	Best frontier score we found
MedQA	US medical licensing exam, thinking mode on	94.2	96.5 (o1, top of the Vals.ai leaderboard)
MedMCQA	Indian medical entrance exams, a fixed 1,000-question sample	81.4	83.9 (o1-preview, 2024)
MMLU medical	Six medical subjects from MMLU, step by step (not thinking mode)	94.2	96.0 (GPT-5)
PubMedQA	Yes/no/maybe questions on research abstracts, thinking mode on (79.4 step by step)	80.2	82.0 (GPT-4 with Medprompt prompting, 2023)

We ran Chaperone-SA-1.0 ourselves on the standard test questions. Other models' scores come from public leaderboards and papers with slightly different setups, so treat differences of 1–2 points as a tie. On MedQA, Chaperone-SA-1.0 is about 2 points below the top of the Vals.ai leaderboard (96.5%, last updated April 2026). Sources are listed below.

How we tested

Test set. 475 documents with 500 drug-level labels. The data was split by document, so no test document was seen in training. The model was trained on 49,339 curated pharma and medical documents.

Reference labels. Made blind by Claude (Anthropic's model). The main test is the 473 labels Claude marked high or medium confidence. A second independent Claude pass agreed with the first on 96% of a sample, which is roughly the ceiling for any model on this test. On all 500 labels, low confidence included, the model scores 88.6%.

Settings. Thinking mode off and greedy decoding. Serving in FP8 gave the same results as full precision (bf16): 91.5–91.8% on the 473 labels in FP8, against 91.3% in bf16.

Small samples. Some numbers rest on few items, so we give ranges. There are 45 Negatives. Three content types have only 21–29 labels each: trial records, press, and social posts.

Comparison runs. The comparison models were tested on the same 473 items, each with long labelling instructions and one request per drug.

Medical benchmarks. Chaperone-SA-1.0 was run by us on the standard test questions. Other models' scores come from public sources with slightly different setups, so treat differences of 1–2 points as a tie.

Intended use. For research and commercial-intelligence use. Not a medical device, and not for diagnosis or treatment decisions.

Availability. Chaperone-SA-1.0 is a commercial model. Its weights are not published; access is through us.

Sources for other models' scores

Vals.ai MedQA leaderboard, last updated 16 April 2026: vals.ai/benchmarks/medqa
Wang et al., "Capabilities of GPT-5 on Multimodal Medical Reasoning" (GPT-5 on MMLU medical): arxiv.org/abs/2508.08224
Nori et al., "From Medprompt to o1" (o1-preview on MedMCQA): arxiv.org/abs/2411.03590
Nori et al., "Can Generalist Foundation Models Outcompete Special-Purpose Tuning?" (GPT-4 with Medprompt): arxiv.org/abs/2311.16452

We use cookies

Some cookies are needed for this site to work. We would also like to set analytics cookies to understand how the site is used — but only if you agree. Privacy policy

Strictly necessary

Session and security cookies that keep forms and logins working. These cannot be switched off.

Always on

Analytics

Google Analytics, to count visits and see which pages are read. Off unless you turn it on.

Allow

CHAPERONE AI