OpenAI's lab benchmark says 66.7%. That is not two in three

The claim

OpenAI built BioBasicsBench with Ligo, an enzyme-design company. It holds 100 tasks drawn from real experiments: gel images, colony plates, microscopy, and instrument readouts across five areas, from protein methods to molecular cloning. Each task was reviewed by Ligo scientists and an external PhD-level wet-lab reviewer (vendor, via AlphaSignal). The headline: GPT-6 Astra scores 66.7% (vendor, via AlphaSignal and Venture Atlas).

OpenAI's own caveat is the right one: higher scores do not yet mean reliable lab judgements (vendor, via Venture Atlas).

Read every figure here as OpenAI's. OpenAI's post returned an error to our tools twice, so we work from two outlets that relay it; neither says it verified the numbers.

The inputs

All vendor, via AlphaSignal. Scoring is a mean rubric reward with partial credit, five attempts per task (vendor, via AlphaSignal). AlphaSignal reports high reasoning effort; Venture Atlas and DEV Community report the xhigh setting. We cannot settle which from the outlets.

The maths

The two intervals overlap by 8.1 points (our maths). Overlap alone does not prove the models are equal, since a paired test on the same tasks would be sharper, but the table on its own does not show Astra ahead of GPT-6.1 Sol.

The newest model writes about 5.9 times fewer output tokens per task than GPT-5.2 while scoring more than twice as high (our maths). Prices and input tokens are not published, so this is not a cost figure. It does say cost per task moves as much as accuracy across model generations.

Assumptions

• The figures relayed by AlphaSignal match OpenAI's post. Two outlets agree on the 66.7% and the 43.8%; only AlphaSignal gives the full table.
• "Partial credit" means a task can score above zero without a correct final call. We have not seen the rubric.
• The tasks resemble your lab's work. Five domains and 100 tasks is broad, not deep.

The result

66.7% is the average share of rubric credit GPT-6 Astra earned across 100 tasks and five attempts each. It is not the share of tasks it got right, which OpenAI does not report. No human baseline is reported, no score per domain, and only OpenAI models are compared (via AlphaSignal).

The failure examples are the useful part. Astra counted colonies about 12% too high, from missed, merged, and falsely marked colonies, and a model read a z-stack focus change as a biological event (vendor, via AlphaSignal). It also caught a wrong-antibiotic plate that would have invalidated a control (vendor, via AlphaSignal). That is the profile of a reviewer's assistant, not a reviewer.

Where it breaks

• As a success rate. Partial credit hides how many answers were incomplete.
• As a ranking. The top two intervals overlap, and no other vendor's model was tested.
• As a release decision. Without a human baseline there is no bar to clear. Your own scientists are the baseline.

What to measure in a pilot

1. Step 1. Pick 30 to 50 of your own completed analyses with known answers. Owner: lab head.
2. Step 2. Score whole-task correctness, not partial credit. Did the agent reach the right call and the right next step? Owner: lab head with a second scientist.
3. Step 3. Count the errors that would have reached a decision. A miscount that changes colony selection matters more than a wrong label in a note. Owner: lab head.
4. Step 4. Record cost per task alongside accuracy. Tokens and minutes per analysis, per model. Owner: lab informatics.
5. Step 5. Keep a scientist on every call until steps 2 and 3 say otherwise. Owner: R&D director.

Missing

• OpenAI's own post. It returned an error to our tools, so we have not read its wording, rubric, or caveats.
• Whether the 100 tasks are released, and under what licence.
• Task-level completion rates and scores per domain.
• A human baseline and any non-OpenAI model.
• A test by Prestanda. We have not run BioBasicsBench or any lab agent.