Local versus cloud model benchmark: the complete figures

01Performance and repeatability

Source: S1. Reconstructed from flattened screenshot text and its accompanying narrative.

Details that matter

Completion and acceptance are different. DeepSeek completed 18/18 runs even though only 136/144 candidates were accepted. A completed request does not mean every candidate passed validation.

Candidate count is not unique business value. These may include repeated outputs across repeated cases. Do not describe 186 accepted candidates as 186 unique discoveries or customer outcomes.

Repeatability is scoped. The narrative says Qwen was “completely deterministic”. The narrower supported statement is exact repeatability in the reported cases. The metric’s comparison rule, raw string, normalised output, structured fields or semantic equivalence, is not supplied. No universal claim about all Qwen outputs follows from this test.

Grounding precision is a separate measure. The 94.0% numeric grounding precision is not the same as the factual-grounding score in either quality table. Their denominators and scoring procedures are not documented here.

Validation-row ambiguity is preserved. The extracted table includes the header “Fatal validation errors”, omits a clear local numeric cell, and includes “8 duplicates” for DeepSeek. The prose independently says Qwen produced no validator errors. That is the wording we use; we have not silently converted the table into a fully verified error taxonomy.

02Business-value quality

Source: S2. Score units and scale endpoints were not supplied. These are reported score points, without a percent sign and without assuming a 100-point rubric.

The local result was not principally a prose-polish win. Clarity/concision differed by only 0.2 points. The striking gaps were material coverage and prioritisation/usefulness. That supports a statement about preserving decision-relevant information, and it supports no causal claim about why Qwen performed better.

The missed-major-facts measure gives the result an accessible business question: what did the report leave out? Local outputs reportedly missed 3.89 major facts on average compared with 10.54 for cloud outputs.

The local model also generated more unsupported claims per output. The larger model’s lower count is a real advantage and it belongs in the main result, not in a footnote. These are counts per output, not normalised rates. Output lengths and total factual-claim counts are unavailable.

Five-of-six judge direction agreement is not unanimous agreement across all cases. The source does not identify the judges, the ensemble rule, the blinding procedure, or the tie rules.

03Daily-summary quality

Source: S3. Again, values are reported score points, not percentages unless explicitly stated elsewhere.

The business-analysis win does not transfer to every task. Daily-summary results show a small cloud aggregate lead, a split case result, and low reported judge direction agreement. This is a mixed outcome with evaluator uncertainty, not a decisive universal ranking.

Exact repeatability and quality are distinct. Qwen repeated its summary outputs exactly in 12/12 cases, and DeepSeek still had the higher aggregate summary quality score. A repeatable output can be a consistently weaker one.

The two task families are not combined into a single overall score anywhere on this page. Their scoring weights, use-case importance, and aggregation rules are unknown.

04Calculations and reusable data

All calculations use rounded values from the source. The extra decimal places below explain the arithmetic. Copy should retain sensible rounding.

The aggregates as an open CSV

The same figures, machine readable. The CSV excludes the ambiguous validation row. Ties and judge agreement are experiment-level values rather than separate model values: business ties 0 and agreement 5/6; summary ties 1 and agreement 3/12.

Download benchmark-aggregates.csv

task,metric,qwen_local,deepseek_cloud,unit,source
business,successful_runs,18,18,count,S1
business,attempted_runs,18,18,count,S1
business,mean_runtime,347.8,17.6,seconds,S1
business,accepted_candidates,186,136,count,S1
business,generated_candidates,186,144,count,S1
business,exact_repeatability_cases,6,0,count,S1
business,repeatability_cases_total,6,6,count,S1
business,ensemble_overall_quality,77.4,63.9,reported_score,S2
business,factual_grounding,81.3,80.6,reported_score,S2
business,material_coverage,72.4,49.1,reported_score,S2
business,prioritization_usefulness,77.3,56.4,reported_score,S2
business,calibration,76.4,71.5,reported_score,S2
business,clarity_concision,76.7,76.5,reported_score,S2
business,unsupported_claims_per_output,1.78,0.69,count_per_output,S2
business,missed_major_facts_per_output,3.89,10.54,count_per_output,S2
business,case_wins,6,0,count,S2
summary,mean_runtime,32.6,3.4,seconds,S1
summary,required_section_coverage,100,100,percent,S1
summary,numeric_grounding_precision,94.0,93.1,percent,S1
summary,exact_repeatability_cases,12,0,count,S1
summary,repeatability_cases_total,12,12,count,S1
summary,ensemble_overall_quality,77.3,80.0,reported_score,S3
summary,factual_grounding,76.6,81.5,reported_score,S3
summary,material_coverage,79.8,79.5,reported_score,S3
summary,prioritization_usefulness,78.3,79.9,reported_score,S3
summary,calibration,74.3,78.9,reported_score,S3
summary,clarity_concision,79.6,82.5,reported_score,S3
summary,unsupported_claims_per_output,1.44,0.92,count_per_output,S3
summary,missed_major_facts_per_output,0.92,1.25,count_per_output,S3
summary,case_wins,5,6,count,S3

05Claim bank and boundaries

The left column includes claims we do not make. They are published so that nobody has to guess where the boundary sits, and so that a reader can check any sentence we publish elsewhere against this table.

The result in one paragraph

We compared a locally run Qwen3.8 27B model with cloud-hosted DeepSeek-V4-Flash, recorded as 284B, on business-value analysis and daily summaries. Qwen won all six business-analysis cases and scored 77.4 versus 63.9 overall, with stronger material coverage and fewer missed major facts. DeepSeek was about 19.8× faster on business analysis, produced fewer unsupported claims, and scored higher overall on summaries. Our result supports evaluating models against the specific workflow and its trade-offs.

06Missing experiment artifacts and reproducibility fields

The following gaps limit independent verification. They do not erase the recorded result, but they define what this evidence note can and cannot substantiate.

Suggested per-run record format for recovered logs

This is the schema we would run the next benchmark against. Take it.

{
  "experiment_id": null,
  "task_family": null,
  "case_id": null,
  "repeat_index": null,
  "executed_at": null,
  "model_label": null,
  "model_digest": null,
  "deployment": null,
  "hardware_or_endpoint": null,
  "quantization": null,
  "prompt_hash": null,
  "input_hash": null,
  "temperature": null,
  "seed": null,
  "thinking_mode": null,
  "context_limit": null,
  "input_tokens": null,
  "output_tokens": null,
  "runtime_seconds": null,
  "retry_count": null,
  "completed": null,
  "generated_candidates": null,
  "accepted_candidates": null,
  "validation_errors": null,
  "output_hash": null,
  "judge_ids": null,
  "rubric_version": null,
  "quality_scores": null,
  "unsupported_claims": null,
  "missed_major_facts": null,
  "source_artifact": null
}

Null means unknown, not zero. This is a collection template, not an actual run record.