PL Book a demo

Obsigen · Measurement · as of 2026-10-22 · 137 reports since Aug 28

Measurement methodology

Every number on this page has a source, a date and a configuration in its footnote. We apply the same principle in the product.

We train GRID on our own test scenes. Customer data is not used for measurement or training - it is stored in the customer's own tenant.

1371

measurement reports since Aug 28. The counter and the date come from the repository at every build of this page.

1Principle

A number without a source and a date does not exist. In the product, numbers taken from files, data and web pages appear in an answer as a component with a reference to their source (file, page, URL, date). On this page, footnotes play that role: each index leads to the evaluation report (name and date), with the configuration that produced the result.

Measurement and GRID training use only our own test scenes. We prepare the battery scenes on which we measure models and train the GRID selection table ourselves. Customer files, conversations and results never enter the battery, the reports or GRID training.

Reports are files in the product repository: goal, configuration, rubric result, run cost, what we changed, what we don't know. We share full versions during deployment talks, once a non-disclosure agreement (NDA) is signed.

2Battery

The battery is a set of test scenes we prepare ourselves - from synthetic data and recordings of public websites (fixtures) - scored by a judge - a model with a rubric, not a human eyeballing it - and checked by a human where the rubric is not enough.

classwhat it checks
K1–K10conversation: identity, sources, refusals, handling uncertainty, “no heavy machinery for a simple hello” (K10: 6/62 - a simple question does not trigger the heavy mode)
Rtone: appreciation, closure, sycophancy, phrase ban list
Ddocuments: specifications, bills of quantities, CAD drawings (Obsigen Magnify), spreadsheets
D-devicethe same with files on the user's device (Forge, Runtime): consent, paths, undo

Counter: 137 measurement reports1 - since the first battery report on Aug 28. The counter and the date come from the repository at every build of this page, not from manual entry.

3Decision rules

  1. Judge noise is 0.1–0.15 → threshold 0.23. A difference below the threshold is not a result - we change nothing because of it.
  2. The default changes only after two paired runs on the same fixtures, with the same judge.
  3. Weak signal → no destructive action. If a measurement is inconclusive, the status quo stays.
  4. “Zero-change” sessions have an identity gate: a refactor must produce the same result byte for byte (snapshot), or it does not go in.
  5. An entry without evidence is not applied: the table from which Obsigen selects the model for a task accepts only rows backed by a report.

4Cost and time as KPIs

Not “works / doesn’t work”, but three numbers per turn: quality from the rubric, model cost and time to answer. We choose the default mode from these three, not from a model ranking. On this site we give cost and time in relative terms - amounts and minutes are in the full reports.

Quality in the battery: Titan 18/20, flagship model 19/20 Quality in the battery / 20 Titan 18/20 flagship model 19/20 Cost per turn: Titan ≈ 10, flagship model 100 Cost per turn index, flagship model = 100 Titan ≈ 10 flagship model 100 Time per turn: Titan ≈ 24, flagship model 100 Time per turn index, flagship model = 100 Titan ≈ 24 flagship model 100
Titan · flagship model - 18/204 vs 19/204 · ≈ 10× lower cost per turn4 · ≈ 4× shorter time per turn4

Titan is the default mode, not a flagship model - because 18/204 versus 19/204 is one point less, in exchange for ≈ 10×4 lower cost and ≈ 4×4 shorter time per turn.

You see the usage of every task in the usage panel. Cost in a measurement is the model cost, not the customer price - how we price.

metricresultcontext
Quality - Titan mode18/204GRID orchestration · open-weight models
Cost - Titan mode≈ 10× lower4relative to a flagship model
Time - Titan mode≈ 4× shorter4relative to a flagship model
Flagship model - reference point19/204a single model, no orchestration
Router - decision agreement39/39 (100%)5automatic choice of work mode, in the test battery
Response tone3.8 → 4.8 / 56factual, no empty compliments
Sycophancy0%6facts over approval, in the test battery
Automated tests828 tests7the whole platform

5Provider health

We measure the rate of failed calls (CFR) separately for each host, model and regime, and separately for what the user sees, per button. Alert thresholds: ≤ 10% raw · ≤ 2% visible8. On top of that, tokens per second per provider - we know when “slow” means the provider, not us.

Fallbacks are in-family (the same model at a different provider), and with a fixed prompt prefix the cache reaches up to 98%9 on a direct connection to a provider in the EEA. Provider churn is a fact; we have a mechanism for it, not hope.

6Engineering gates

Turn-controller refactor: 4,576 → 170 lines10 - with an identity gate: recorded responses before and after the change compared byte for byte in 15 scenarios. Tests: 828 tests7; the Forge extension separately: 37/37 extension tests7.

7Orchestration: the same model, with GRID and without

The research question: what does GRID add if the model is the same? Five very hard programming tasks, three repetitions of each, the same model in both variants - once on its own, once driven by GRID.

We measured it - the same model with Obsigen GRID and without it

measured September 12–13, 2026

Was the work delivered at all?

Runs that ended with a working result

Obsigen GRID 12 of 1511 Model alone 4 of 1511

When both variants deliver a result, the scores are close: 19.6 vs 18.9 / 2011.

Tasks passed

Very hard tasks, full scoring rubric

0 25 50

Model alone 7%11Obsigen GRID 47%11

Cost of one passed task

Index: model alone = 100. Model cost in the measurement, not a customer price

Model alone10011 Obsigen GRID≈ 2611

≈ 3.9× cheaper11 - even though a single run with Obsigen GRID costs 1.8×11 more.

Time per run

Index: model alone = 100. We include it because without it the rest would not be honest

Model alone10011 Obsigen GRID≈ 19011

1.9× slower11 - checking the result takes time.

Source: internal Obsigen measurement, September 12–13, 2026 · the same model in both variants · 5 very hard programming tasks × 3 repetitions (small sample) · difference in mean score: p = 0.00211. Tasks passed: 7% → 47%11.

The difference is not in how the model writes - it is in whether the work gets finished. On easy tasks we measured no significant difference.

Measurement configuration: in the full report, shared under NDA.

Demo · 45 min

See Obsigen on your own example

We show Obsigen live, online, on a document you bring - with no confidential data. Then a quote and deployment on dedicated infrastructure.

Book a demo Let's talk

  1. step 1Demo · 45 minonline, on your example
  2. step 2A one-page quotewithin two business days
  3. step 3Deployment and team starta dedicated tenant and team accounts

Footnotes

  1. build-time counter · 2026-09-28 · evaluation reports since Aug 28
  2. acceptance tests 2026-09-01 · a simple question does not trigger the heavy mode
  3. measurement methodology 2026-09-05 · scoring variance; two paired runs
  4. work-mode measurement 2026-08-31 and 2026-09-02 · Titan mode · GRID orchestration, open-weight models
  5. model selection measurement 2026-09-02 · router decisions matching the expected ones; test battery, internal measurement
  6. response tone measurement 2026-09-04/05 · 2026-09-05 · appreciation with fewer compliments
  7. test reports 2026-09-06/07 · 2026-09-07 · 531 backend + 256 frontend + 37 extension + 4 hub - automated tests
  8. telemetry reports 2026-08-29 and 2026-09-02 · alerts for each provider and model, and for each mode
  9. cache report 2026-09 · cache with a fixed prompt prefix; direct connection to an EEA provider
  10. code review 2026-09-05 · turn controller; responses identical before and after the change in 15 scenarios
  11. summary report 2026-09-13 · runs that ended with a working result - Obsigen GRID