Conclusion
Current thesis
300 selected cases, 291 graded. Selected evidence supports the company-level thesis (leaning supported). The selected cases support a measurable component: 97% of selected cases link decision, action, outcome, and resolved grade. No applied synthetic mechanism result is attached to this dataset. The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: No material pooling advantage Local minus pooled Brier: 0.003. The modeled mechanism is challenged conditionally: The selected dataset’s 20% pattern-overlap proxy is below the approximately 50% pooling threshold. Conditional experiment boundaries: CHALLENGES: The selected dataset’s 20% pattern-overlap proxy is below the approximately 50% pooling threshold. SUPPORTS: The challenger retains a 0.014 value gap after 500 calibration cases. SUPPORTS: Full history remained competitive throughout the tested drift range. The current evidence suggests possible capability advantage, but not demonstrated durable Power. The strongest current hypothesis is Process Power, but direct evidence remains low. The largest unresolved dependency is Cross-customer transfer.
CE Thesis
PARTIALLY SUPPORTEDEvidence Quality
PARTIALPrimary Power Hypothesis
Process PowerCritical Unresolved Dependency
Cross-customer transferWhere Power may reside
The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: Validation selected the base-rate model; richer historical models produced no validated out-of-sample gain. No applied synthetic mechanism result is attached to this dataset.
The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: 25% of development history reaches within 0.005 Brier of the full-history estimator. The modeled mechanism is supported conditionally: The challenger retains a 0.014 value gap after 500 calibration cases.
No direct unit-cost or fixed-cost leverage evidence is attached.
The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: No material pooling advantage Local minus pooled Brier: 0.003. The modeled mechanism is challenged conditionally: The selected dataset’s 20% pattern-overlap proxy is below the approximately 50% pooling threshold.
Evidence that matters
Supports
291/300 cases graded; 291/300 outcomes represented; median feedback 23 days.
Supports graded experience capture within the selected dataset.
Source: Experience → active CaseSet
View evidence →Contradicts
No direct contradicting evidence is attached yet.
Context
CHALLENGES: The selected dataset’s 20% pattern-overlap proxy is below the approximately 50% pooling threshold. Selective-minus-local value ranges from -0.009 to 0.054 across the tested boundary; material benefit begins around 50% shared structure. Works when: Shared decision structure is at or above roughly 50% under the displayed assumptions. Fails when: Unconditional pooling can destroy value through approximately 25% shared structure.
Treat cross-customer learning as a conditional mechanism with an estimable transfer boundary, not as an automatic consequence of having more customers. Next experiment: Estimate shared structure and segment-level harm on newly held-out customers using the selected dataset’s actual features and actions.
Source: Experience → Automated Experiment Program
View evidence →28/59 decision-class/case-type patterns occur across segments; 225/291 eligible cases belong to shared patterns.
Measures whether segments share potentially reusable decision structure.
Source: Experience → active CaseSet
View evidence →No material pooling advantage. 88 held-out cases; local Brier 0.246, pooled 0.243.
Segment transfer probe v1: earliest 70% train, latest 30% test; only feedback available before each test decision is used. Predict CORRECT versus other resolved grades from decision class and case type. Same fixed smoothing in each arm; lower Brier score is better. A 0.005 Brier difference defines the descriptive effect band, not statistical significance.
Source: Experience → active CaseSet
View evidence →Validation selected base-rate; final-test Brier 0.246 versus base-rate 0.246 on 59 cases. The final-set leader is blended (0.223): the ranking changed across windows. The validation-selected model is retained, not replaced after seeing test outcomes.
Tests whether past experience predicts grades on later cases. It does not measure improvement from choosing different actions.
Source: Experience → active CaseSet
View evidence →Limitations
Grade prediction tested; decision-policy improvement remains untested.
Use identified customers and compare local versus pooled policies on held-out decisions, measuring correctness and economic outcomes.
Source: Experience → action-policy experiment
UNKNOWN
No evidence currently shows that grades alter future model or policy behavior.
Source: Company Model → Learning Loop → UPDATE
UNKNOWN
No evidence currently shows learned improvements reach production.
Source: Company Model → Learning Loop → DEPLOY
No controlled before/after learning comparison.
Compare held-out decisions with and without updates from accumulated grades.
Source: Experience → learning comparison
Experience
A complete scorebook can still fail to create Compounding Expertise if grades do not improve future behavior or if resulting expertise is easy to reproduce.
Inspect Experience →What could break the thesis?
The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: No material pooling advantage Local minus pooled Brier: 0.003. The modeled mechanism is challenged conditionally: The selected dataset’s 20% pattern-overlap proxy is below the approximately 50% pooling threshold.
Best next evidence: Compare held-out customer performance using customer-only history versus pooled cross-customer experience.
Inspect debate →The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: Validation selected the base-rate model; richer historical models produced no validated out-of-sample gain. No applied synthetic mechanism result is attached to this dataset.
Best next evidence: Compare performance before and after incorporating graded cases while controlling for foundation-model changes.
Inspect debate →The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: 25% of development history reaches within 0.005 Brier of the full-history estimator. The modeled mechanism is supported conditionally: The challenger retains a 0.014 value gap after 500 calibration cases.
Best next evidence: Give a challenger policy documentation plus limited calibration data and measure the performance gap versus the incumbent.
Inspect debate →Selected evidence supports the company-level thesis (leaning supported). The selected cases support a measurable component: 97% of selected cases link decision, action, outcome, and resolved grade. No applied synthetic mechanism result is attached to this dataset.
Best next evidence: Inspect incomplete cases and grade reliability by decision class, then test whether using these grades improves held-out decisions.
Inspect debate →What should we diligence next?
Compare performance before and after incorporating graded cases while controlling for foundation-model changes.
No evidence currently shows that grades alter future model or policy behavior.
Compare held-out customer performance using customer-only history versus pooled cross-customer experience.
Use identified customers and compare local versus pooled policies on held-out decisions, measuring correctness and economic outcomes.
Give a challenger policy documentation plus limited calibration data and measure the performance gap versus the incumbent.
Need evidence that a capable challenger cannot reproduce performance cheaply.
Review contractual and data-governance rights for retention, derived features, evaluation, and cross-customer training.
Need retention, derived-feature, evaluation, and cross-customer training rights.
Measure incremental performance gain from successive case cohorts after policy and model baselines are included.
Measure action-policy performance across successive training cohorts.
Stress-test finding
No stress test has been run for this analysis yet.
Inspect Stress Test →Investor view
Cross-customer transfer: 40% investor belief; CE assessment UNPROVEN. Investor belief remains separate from CE evidence. Experience capture: 35% investor belief; CE assessment LEANING SUPPORTED. Your belief is more skeptical than the current evidence model. Note the concern or alternative explanation.
ANALYST BELIEF — NOT CE EVIDENCE