TOOL
Key Debates

What would change the thesis?

What am I seeing?

The unresolved propositions with the greatest leverage over whether operating experience becomes durable Compounding Expertise.

Why does it matter?

A company can have abundant cases, fast feedback, and a complete scorebook while still failing to create durable Power.

What should I do?

Compare CE's categorical evidence assessment with your own belief, inspect typed evidence, and identify the next test most likely to change your mind.

Optional. AI suggestions are not evidence. Generating replaces the saved debate suggestions; save any edits first.

Key questions

  1. To what extent can cross-customer learning amplify Casap's competitive power in AI-driven dispute management?
    UNPROVEN · VERY HIGH impact
  2. Do accumulated grades actually improve future decisions?
    UNPROVEN · VERY HIGH impact
  3. Could a capable challenger compress, infer, simulate, or relearn the useful scorebook knowledge?
    UNPROVEN · VERY HIGH impact
  4. Can Casap accumulate graded experience from dispute and fraud workflow data to develop a durable competitive advantage?
    LEANING SUPPORTED · HIGH impact
  5. Can the company legally and operationally retain and exploit the experience?
    UNKNOWN · HIGH impact
  6. Do additional graded cases continue to add useful decision-relevant information?
    UNPROVEN · HIGH impact

Evidence → belief debates

Debate 1 · Cross-customer transfer

To what extent can cross-customer learning amplify Casap's competitive power in AI-driven dispute management?

UNPROVENLOW confidenceVERY HIGH impact
Company evidenceUNPROVENLOW confidence
Modeled mechanismCHALLENGED CONDITIONALLYConditional, not company evidence

Finding · integrated conclusion

The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: No material pooling advantage Local minus pooled Brier: 0.003. The modeled mechanism is challenged conditionally: The selected dataset’s 20% pattern-overlap proxy is below the approximately 50% pooling threshold.

Assessment scope: Experience from one customer improves decisions for other customers.

Why it matters: Cross-customer transfer is central to network-like compounding rather than isolated customer-specific learning.

What the selected cases tell us

  1. Shared patterns: present. 28 of 59 patterns span segments, covering 225 of 291 eligible cases. The pattern excludes segment identity, allowing us to detect reusable structure.
  2. Pooled learning: no material pooling advantage. On 88 held-out cases, local Brier score is 0.246 versus pooled 0.243. Lower is better. This directly tests whether sharing past grades improves prediction of future grades.
  3. Generalization: 2/5 adequately sampled segments benefit; 0 worsen. Segments with fewer than five scored cases remain unresolved. Negative transfer is shown rather than averaged away.
SegmentTest casesLocal BrierPooled BrierOther segments onlyLocal − pooled
Credit Union A250.2650.2620.2950.003
Credit Union B200.2290.2250.2410.004
Credit Union C260.2450.2460.249-0.001
Credit Union D110.2420.2340.1990.008
Credit Union E60.2340.2240.2150.010

Implication: The current estimator has not found a clear pooling advantage. Test richer pre-decision features and segment-aware pooling.

Experiment design and remaining test

Segment transfer probe v1: earliest 70% train, latest 30% test; only feedback available before each test decision is used. Predict CORRECT versus other resolved grades from decision class and case type. Same fixed smoothing in each arm; lower Brier score is better. A 0.005 Brier difference defines the descriptive effect band, not statistical significance.

Groups are customer segments, not identified customers. This tests transfer of grade prediction; improving chosen actions requires an action-policy experiment.

203 training cases; 88 held-out candidates; 9 rows excluded for missing segment, case type, resolved grade, or valid timestamps. Reported differences are descriptive; no significance claim is made.

Thesis being tested

Experience from one customer improves decisions for other customers.

Subclaims: what must be true

Each subclaim is resolved separately against the selected cases, conditional experiments, or required company evidence.

  1. Customers share decision-relevant patterns.CHALLENGED BY SELECTED CASES

    28/59 patterns cross segments; 0/5 evaluated segments show material negative transfer. The positive pooled-learning sub-thesis fails the pre-set 0.005 Brier materiality threshold in this dataset.

  2. Pooled experience beats customer-only history on held-out customers.CHALLENGED BY SELECTED CASES

    No material pooling advantage Local minus pooled Brier: 0.003.

  3. The benefit generalizes beyond one customer or segment without material negative transfer.CHALLENGED BY SELECTED CASES

    Selected-case negative-transfer check: 0 evaluated segments show material harm. The selected dataset’s 20% pattern-overlap proxy is below the approximately 50% pooling threshold.

What evidence exists?

Supports the thesis 0

No supporting evidence is attached to this proposition.

Challenges the thesis 0

No contradicting evidence is attached.

What the cases show 3

CHALLENGES: The selected dataset’s 20% pattern-overlap proxy is below the approximately 50% pooling threshold. Selective-minus-local value ranges from -0.009 to 0.054 across the tested boundary; material benefit begins around 50% shared structure. Works when: Shared decision structure is at or above roughly 50% under the displayed assumptions. Fails when: Unconditional pooling can destroy value through approximately 25% shared structure.

Meaning: Treat cross-customer learning as a conditional mechanism with an estimable transfer boundary, not as an automatic consequence of having more customers. Next experiment: Estimate shared structure and segment-level harm on newly held-out customers using the selected dataset’s actual features and actions.

Limit: Automated sensitivity results are conditional simulations calibrated from the selected dataset’s size, group structure, missing feedback, and descriptive time shift. They identify mechanism boundaries; they do not convert simulated outcomes into observed company evidence.

Experience → Automated Experiment Program

2 more in the full evidence ledger below.

What is still lacking?

  1. Local versus pooled customer-level held-out policy comparison and segment harm analysis.

    Use identified customers and compare local versus pooled policies on held-out decisions, measuring correctness and economic outcomes.

Next decisive test

Compare held-out customer performance using customer-only history versus pooled cross-customer experience.

Decision value: Use identified customers and compare local versus pooled policies on held-out decisions, measuring correctness and economic outcomes.

EXECUTABLE NEXT TEST

Test whether a changed action policy improves outcomes

Compare held-out customer performance using customer-only history versus pooled cross-customer experience.

Configure experiment
Evidence detailsInspect measurements, provenance, and original sources.

What the current evidence says

Cross-customer transfer evidence dashboard

DATA

Segment diversity is context, not proof. The adapter tests whether pooled segment experience beats local history on later selected cases, then keeps customer-level action-policy claims separate.

Customer segment context

Multiple segments create an opportunity to test transfer, but do not establish it.

Customer segments

5

300 active CaseSet cases

Cross-customer pooling architecture

UNKNOWN

Learning rights state

UNKNOWN

Segment transfer verdict

CHALLENGES

No material pooling advantage Local minus pooled Brier: 0.003.

Credit Union A85 cases · 83 graded · 81% correct/partial
Credit Union C79 cases · 76 graded · 86% correct/partial
Credit Union B59 cases · 57 graded · 88% correct/partial
Credit Union D48 cases · 48 graded · 85% correct/partial
Credit Union E29 cases · 27 graded · 96% correct/partial

Transfer test availability

Groups are customer segments, not identified customers. This tests transfer of grade prediction; improving chosen actions requires an action-policy experiment.

Held-out pooled-vs-local result

No material pooling advantage Local minus pooled Brier: 0.003.

n=88

Customer-level action-policy result

Not available

External / market evidence

Separate from CaseSet
Casap homepage / product materialsCONTEXT

Purpose-built AI agents and dispute-management software for banks, credit unions, and fintechs; public materials describe dispute model/API, win-score prediction, fraud functionality, and built-in Reg E, Reg Z, Nacha, and card-network expertise.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Raises $25M Series ACONTEXT

$25M Series A announced August 7, 2025; $33.5M total funding; AI agents analyze evidence, predict outcomes, and automate lifecycle from intake to chargeback filing; proprietary first-party fraud score; company-reported customer results.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Chartway customer storyCONTEXT

Company-published customer story reports estimated $875K annual savings, resolution time improved from 90 days to 12 days, 95% chargeback win rate, writeoffs down 72%, costs down 85%, and monthly capacity from 1,200 to 4,000 transactions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap MidSouth customer storyCONTEXT

Company-published customer story reports fraud loss reduced 51%, cost per dispute reduced 90%+ from $37, chargeback floor reduced 85%, real-time merchant-response/chargeback tracking, and Visa transaction metadata used in fraud investigation.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Filene Research Institute, 2025 FiLab Results: CasapCONTEXT

Filene Research Institute public report summary describes testing Casap's fraud dispute platform in sandbox and live environments, with participating credit unions reporting usability, reduced manual effort, and confidence supporting compliance/dispute outcomes.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Filene blog: Chartway Credit Union saved an estimated $875K in one year testing CasapCONTEXT

Filene reports testing Casap with five credit unions; one institution had been live on the platform for more than a year; staff reported a 122% improvement in ease of use versus prior systems; reported dispute-processing error frequency declined 63%.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap October 2024 funding announcementCONTEXT

Company funding announcement claims Casap's platform becomes more effective at fraud detection with customer interaction.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Privacy PolicyCONTEXT

Privacy policy says Casap may use collected information to develop/improve products and conduct analytics, including aggregated non-specific information. It also says third-party personal data uploaded through the service is processed on behalf of customers and according to their instructions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Primary Venture Partners insightsCONTEXT

Primary Venture Partners public materials identify Casap in its portfolio/insights context and state Primary led the seed round and invested in every round.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
View all sources and evidence

Evidence that supports

No evidence in this category yet.

Evidence that contradicts / alternatives

No evidence in this category yet.

Context, not proof

CONTEXT · CONTEXT

Experience → Automated Experiment Program
DERIVED — SYNTHETIC EXPERIMENT

Value / observation: CHALLENGES: The selected dataset’s 20% pattern-overlap proxy is below the approximately 50% pooling threshold. Selective-minus-local value ranges from -0.009 to 0.054 across the tested boundary; material benefit begins around 50% shared structure. Works when: Shared decision structure is at or above roughly 50% under the displayed assumptions. Fails when: Unconditional pooling can destroy value through approximately 25% shared structure.

Interpretation: Treat cross-customer learning as a conditional mechanism with an estimable transfer boundary, not as an automatic consequence of having more customers. Next experiment: Estimate shared structure and segment-level harm on newly held-out customers using the selected dataset’s actual features and actions.

Limitation: Automated sensitivity results are conditional simulations calibrated from the selected dataset’s size, group structure, missing feedback, and descriptive time shift. They identify mechanism boundaries; they do not convert simulated outcomes into observed company evidence.

Inspect source

CONTEXT · CONTEXT

Experience → active CaseSet
DERIVED — SYNTHETIC SIMULATION

Value / observation: 28/59 decision-class/case-type patterns occur across segments; 225/291 eligible cases belong to shared patterns.

Interpretation: Measures whether segments share potentially reusable decision structure.

Limitation: Shared structure is a prerequisite for transfer, not its performance result.

Inspect source

CONTEXT · CONTEXT

Experience → active CaseSet
DERIVED — SYNTHETIC SIMULATION

Value / observation: No material pooling advantage. 88 held-out cases; local Brier 0.246, pooled 0.243.

Interpretation: Segment transfer probe v1: earliest 70% train, latest 30% test; only feedback available before each test decision is used. Predict CORRECT versus other resolved grades from decision class and case type. Same fixed smoothing in each arm; lower Brier score is better. A 0.005 Brier difference defines the descriptive effect band, not statistical significance.

Limitation: Groups are customer segments, not identified customers. This tests transfer of grade prediction; improving chosen actions requires an action-policy experiment.

Inspect source

Missing evidence

MISSING · MISSING

Experience → action-policy experiment
UNKNOWN / DILIGENCE REQUIRED

Value / observation: Grade prediction tested; decision-policy improvement remains untested.

Interpretation: Use identified customers and compare local versus pooled policies on held-out decisions, measuring correctness and economic outcomes.

Limitation: Missing evidence cannot support the proposition.

Obtain via: Policy experiment

Expected evidence: Local versus pooled customer-level held-out policy comparison and segment harm analysis.

Argument & implicationsThe case for and against, its consequences, and what would resolve it.

Case for

If Casap has learning rights and operational processes that aggregate experience across diverse customers effectively, it may generate generalized fraud/dispute patterns and insights that individual competitors with fewer clients cannot achieve, enhancing durability.

Case against

Unknown contractual learning rights and customer heterogeneity imply Casap may be legally or technically constrained from pooling data across clients, or data differences may inhibit meaningful cross-customer transfer, reducing potential learning-scale advantages.

If true

Additional customers may generate experience that improves value for other customers, making Network Economies more plausible.

If false

Expertise may remain customer-specific; CE may exist locally, but the cross-customer compounding mechanism weakens.

What would resolve this?

What would increase belief?

Clear rights and technical evidence of effective cross-customer learning; demonstration that pooled data improves predictive accuracy and robustness.

What would decrease belief?

Legal or operational limitations on cross-customer data use, or data heterogeneity causing poor model transferability.

Next decisive test: Compare held-out customer performance using customer-only history versus pooled cross-customer experience.

Your assessmentYour recorded belief and working notes, separate from evidence.
40%AI-suggested; review before adopting

Belief remains separate from the CE evidence assessment.

Use Edit debate to change and save your belief.

Debate 2 · Learning causality

Do accumulated grades actually improve future decisions?

UNPROVENLOW confidenceVERY HIGH impact
Company evidenceUNPROVENLOW confidence
Modeled mechanismNOT TESTEDConditional, not company evidence

Finding · integrated conclusion

The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: Validation selected the base-rate model; richer historical models produced no validated out-of-sample gain. No applied synthetic mechanism result is attached to this dataset.

Assessment scope: Learning from graded experience causes better future decisions.

Why it matters: A scorebook only matters strategically if grades change future behavior and improve decisions.

Thesis being tested

Learning from graded experience causes better future decisions.

Subclaims: what must be true

Each subclaim is resolved separately against the selected cases, conditional experiments, or required company evidence.

  1. Outcome grades inform identifiable model or policy changes.REQUIRES EXTERNAL / COMPANY EVIDENCE

    Cases do not identify which grades changed a model or policy version.

  2. Those changes reach production and improve held-out decisions.CHALLENGED BY SELECTED CASES

    Validation selected the base-rate model; richer historical models produced no validated out-of-sample gain. This tests later-case prediction, not deployed action improvement.

  3. The improvement is attributable to experience, not a newer base model or an easier case mix.REQUIRES EXTERNAL / COMPANY EVIDENCE

    Attribution requires a versioned, randomized, or otherwise controlled comparison.

What evidence exists?

Supports the thesis 0

No supporting evidence is attached to this proposition.

Challenges the thesis 0

No contradicting evidence is attached.

What the cases show 3

Validation selected base-rate; final-test Brier 0.246 versus base-rate 0.246 on 59 cases. The final-set leader is blended (0.223): the ranking changed across windows. The validation-selected model is retained, not replaced after seeing test outcomes.

Meaning: Tests whether past experience predicts grades on later cases. It does not measure improvement from choosing different actions.

Limit: Compare decisions made with and without the selected learner on a new evaluation batch.

Experience → active CaseSet

2 more in the full evidence ledger below.

What is still lacking?

  1. Model or policy update record tied to graded outcomes.

    No evidence currently shows that grades alter future model or policy behavior.

  2. Deployment history showing learned changes reached production.

    No evidence currently shows learned improvements reach production.

1 more gaps in the full evidence ledger.

Next decisive test

Compare performance before and after incorporating graded cases while controlling for foundation-model changes.

Decision value: No evidence currently shows that grades alter future model or policy behavior.

EXECUTABLE NEXT TEST

Test whether a changed action policy improves outcomes

Compare performance before and after incorporating graded cases while controlling for foundation-model changes.

Configure experiment
Evidence detailsInspect measurements, provenance, and original sources.

What the current evidence says

Learning causality evidence dashboard

DATA

Cases and grades are prerequisites, not proof that learning caused future performance improvement.

Scorebook prerequisites

Graded cases

291

291/300

Grade coverage

97%

291/300

Human override rate

17%

51/300

Model/policy versions represented

Not available
CORRECT175 cases
PARTIALLY_CORRECT74 cases
INCORRECT42 cases
UNRESOLVED9 cases

Causality test availability

Cases + grades ≠ evidence that learning caused future performance improvement.

Update state

UNKNOWN

Deployment state

UNKNOWN

Chronological prediction result

CHALLENGES

Validation selected the base-rate model; richer historical models produced no validated out-of-sample gain.

Before/after or treatment comparison

Not available

Performance by model/policy version

Not available

External / market evidence

Separate from CaseSet
Casap homepage / product materialsCONTEXT

Purpose-built AI agents and dispute-management software for banks, credit unions, and fintechs; public materials describe dispute model/API, win-score prediction, fraud functionality, and built-in Reg E, Reg Z, Nacha, and card-network expertise.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Raises $25M Series ACONTEXT

$25M Series A announced August 7, 2025; $33.5M total funding; AI agents analyze evidence, predict outcomes, and automate lifecycle from intake to chargeback filing; proprietary first-party fraud score; company-reported customer results.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Chartway customer storyCONTEXT

Company-published customer story reports estimated $875K annual savings, resolution time improved from 90 days to 12 days, 95% chargeback win rate, writeoffs down 72%, costs down 85%, and monthly capacity from 1,200 to 4,000 transactions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap MidSouth customer storyCONTEXT

Company-published customer story reports fraud loss reduced 51%, cost per dispute reduced 90%+ from $37, chargeback floor reduced 85%, real-time merchant-response/chargeback tracking, and Visa transaction metadata used in fraud investigation.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Filene Research Institute, 2025 FiLab Results: CasapCONTEXT

Filene Research Institute public report summary describes testing Casap's fraud dispute platform in sandbox and live environments, with participating credit unions reporting usability, reduced manual effort, and confidence supporting compliance/dispute outcomes.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Filene blog: Chartway Credit Union saved an estimated $875K in one year testing CasapCONTEXT

Filene reports testing Casap with five credit unions; one institution had been live on the platform for more than a year; staff reported a 122% improvement in ease of use versus prior systems; reported dispute-processing error frequency declined 63%.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap October 2024 funding announcementCONTEXT

Company funding announcement claims Casap's platform becomes more effective at fraud detection with customer interaction.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Privacy PolicyCONTEXT

Privacy policy says Casap may use collected information to develop/improve products and conduct analytics, including aggregated non-specific information. It also says third-party personal data uploaded through the service is processed on behalf of customers and according to their instructions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
View all sources and evidence

Evidence that supports

No evidence in this category yet.

Evidence that contradicts / alternatives

No evidence in this category yet.

Context, not proof

CONTEXT · CONTEXT

Experience → active CaseSet
DERIVED — SYNTHETIC SIMULATION

Value / observation: Validation selected base-rate; final-test Brier 0.246 versus base-rate 0.246 on 59 cases. The final-set leader is blended (0.223): the ranking changed across windows. The validation-selected model is retained, not replaced after seeing test outcomes.

Interpretation: Tests whether past experience predicts grades on later cases. It does not measure improvement from choosing different actions.

Limitation: Compare decisions made with and without the selected learner on a new evaluation batch.

Inspect source

CONTEXT · CONTEXT

Experience → active CaseSet
DERIVED — SYNTHETIC SIMULATION

Value / observation: Correct-grade rate: early 0.621, late 0.576.

Interpretation: Descriptive time shift may reflect case mix, policy or grading changes; it is not attributed to learning.

Limitation: Compare matched decision types across time and identify policy/model versions.

Inspect source

CONTEXT · CONTEXT

Experience → active CaseSet
DERIVED — SYNTHETIC SIMULATION

Value / observation: 291/300 graded cases.

Interpretation: Grades exist to learn from if connected to updates.

Limitation: Grade coverage does not establish that grades improve future decisions.

Inspect source

Missing evidence

MISSING · MISSING

Company Model → Learning Loop → UPDATE
UNKNOWN / DILIGENCE REQUIRED

Value / observation: UNKNOWN

Interpretation: No evidence currently shows that grades alter future model or policy behavior.

Limitation: Missing evidence cannot support the proposition.

Obtain via: Product/process diligence

Expected evidence: Model or policy update record tied to graded outcomes.

MISSING · MISSING

Company Model → Learning Loop → DEPLOY
UNKNOWN / DILIGENCE REQUIRED

Value / observation: UNKNOWN

Interpretation: No evidence currently shows learned improvements reach production.

Limitation: Missing evidence cannot support the proposition.

Obtain via: Release/process diligence

Expected evidence: Deployment history showing learned changes reached production.

MISSING · MISSING

Experience → learning comparison
UNKNOWN / DILIGENCE REQUIRED

Value / observation: No controlled before/after learning comparison.

Interpretation: Compare held-out decisions with and without updates from accumulated grades.

Limitation: Missing evidence cannot support the proposition.

Obtain via: Experiment

Expected evidence: Versioned, held-out performance before and after learning from these cases.

Argument & implicationsThe case for and against, its consequences, and what would resolve it.

Case for

No bull case supplied yet.

Case against

No bear case supplied yet.

If true

The scorebook becomes a plausible mechanism for improving future decisions.

If false

The dataset may be useful for reporting, but it does not establish Compounding Expertise.

What would resolve this?

What would increase belief?

A measured performance lift appears after grade-driven updates are deployed.

What would decrease belief?

Performance does not improve after updates, or gains are explained by non-scorebook factors.

Next decisive test: Compare performance before and after incorporating graded cases while controlling for foundation-model changes.

Your assessmentYour recorded belief and working notes, separate from evidence.
Not setNo belief recorded

Belief remains separate from the CE evidence assessment.

Use Edit debate to change and save your belief.

Debate 3 · Rebuildability / compression

Could a capable challenger compress, infer, simulate, or relearn the useful scorebook knowledge?

UNPROVENLOW confidenceVERY HIGH impact
Company evidenceUNPROVENLOW confidence
Modeled mechanismSUPPORTED CONDITIONALLYConditional, not company evidence

Finding · integrated conclusion

The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: 25% of development history reaches within 0.005 Brier of the full-history estimator. The modeled mechanism is supported conditionally: The challenger retains a 0.014 value gap after 500 calibration cases.

Assessment scope: A capable challenger cannot cheaply reconstruct the useful expertise.

Why it matters: Valuable learning is not durable Power if competitors can cheaply reproduce it.

Thesis being tested

A capable challenger cannot cheaply reconstruct the useful expertise.

Subclaims: what must be true

Each subclaim is resolved separately against the selected cases, conditional experiments, or required company evidence.

  1. The incumbent has a measurable performance advantage.MIXED / INCOMPLETE

    The observed holdout measures predictive performance, but no external challenger benchmark establishes an incumbent gap.

  2. Public data, compressed rules, simulation, and limited calibration do not close the gap cheaply.CHALLENGED BY SELECTED CASES

    25% of development history reaches within 0.005 Brier of the full-history estimator. The challenger retains a 0.014 value gap after 500 calibration cases.

  3. The time or cost to catch up is material, even as foundation models improve.SUPPORTED CONDITIONALLY

    The incumbent gap changes from 0.023 with no calibration to 0.014 at 500 cases.

What evidence exists?

Supports the thesis 0

No supporting evidence is attached to this proposition.

Challenges the thesis 0

No contradicting evidence is attached.

What the cases show 3

Reconstruction has not been tested: no challenger with restricted data access or a calibration budget has been evaluated.

Meaning: The grade-prediction models share historical data access. Their scores cannot establish how cheaply a competitor could reproduce the capability.

Limit: Compare against a challenger restricted to public features, rules and a fixed calibration budget.

Experience → active CaseSet

2 more in the full evidence ledger below.

What is still lacking?

  1. Incumbent-vs-challenger performance gap after policy docs, public data, synthetic cases, and limited calibration.

    Need evidence that a capable challenger cannot reproduce performance cheaply.

Next decisive test

Give a challenger policy documentation plus limited calibration data and measure the performance gap versus the incumbent.

Decision value: Need evidence that a capable challenger cannot reproduce performance cheaply.

Evidence detailsInspect measurements, provenance, and original sources.

What the current evidence says

Rebuildability / compression evidence dashboard

DATA

Shows whether durable separation has been empirically tested against challengers, simulation, or relearning.

Rebuildability context

Historical case volume

300

Foundation-model substitution

UNKNOWN

Competitor relearning difficulty

UNKNOWN

History-compression proxy

CHALLENGES

25% of development history reaches within 0.005 Brier of the full-history estimator.

Challenger benchmark

Not available
Synthetic simulation300 cases

Empirical test state

A competitor-access challenger has not been empirically tested. This is an internal history-ablation proxy, not a competitor-access, public-data, or calibration-budget challenger test.

Observed proxy

25% of development history reaches within 0.005 Brier of the full-history estimator.

n=59

Competitor-access calibration experiment

Not available

External / market evidence

Separate from CaseSet
Casap homepage / product materialsCONTEXT

Purpose-built AI agents and dispute-management software for banks, credit unions, and fintechs; public materials describe dispute model/API, win-score prediction, fraud functionality, and built-in Reg E, Reg Z, Nacha, and card-network expertise.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Raises $25M Series ACONTEXT

$25M Series A announced August 7, 2025; $33.5M total funding; AI agents analyze evidence, predict outcomes, and automate lifecycle from intake to chargeback filing; proprietary first-party fraud score; company-reported customer results.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Chartway customer storyCONTEXT

Company-published customer story reports estimated $875K annual savings, resolution time improved from 90 days to 12 days, 95% chargeback win rate, writeoffs down 72%, costs down 85%, and monthly capacity from 1,200 to 4,000 transactions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap MidSouth customer storyCONTEXT

Company-published customer story reports fraud loss reduced 51%, cost per dispute reduced 90%+ from $37, chargeback floor reduced 85%, real-time merchant-response/chargeback tracking, and Visa transaction metadata used in fraud investigation.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Filene Research Institute, 2025 FiLab Results: CasapCONTEXT

Filene Research Institute public report summary describes testing Casap's fraud dispute platform in sandbox and live environments, with participating credit unions reporting usability, reduced manual effort, and confidence supporting compliance/dispute outcomes.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Filene blog: Chartway Credit Union saved an estimated $875K in one year testing CasapCONTEXT

Filene reports testing Casap with five credit unions; one institution had been live on the platform for more than a year; staff reported a 122% improvement in ease of use versus prior systems; reported dispute-processing error frequency declined 63%.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Privacy PolicyCONTEXT

Privacy policy says Casap may use collected information to develop/improve products and conduct analytics, including aggregated non-specific information. It also says third-party personal data uploaded through the service is processed on behalf of customers and according to their instructions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
View all sources and evidence

Evidence that supports

No evidence in this category yet.

Evidence that contradicts / alternatives

No evidence in this category yet.

Context, not proof

CONTEXT · CONTEXT

Experience → active CaseSet
DERIVED — SYNTHETIC SIMULATION

Value / observation: Reconstruction has not been tested: no challenger with restricted data access or a calibration budget has been evaluated.

Interpretation: The grade-prediction models share historical data access. Their scores cannot establish how cheaply a competitor could reproduce the capability.

Limitation: Compare against a challenger restricted to public features, rules and a fixed calibration budget.

Inspect source

CONTEXT · CONTEXT

Experience → Automated Experiment Program
DERIVED — SYNTHETIC EXPERIMENT

Value / observation: SUPPORTS: The challenger retains a 0.014 value gap after 500 calibration cases. The incumbent gap changes from 0.023 with no calibration to 0.014 at 500 cases. Works when: The challenger remains below the incumbent throughout the tested budget. Fails when: The failure boundary exceeds the tested 500-case budget.

Interpretation: The simulated experience advantage resists the tested calibration budget, but public priors and richer challenger strategies remain untested. Next experiment: Run a restricted-access challenger on actual policy artifacts, public data, synthetic cases, and a fixed labeled calibration budget.

Limitation: Automated sensitivity results are conditional simulations calibrated from the selected dataset’s size, group structure, missing feedback, and descriptive time shift. They identify mechanism boundaries; they do not convert simulated outcomes into observed company evidence.

Inspect source

CONTEXT · CONTEXT

Company Model → Competitive Architecture
ANALYST ASSUMPTION

Value / observation: Relearning difficulty: UNKNOWN

Interpretation: Frames the rebuildability hypothesis.

Limitation: Assumption is not a challenger benchmark.

Inspect source

Missing evidence

MISSING · MISSING

Future challenger benchmark
UNKNOWN / DILIGENCE REQUIRED

Value / observation: No compression/relearning benchmark.

Interpretation: Need evidence that a capable challenger cannot reproduce performance cheaply.

Limitation: Missing evidence cannot support the proposition.

Obtain via: Benchmark

Expected evidence: Incumbent-vs-challenger performance gap after policy docs, public data, synthetic cases, and limited calibration.

Argument & implicationsThe case for and against, its consequences, and what would resolve it.

Case for

No bull case supplied yet.

Case against

No bear case supplied yet.

If true

If hard to rebuild, historical and ongoing experience may create durable separation.

If false

The scorebook may be valuable operationally but unlikely to constitute durable Power.

What would resolve this?

What would increase belief?

A challenger remains materially behind after access to policy docs, public data, synthetic cases, and limited calibration.

What would decrease belief?

A challenger reaches similar performance with compressed rules, simulated examples, or short relearning.

Next decisive test: Give a challenger policy documentation plus limited calibration data and measure the performance gap versus the incumbent.

Your assessmentYour recorded belief and working notes, separate from evidence.
Not setNo belief recorded

Belief remains separate from the CE evidence assessment.

Use Edit debate to change and save your belief.

Debate 4 · Experience capture

Can Casap accumulate graded experience from dispute and fraud workflow data to develop a durable competitive advantage?

LEANING SUPPORTEDMEDIUM confidenceHIGH impact
Company evidenceLEANING SUPPORTEDMEDIUM confidence
Modeled mechanismNOT TESTEDConditional, not company evidence

Finding · integrated conclusion

Selected evidence supports the company-level thesis (leaning supported). The selected cases support a measurable component: 97% of selected cases link decision, action, outcome, and resolved grade. No applied synthetic mechanism result is attached to this dataset.

Assessment scope: The selected cases contain linked, graded experience available for learning.

Why it matters: Compounding Expertise requires a scorebook-like loop, not merely activity or stored data.

Thesis being tested

The selected cases contain linked, graded experience available for learning.

Subclaims: what must be true

Each subclaim is resolved separately against the selected cases, conditional experiments, or required company evidence.

  1. Decisions and actions can be linked to their outcomes.SUPPORTED BY SELECTED CASES

    97% of selected cases link decision, action, outcome, and resolved grade.

  2. Outcomes receive meaningful, reliable grades.MIXED / INCOMPLETE

    Resolved grades are present, but reliability still requires independent regrading.

  3. Coverage across the selected cases is sufficient to evaluate the loop.SUPPORTED BY SELECTED CASES

    The selected dataset is sufficiently complete to analyze the operating feedback loop.

What evidence exists?

Supports the thesis 1

291/300 cases graded; 291/300 outcomes represented; median feedback 23 days.

Meaning: Supports graded experience capture within the selected dataset.

Limit: Still does not prove that captured grades improve future behavior.

Experience → active CaseSet

Challenges the thesis 0

No contradicting evidence is attached.

What the cases show 1

291/300 graded cases; 291/300 outcomes represented.

Meaning: Shows whether the active CaseSet is scorebook-like.

Limit: Completeness alone does not prove learning improves future decisions.

Experience → active CaseSet

What is still lacking?

No explicit gaps are recorded. This does not establish that every subclaim has been tested.

Next decisive test

Inspect incomplete cases and grade reliability by decision class, then test whether using these grades improves held-out decisions.

Decision value: Inspect incomplete cases and grade reliability by decision class, then test whether using these grades improves held-out decisions.

Evidence detailsInspect measurements, provenance, and original sources.

What the current evidence says

Experience capture evidence dashboard

DATA

Shows whether the active CaseSet is complete enough to function like a scorebook.

Scorebook completeness

Cases

300

Outcome completion

97%

291/300

Grade coverage

97%

291/300

Median feedback latency

23 days

n=291

Action coverage

100%

300/300

Unresolved share

3%

Human override capture

17%

51/300

Provenance quality

DERIVED — SYNTHETIC SIMULATION

Selected dataset

External / market evidence

Separate from CaseSet
Casap homepage / product materialsSUPPORTS

Purpose-built AI agents and dispute-management software for banks, credit unions, and fintechs; public materials describe dispute model/API, win-score prediction, fraud functionality, and built-in Reg E, Reg Z, Nacha, and card-network expertise.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Raises $25M Series ASUPPORTS

$25M Series A announced August 7, 2025; $33.5M total funding; AI agents analyze evidence, predict outcomes, and automate lifecycle from intake to chargeback filing; proprietary first-party fraud score; company-reported customer results.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Chartway customer storySUPPORTS

Company-published customer story reports estimated $875K annual savings, resolution time improved from 90 days to 12 days, 95% chargeback win rate, writeoffs down 72%, costs down 85%, and monthly capacity from 1,200 to 4,000 transactions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap MidSouth customer storySUPPORTS

Company-published customer story reports fraud loss reduced 51%, cost per dispute reduced 90%+ from $37, chargeback floor reduced 85%, real-time merchant-response/chargeback tracking, and Visa transaction metadata used in fraud investigation.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Filene Research Institute, 2025 FiLab Results: CasapSUPPORTS

Filene Research Institute public report summary describes testing Casap's fraud dispute platform in sandbox and live environments, with participating credit unions reporting usability, reduced manual effort, and confidence supporting compliance/dispute outcomes.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Filene blog: Chartway Credit Union saved an estimated $875K in one year testing CasapSUPPORTS

Filene reports testing Casap with five credit unions; one institution had been live on the platform for more than a year; staff reported a 122% improvement in ease of use versus prior systems; reported dispute-processing error frequency declined 63%.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap October 2024 funding announcementSUPPORTS

Company funding announcement claims Casap's platform becomes more effective at fraud detection with customer interaction.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Privacy PolicySUPPORTS

Privacy policy says Casap may use collected information to develop/improve products and conduct analytics, including aggregated non-specific information. It also says third-party personal data uploaded through the service is processed on behalf of customers and according to their instructions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Primary Venture Partners insightsSUPPORTS

Primary Venture Partners public materials identify Casap in its portfolio/insights context and state Primary led the seed round and invested in every round.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
View all sources and evidence

Evidence that supports

SUPPORTS · INDIRECT

Experience → active CaseSet
DERIVED — SYNTHETIC SIMULATION

Value / observation: 291/300 cases graded; 291/300 outcomes represented; median feedback 23 days.

Interpretation: Supports graded experience capture within the selected dataset.

Limitation: Still does not prove that captured grades improve future behavior.

Inspect source

Evidence that contradicts / alternatives

No evidence in this category yet.

Context, not proof

CONTEXT · CONTEXT

Experience → active CaseSet
DERIVED — SYNTHETIC SIMULATION

Value / observation: 291/300 graded cases; 291/300 outcomes represented.

Interpretation: Shows whether the active CaseSet is scorebook-like.

Limitation: Completeness alone does not prove learning improves future decisions.

Inspect source

Missing evidence

No evidence in this category yet.

Argument & implicationsThe case for and against, its consequences, and what would resolve it.

Case for

If Casap systematically grades cases with accurate outcomes and integrates these grades into model updates, it could improve decision quality over time, creating a compounding expertise effect that competitors without similar data may struggle to replicate.

Case against

Public evidence shows no graded or CaseSet data, and unknowns around whether grading is captured or learning rights exist suggest Casap may not effectively accumulate or leverage experience systematically, limiting durability of any advantage.

If true

A complete capture loop makes later learning and diligence tests feasible.

If false

The company may have data, but not the graded operating experience required for Compounding Expertise.

What would resolve this?

What would increase belief?

Verified existence of systematic, high-coverage grading and feedback loops integrated in model updates; evidence of positive performance trends attributable to accumulated expertise.

What would decrease belief?

Lack of grading data, partial or missing outcome observation, unclear or restricted learning rights, or absence of demonstrable improvement linked to experience accumulation.

Next decisive test: Inspect incomplete cases and grade reliability by decision class, then test whether using these grades improves held-out decisions.

Your assessmentYour recorded belief and working notes, separate from evidence.
35%AI-suggested; review before adopting

Your belief is more skeptical than the current evidence model. Note the concern or alternative explanation.

Use Edit debate to change and save your belief.

Debate 5 · Learning rights / privileged access

Can the company legally and operationally retain and exploit the experience?

UNKNOWNLOW confidenceHIGH impact
Company evidenceUNKNOWNLOW confidence
Modeled mechanismNOT TESTEDConditional, not company evidence

Finding · integrated conclusion

The full company-level thesis is not yet resolved. Case rows cannot establish contractual retention, training, or cross-customer reuse rights.

Assessment scope: The company has usable rights and access to retain and learn from experience.

Why it matters: Rights and access determine whether captured experience can become a reusable learning asset.

Thesis being tested

The company has usable rights and access to retain and learn from experience.

Subclaims: what must be true

Each subclaim is resolved separately against the selected cases, conditional experiments, or required company evidence.

  1. Contracts permit retaining the relevant records or derived features.REQUIRES EXTERNAL / COMPANY EVIDENCE

    Review retention and derived-feature clauses.

  2. The intended evaluation, training, and cross-customer reuse are permitted.REQUIRES EXTERNAL / COMPANY EVIDENCE

    Review evaluation, training, and cross-customer reuse permissions.

  3. Operational access and governance make those permissions usable in practice.REQUIRES EXTERNAL / COMPANY EVIDENCE

    Verify operational access, deletion, isolation, and governance controls.

What evidence exists?

Supports the thesis 0

No supporting evidence is attached to this proposition.

Challenges the thesis 0

No contradicting evidence is attached.

What the cases show 1

Learning rights: UNKNOWN

Meaning: Rights determine whether experience can be retained and reused.

Limit: Review scope of allowed uses.

Company Model → Learning rights

What is still lacking?

  1. Contractual language covering retention, derived features, evaluation, and cross-customer training.

    Need retention, derived-feature, evaluation, and cross-customer training rights.

Next decisive test

Review contractual and data-governance rights for retention, derived features, evaluation, and cross-customer training.

Decision value: Need retention, derived-feature, evaluation, and cross-customer training rights.

Evidence detailsInspect measurements, provenance, and original sources.

What the current evidence says

Learning rights / privileged access evidence dashboard

DATA

Shows currently attached data and explicit evidence gaps for this debate.

Current evidence state

Active CaseSet cases

300

External evidence records

3

External / market evidence

Separate from CaseSet
Casap Chartway customer storyCONTEXT

Company-published customer story reports estimated $875K annual savings, resolution time improved from 90 days to 12 days, 95% chargeback win rate, writeoffs down 72%, costs down 85%, and monthly capacity from 1,200 to 4,000 transactions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Filene blog: Chartway Credit Union saved an estimated $875K in one year testing CasapCONTEXT

Filene reports testing Casap with five credit unions; one institution had been live on the platform for more than a year; staff reported a 122% improvement in ease of use versus prior systems; reported dispute-processing error frequency declined 63%.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap Privacy PolicyCONTEXT

Privacy policy says Casap may use collected information to develop/improve products and conduct analytics, including aggregated non-specific information. It also says third-party personal data uploaded through the service is processed on behalf of customers and according to their instructions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
View all sources and evidence

Evidence that supports

No evidence in this category yet.

Evidence that contradicts / alternatives

No evidence in this category yet.

Context, not proof

CONTEXT · DIRECT

Company Model → Learning rights
SOURCED — EXTERNAL EVIDENCE

Value / observation: Learning rights: UNKNOWN

Interpretation: Rights determine whether experience can be retained and reused.

Limitation: Review scope of allowed uses.

Inspect source

Missing evidence

MISSING · MISSING

Contracts / data governance
UNKNOWN / DILIGENCE REQUIRED

Value / observation: No sourced contractual rights evidence.

Interpretation: Need retention, derived-feature, evaluation, and cross-customer training rights.

Limitation: Missing evidence cannot support the proposition.

Obtain via: Legal/data-room review

Expected evidence: Contractual language covering retention, derived features, evaluation, and cross-customer training.

Argument & implicationsThe case for and against, its consequences, and what would resolve it.

Case for

No bull case supplied yet.

Case against

No bear case supplied yet.

If true

The company may be able to pool and reuse experience in ways challengers or customers cannot easily replicate.

If false

The company may observe cases but lack the rights to retain, pool, train on, or evaluate with them.

What would resolve this?

What would increase belief?

Contracts explicitly allow retention, derived features, evaluation, and cross-customer learning.

What would decrease belief?

Customer contracts restrict retention, pooling, model training, or evaluation reuse.

Next decisive test: Review contractual and data-governance rights for retention, derived features, evaluation, and cross-customer training.

Your assessmentYour recorded belief and working notes, separate from evidence.
Not setNo belief recorded

Belief remains separate from the CE evidence assessment.

Use Edit debate to change and save your belief.

Debate 6 · Marginal information value

Do additional graded cases continue to add useful decision-relevant information?

UNPROVENLOW confidenceHIGH impact
Company evidenceUNPROVENLOW confidence
Modeled mechanismSUPPORTED CONDITIONALLYConditional, not company evidence

Finding · integrated conclusion

The full company-level thesis is not yet resolved. The selected cases challenge a measurable component: Expanding from the earliest cohort to full history changed Brier by -0.000; the curve reverses direction. The modeled mechanism is supported conditionally: Full history remained competitive throughout the tested drift range.

Assessment scope: Additional graded experience continues to improve decision quality.

Why it matters: High case volume does not matter if new cases are redundant or quickly exhausted.

Thesis being tested

Additional graded experience continues to improve decision quality.

Subclaims: what must be true

Each subclaim is resolved separately against the selected cases, conditional experiments, or required company evidence.

  1. New cohorts add useful lessons or improve estimates of known patterns.CHALLENGED BY SELECTED CASES

    Expanding from the earliest cohort to full history changed Brier by -0.000; the curve reverses direction.

  2. Adding those cohorts produces incremental performance gains against a fixed baseline.CHALLENGED BY SELECTED CASES

    The positive marginal-value sub-thesis fails: the gain is below 0.005 Brier or reverses as history grows.

  3. Those gains persist after accounting for redundancy and changing conditions.SUPPORTED CONDITIONALLY

    Full history remained competitive throughout the tested drift range.

What evidence exists?

Supports the thesis 0

No supporting evidence is attached to this proposition.

Challenges the thesis 0

No contradicting evidence is attached.

What the cases show 3

With a fixed pooled estimator and evaluation set, Brier 0.224 → 0.217 → 0.215 → 0.224 as training grows through 58 / 116 / 174 / 232 cases. The learning curve reverses direction: more history is not consistently better.

Meaning: A fixed-test learning curve measures whether additional historical cases help this estimator; results may be non-monotonic.

Limit: Repeat on a fresh time window and examine changing case mix.

Experience → active CaseSet

2 more in the full evidence ledger below.

What is still lacking?

  1. Decision-policy value by successive cohort on a fixed evaluation set.

    Measure action-policy performance across successive training cohorts.

Next decisive test

Measure incremental performance gain from successive case cohorts after policy and model baselines are included.

Decision value: Measure action-policy performance across successive training cohorts.

EXECUTABLE NEXT TEST

Test whether a changed action policy improves outcomes

Measure incremental performance gain from successive case cohorts after policy and model baselines are included.

Configure experiment
Evidence detailsInspect measurements, provenance, and original sources.

What the current evidence says

Marginal information value evidence dashboard

DATA

Information Structure diagnostics are descriptive evidence about the active CaseSet, not proof that additional cases improve future performance.

Information Structure context

NEW INFORMATION ENTERS THE SCOREBOOK is not the same as THE SYSTEM LEARNS FROM THAT INFORMATION.

Total cases

300

Decision class entropy

1.56 bits

HIGH DIVERSITY; n=300/300

Pattern repetition

68%

204/300 repeated-pattern cases

Structural novelty

NOVELTY DECLINING

latest cohort 40%

Case type → grade

23% uncertainty reduction

0.35 bits; SPARSE CATEGORIES — INTERPRET CAUTIOUSLY

Successive-cohort test

CHALLENGES

Expanding from the earliest cohort to full history changed Brier by -0.000; the curve reverses direction.

Cohort 144 new patterns · novelty 73%
Cohort 245 new patterns · novelty 75%
Cohort 327 new patterns · novelty 45%
Cohort 425 new patterns · novelty 42%
Cohort 524 new patterns · novelty 40%

Interpretation limits

Entropy, repetition, novelty, and mutual information do not establish learning causality, cross-customer transfer, compressibility, or durable Power.

Learning-causality separation

Context only

Information Structure evaluates scorebook contents, not whether the system learned from them.

Power separation

No direct Power inference

Information exists is not the same as proprietary, performance-improving, irreproducible information.

External / market evidence

Separate from CaseSet
Casap Chartway customer storyCONTEXT

Company-published customer story reports estimated $875K annual savings, resolution time improved from 90 days to 12 days, 95% chargeback win rate, writeoffs down 72%, costs down 85%, and monthly capacity from 1,200 to 4,000 transactions.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Filene blog: Chartway Credit Union saved an estimated $875K in one year testing CasapCONTEXT

Filene reports testing Casap with five credit unions; one institution had been live on the platform for more than a year; staff reported a 122% improvement in ease of use versus prior systems; reported dispute-processing error frequency declined 63%.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
Casap October 2024 funding announcementCONTEXT

Company funding announcement claims Casap's platform becomes more effective at fraud detection with customer interaction.

SOURCED — EXTERNAL EVIDENCE · Public-source evidence record for the Casap Public Evidence Analysis. Treat as sourced context for diligence, not production case-level evidence.

Inspect source
View all sources and evidence

Evidence that supports

No evidence in this category yet.

Evidence that contradicts / alternatives

No evidence in this category yet.

Context, not proof

CONTEXT · CONTEXT

Experience → active CaseSet
DERIVED — SYNTHETIC SIMULATION

Value / observation: With a fixed pooled estimator and evaluation set, Brier 0.224 → 0.217 → 0.215 → 0.224 as training grows through 58 / 116 / 174 / 232 cases. The learning curve reverses direction: more history is not consistently better.

Interpretation: A fixed-test learning curve measures whether additional historical cases help this estimator; results may be non-monotonic.

Limitation: Repeat on a fresh time window and examine changing case mix.

Inspect source

CONTEXT · CONTEXT

Experience → Automated Experiment Program
DERIVED — SYNTHETIC EXPERIMENT

Value / observation: SUPPORTS: Full history remained competitive throughout the tested drift range. The best selected-history gain versus full history moves from -0.009 at zero drift to 0.009 at maximum tested drift. Works when: Full accumulated history remains useful across the tested range. Fails when: A staleness boundary was not reached in the tested range.

Interpretation: The learning system needs explicit recency and selection logic; accumulating every case is not itself the advantage. Next experiment: Backtest rolling time windows on the selected dataset and compare stable versus rapidly changing decision classes.

Limitation: Automated sensitivity results are conditional simulations calibrated from the selected dataset’s size, group structure, missing feedback, and descriptive time shift. They identify mechanism boundaries; they do not convert simulated outcomes into observed company evidence.

Inspect source

CONTEXT · CONTEXT

Experience → active CaseSet
DERIVED — SYNTHETIC SIMULATION

Value / observation: 300 cases. HIGH DIVERSITY; MODERATE REPETITION; NOVELTY DECLINING.

Interpretation: Pattern diversity and cohort novelty describe the supply of new information available to learn from.

Limitation: A held-out learning curve is needed to connect novelty to performance gain.

Inspect source

Missing evidence

MISSING · MISSING

Experience / future experiment
UNKNOWN / DILIGENCE REQUIRED

Value / observation: Predictive learning curve measured; policy-value curve still needed.

Interpretation: Measure action-policy performance across successive training cohorts.

Limitation: Missing evidence cannot support the proposition.

Obtain via: Experiment

Expected evidence: Decision-policy value by successive cohort on a fixed evaluation set.

Argument & implicationsThe case for and against, its consequences, and what would resolve it.

Case for

No bull case supplied yet.

Case against

No bear case supplied yet.

If true

Ongoing operation continues adding useful expertise rather than merely accumulating redundant logs.

If false

The scorebook may plateau; historical cases may be compressible into rules, policies, or small calibration sets.

What would resolve this?

What would increase belief?

Recent cohorts continue improving decision quality or edge-case handling.

What would decrease belief?

Performance plateaus quickly or new cases duplicate already-known lessons.

Next decisive test: Measure incremental performance gain from successive case cohorts after policy and model baselines are included.

Your assessmentYour recorded belief and working notes, separate from evidence.
Not setNo belief recorded

Belief remains separate from the CE evidence assessment.

Use Edit debate to change and save your belief.

+ Add debate
Back to ExperienceContinue to Power