An AI platform can show hundreds of historical deals, a polished performance curve, and an impressive accuracy rate while still failing to answer the question that matters most.
But could an investment team have achieved the same result using only the information available when each deal was originally screened?
The 2026 DIALECTIC research shows the challenges. Researchers evaluated an AI startup-screening system using 259 opportunities collected from the watchlists of five venture capital funds.
About 25% of the companies met the study’s success definition by raising a Series A or later funding round by September 2025. The system recorded an AUC-PR of 0.3681 on the validation set, but the result fell to 0.2422 on the test set.
That decline does not make the research useless. In fact, it shows why predictive accuracy in AI-assisted private-market screening cannot be judged from a single headline metric.
A reliable AI deal-discovery backtest recreates historical investment decisions using only the information available at each original screening date. It defines a measurable outcome, preserves unsuccessful and rejected opportunities, separates data chronologically, compares the model with simple baselines, evaluates calibration, and protects the final test period from model development.
To understand how these controls fit into the broader system, explore our guide to AI-driven deal-screening architecture, model validation, and known limitations.
Why Strong Historical Results Often Fail in Real-World Screening
The first problem is usually not the algorithm. In fact, it is the historical record supplied to the algorithm.
Private-market databases change over time. Funding totals are corrected, investor identities are added, employee estimates are revised, and company categories are revised or reclassified. When a platform also evaluates crypto or tokenized assets, its datasets may change as exchanges revise circulating-supply figures, update wallet labels, remove inactive assets, or modify historical coverage
Consequently, a database downloaded today may describe a 2021 company far more completely than an investor could have described it in 2021.
A funding round may have closed in January, been announced in March, and entered a commercial database in April. A model recreating a February screening decision cannot use that round simply because its transaction date says January.
This is look-ahead leakage. The backtest gives the model access to evidence that was unavailable at the original decision point.
Survivorship bias creates a separate distortion. Successful companies usually retain websites, management profiles, press coverage, and funding records. Failed startups often lose those records. Likewise, abandoned crypto projects and delisted tokens may disappear from commonly used screens.
Therefore, reconstructing an older investment universe from current databases can remove many of the hardest failures.
Selection bias also matters. Companies receiving investment gain capital, introductions, visibility, and credibility. Their later outcomes may partly reflect the investor’s involvement, not only the quality of the original screening decision.
A portfolio-only backtest cannot determine which rejected companies later succeeded or which selected companies succeeded partly because they received investment.
For this reason, the test population should include the original opportunity universe wherever records permit. That means retaining funded, rejected, failed, unresolved, acquired, and inactive companies under rules defined before model development.
The prediction target must also be explicit. “AI identifies good deals” is not testable because good may refer to revenue growth, another funding round, an exit, governance quality, liquidity, or investment return.
A valid claim looks more like this:
Estimate the probability that a seed-stage enterprise software company raises at least $20 million within 36 months, using only evidence available on its original screening date.
This statement identifies the eligible population, target event, time horizon, and information boundary.
It also prevents a common overclaim. Predicting follow-on financing does not prove that a model can predict investor returns. Similarly, identifying a liquid token does not prove that the token represents enforceable ownership, sufficient market depth, or profitable exit liquidity.
Finally, the platform must report the target’s base rate. A model claiming 90% accuracy may be ineffective if 90% of the sample belongs to one class. It could achieve that result by assigning every opportunity to the majority class.
A Rigorous Backtesting Methodology for AI Deal Discovery
A credible protocol begins before the development team compares algorithms. Once the team has seen which target, feature, or period produces the strongest result, it becomes difficult to separate a genuine research decision from retrospective optimization.
Freeze the Outcome and Observation Window
First, document what qualifies as success and failure.
The protocol should explain how it treats follow-on financing, down rounds, acquisitions, shutdowns, defaults, token delistings, inactive contracts, and unresolved outcomes. It should also specify how long each opportunity must be observed.
A company screened in 2025 cannot be evaluated fairly against a five-year exit target in 2026. It has not yet completed the outcome window.
Such an observation is censored, not necessarily unsuccessful. Censoring means the final event has not been observed within the available period. Survival analysis can handle this problem when the objective is to estimate time until funding, failure, default, or exit.
Alternatively, researchers can limit the final evaluation to vintages with mature outcomes. Either method is preferable to labeling immature deals as failures.
Reconstruct Point-in-Time Evidence
Next, rebuild the information set that existed on every screening date.
Each field should retain:
- Its original source
- The value available at the time
- The event date
- The publication or availability date
- The database capture date
- Any later revisions
- The reason for a missing value
- A confidence or verification status
This distinction is important because event time, publication time, and database-ingestion time may be different.
Point-in-time records may include archived company websites, dated investment memos, regulatory filings, historical vendor snapshots, original cap tables, timestamped smart-contract data, and time-filtered news results.
Missing values should not automatically become zero. A blank revenue field may mean no revenue, non-disclosure, incomplete vendor coverage, or an inapplicable metric. Combining those states can make the model predict reporting behavior instead of investment quality.
Move Through Deal Vintages Chronologically
Random train-test splits are convenient, but they often fail to reproduce real deployment.
A model trained on randomly selected opportunities from 2018 through 2025 may learn from a 2024 financing environment before being evaluated on a 2019 company. Even without direct leakage, the test no longer represents a forward investment decision.
A more credible out-of-sample test for private markets moves through time.
An illustrative protocol could:
- Train on deals screened from 2016 through 2019.
- Select features and tune parameters within that historical development period.
- Validate model choices on the 2020 and 2021 vintages.
- Lock the feature set, algorithm, threshold, and missing-data rules.
- Test once on the 2022 vintage.
- Observe outcomes through 2025 for a 36-month prediction target.
Point-in-Time Backtesting Framework for AI Deal Discovery
A defensible AI deal-discovery backtest preserves the historical information boundary, separates deal vintages chronologically, isolates the final holdout, and continues into prospective monitoring.
These dates are illustrative, not reported investment results.
The core principle is that training information must predate validation information, while the final test vintage must remain unavailable until development is complete.
4. Keep Every Development Choice Away From the Final Test
Feature selection, class weighting, imputation methods, model families, hyperparameters, decision thresholds, and label definitions must be chosen without repeatedly checking the final holdout.
Otherwise, the holdout gradually becomes another validation set.
The research record should preserve every material experiment, including configurations that performed poorly. This matters because testing enough combinations will eventually produce an apparently strong result by chance.
Once the model is locked, the team should run it once on the final test vintage and report the result, whether it is positive, negative, or inconclusive.
5. Compare the Model With Simple Alternatives
A complex AI system has not demonstrated economic usefulness merely because it performs better than random chance.
It should also be compared with:
- Sector and stage success rates
- A written analyst scorecard
- The existing investment review process
- A simple logistic regression
- A basic ranking model
- The previous production system
If a simple scorecard produces the same shortlist, the complex model may add cost and governance risk without improving the decision.
6. Begin Prospective Shadow Testing
Historical validation should be followed by prospective testing.
During shadow testing, the system records predictions on new opportunities without determining the investment decision. The team then compares those predictions with later outcomes.
The required period depends on the target. Operational measures, such as research time saved or document-extraction accuracy, may be evaluated within months. Funding, default, liquidity, and exit outcomes may require several years.
Historical and prospective results should remain separate. Combining reconstructed scores with live predictions can make a model appear more established than it is.
Material changes should also create a new model version. These include changing the data vendor, adding features, revising the success label, replacing the underlying language model, or moving the decision threshold.
Metrics That Reveal Predictive Accuracy and Overfitting
Accuracy can be misleading when positive outcomes are rare. If only 10% of companies succeed, a model can report 90% accuracy by predicting failure for every opportunity.
For funds with limited review capacity, precision at K and recall at K are more useful. Precision at K measures how many top-ranked recommendations succeed, while recall at K shows how many total successful opportunities appear in that shortlist.
For example, if eight of the top 20 recommendations succeed, precision at 20 is 40%. If 32 companies across the full opportunity universe succeed, the recall at 20 is 25%.
Ranking and calibration measure different capabilities. Ranking determines whether stronger opportunities appear above weaker ones. Calibration tests whether predicted probabilities match actual outcomes. If only 35% of companies assigned a 70% success probability reach the target, the model is overconfident.
PR-AUC evaluates precision and recall across thresholds and should be interpreted alongside the positive-class prevalence. If 25% of observations are positive, 0.25 is an approximate no-skill reference for average precision, although the exact baseline depends on the calculation method.
The Brier score measures the average squared difference between predicted probabilities and actual outcomes. Lower scores generally indicate better probability forecasts.
Institutional reports should include precision and recall at the operating threshold, precision and recall at K, PR-AUC beside the base rate, Brier scores, calibration plots, confidence intervals, subgroup results, sample sizes, and performance by deal vintage.
Backtest overfitting requires separate analysis. Bailey, Borwein, López de Prado, and Zhu define the probability of backtest overfitting as the likelihood that the best in-sample configuration performs below the median of competing models out-of-sample.
Their framework also highlights the file-drawer problem. Hiding unsuccessful experiments makes reported results appear more reliable than they are. Vendors should therefore disclose tested features, model families, hyperparameters, thresholds, alternative targets, rejected configurations, and negative findings.
Because the framework was developed for investment-strategy backtests, applying it to private-market classifiers requires an adapted method that preserves chronology and accounts for sparse or censored outcomes. However, no overfitting test can repair data already affected by look-ahead leakage, survivorship bias, or future information.
What the DIALECTIC Results Show Institutional Buyers
DIALECTIC is useful because the authors disclose both performance and limitations.
The system gathers startup facts, organizes them into a question tree, develops arguments for and against investment, and produces scores. Researchers evaluated 259 companies added to five VC watchlists during 2021. Of these, 65 companies, or approximately 25%, later raised a Series A or higher round by September 1, 2025.
The dataset was divided using random stratified sampling into 129 validation companies and 130 test companies. The participating funds had invested in only six companies, and two met the study’s success definition.
The preferred configuration produced an AUC-PR of 0.3681 during validation and 0.2422 on the test set, showing why model selection and unseen evaluation must remain separate.
Because approximately 25% of the sample met the success definition, the test AUC-PR was close to the positive-class prevalence. However, the precise no-skill comparison depends on how the precision–recall area was calculated.
The final sample declined from 3,441 CRM entries to 259 companies after cleaning, entity matching, temporal cutoffs, and historical-data availability requirement
Even then, the authors acknowledged residual look-ahead risk because a January 24, 2022, Crunchbase snapshot supplied some historical information.
DIALECTIC presents an inspectable screening process. However, a random split within one cohort is not a later-vintage test, andsix observed investment decisions are insufficient to establish broad superiority over human investors. Its transparency lets buyers see what remains unproven.
The Evidence Institutional Buyers Should Demand
An investment committee should classify a platform’s claim according to the evidence supplied.
Claim status | Minimum evidence | Appropriate use |
Unverified claim | No fixed target, dated evidence, or isolated test | Research demonstration |
Historical research result | Defined target, point-in-time reconstruction, baselines, and out-of-sample test | Limited screening experiment |
Provisionally validated model | Later-vintage test, calibration, independent review, and documented limits | Controlled use with human oversight |
Production-supported track record | Prospective predictions, mature outcomes, version history, and drift monitoring | Decision support within approved limits |
Before procurement, buyers should request the model purpose, prohibited uses, prediction target, point-in-time data dictionary, vintage definitions, opportunity universe, training and test periods, model-search record, baseline results, calibration analysis, override history, and abstention rules.
An abstention rule is essential. When evidence is missing, sources conflict, or an opportunity lies outside the training distribution, “insufficient evidence” is more credible than a precise score.
On April 17, 2026, the Federal Reserve, FDIC, and OCC jointly issued revised model-risk management guidance, which the Federal Reserve transmitted through SR 26-2. The guidance supersedes SR 11-7 and SR 21-8 and is expected to be most relevant to banking organizations with more than $30 billion in total assets. It outlines supervisory principles rather than enforceable or prescriptive requirements.
The guidance directly covers traditional quantitative models and non-generative, non-agentic AI. Generative and agentic systems fall outside its direct scope, although organizations should still determine suitable governance and controls.
For third-party models, it calls for understanding conceptual soundness, design, development data, performance, monitoring, and outcome analysis.
NIST AI RMF 1.0 adds a broader voluntary framework through
- Govern,
- Map,
- Measure,
- And manage.
It treats governance, monitoring, documentation, and third-party data risk as lifecycle responsibilities.
For investment teams, this means documenting the model’s intended use, mapping the potential harm caused by false positives and false negatives, measuring performance and calibration, and managing limitations through human review, abstention rules, access controls, and ongoing monitoring.
Can an independent reviewer recreate the historical information boundary, understand major model choices, reproduce the metrics, and identify where the system should not be trusted?
If not, the score remains an unverified research claim.
How to Judge an AI Deal-Discovery Backtest
A credible AI deal-discovery backtest should define a measurable prediction target, preserve the point-in-time information available at each screening date, and include the complete historical opportunity universe rather than only successful or funded companies.
A backtest without these controls may measure database hindsight rather than genuine predictive ability. Institutional buyers should therefore treat historical accuracy as preliminary evidence until a locked model performs consistently on later vintages and prospectively recorded opportunities.
FAQs
What is point-in-time backtesting?
It recreates each historical decision using only information available on that date, including publication delays, database revisions, and vendor-ingestion timing.
Why are random splits risky in private markets?
They can mix market conditions and fail to simulate deployment. Chronological testing trains on older vintages and evaluates a locked model on a later period.
Which metric best measures AI deal-sourcing performance?
No single metric is enough. Use precision at K, recall at K, PR-AUC, calibration, Brier score, lift over the base rate, confidence intervals, and vintage stability.
Can backtesting prove future returns?
No. It evaluates a defined historical prediction under stated assumptions. It cannot guarantee returns or prove that the same relationships will survive future markets.
Investment Risk Notice
AI deal-screening scores are research tools. They do not guarantee performance or replace legal, financial, tax, regulatory, technical, or investment due diligence.
