A 2026 EACL study tested the DIALECTIC multi-agent system on startup-screening decisions. Its AUC-PR declined from 0.3681 during validation to 0.2422 on the held-out test set. The researchers also found that predictive performance improved through two reasoning rounds but weakened when further rounds added longer arguments and more referenced facts. Greater analytical complexity did not produce a stronger investment signal.
That result captures the central model risk in algorithmic investment platforms. An impressive historical result may reflect overfitting rather than repeatable skill. Missing failures can make the training universe look cleaner than the market investors actually faced. Limited disclosure can then prevent buyers from identifying either defect.
Before approving a platform, the investment committee must determine whether the signal is reproducible, the historical sample is complete and the model can withstand independent challenge.
Figure 1. Model risk audit workflow for evaluating an algorithmic investment platform before capital allocation.
Model Risk Starts With the Decision Behind the Score
Model risk is the possibility of adverse financial consequences caused by decisions based on model output. It depends not only on technical accuracy, but also on the model’s assumptions, data quality, complexity, purpose, use and financial exposure. Even a sound model can create material risk when it is applied outside its intended purpose.
On April 17, 2026, the Federal Reserve, OCC and FDIC issued revised model-risk-management guidance. The Federal Reserve communicated it through SR 26-2, replacing SR 11-7 and SR 21-8. The guidance adopts a risk-based approach and is expected to be most relevant to banking organizations with more than $30 billion in total assets. It does not create universal requirements for nonbank investment platforms, but its development, validation and governance principles provide a useful due-diligence benchmark.
The first question is therefore not whether the platform reports high accuracy. It is whether the model predicts an outcome that matters to the proposed investment decision.
A platform should define:
- The event being predicted
- The eligible investment population
- The forecast period
- The information cutoff date
- The permitted use of the score
- The decisions the score must not control
A model trained to predict whether a startup will raise another financing round has not demonstrated that investors will earn a positive return. Similarly, a model forecasting token activity has not proved that a position can be exited at the quoted price.
The broader AI-driven deal screening model architecture, validation and known limitations explains how raw evidence becomes a score. Model-risk review determines whether that score is suitable for the amount of capital and type of decision placed behind it.
Overfitting Makes Historical Fit Look Like Durable Skill
Overfitting occurs when a model learns relationships that work in its development sample but fail on new opportunities.
Algorithmic deal platforms are especially exposed because their feature sets can be large while reliable outcomes remain limited. A system may evaluate founder histories, funding activity, employee growth, revenue estimates, product signals, wallet movements and market conditions. Yet the same dataset may contain only a small number of mature exits, shutdowns, defaults or realized returns.
Every additional feature combination, algorithm, hyperparameter, outcome definition and decision threshold creates another opportunity to find an attractive result by chance.
Bailey, Borwein, López de Prado and Zhu developed the probability of backtest overfitting to measure this problem in investment simulations. Their research shows that conventional holdout methods can become unreliable when many alternatives are tested and only the strongest in-sample result is presented. They proposed combinatorially symmetric cross-validation to estimate how often the apparent winner in development becomes a weak out-of-sample performer.
A credible platform should therefore provide its complete experiment register, including:
- The original feature universe
- Every material model family tested
- Hyperparameter ranges
- Alternative outcome definitions
- Rejected specifications
- The model-selection rule
- The period isolated for final evaluation
Chronology also matters. SR 26-2 identifies out-of-sample and out-of-time testing, comparisons with alternative assumptions and critical data-quality review as relevant testing activities.
The DIALECTIC research used a random stratified split containing 129 validation companies and 130 test companies. Both sets preserved the same proportion of successful startups, and the final configuration was chosen using validation performance. This design provides held-out evidence within the constructed dataset, but it does not test whether the model would survive a later funding cycle or a materially different market regime.
Outcome quality is another constraint. A 2025 NBER working paper examined more than 8,000 opportunities sourced by one early-stage venture fund. The fund intensively scored 366 companies and invested in 114. Although selected firms performed better than rejected opportunities, only 32% of the investments later raised more than $10 million, while 13% raised more than $25 million. The findings show that meaningful selection ability can coexist with noisy outcomes. Later financing also remains different from realized investor return.
The practical conclusion is simple. A high development score is not enough. The platform must show that its result survives new time periods, simple baselines and reasonable changes to the outcome definition.
Survivorship Bias Changes the Market the Model Remembers
Survivorship bias arises when failed, closed, rejected or delisted investments disappear while visible survivors remain.
The risk is acute in private-market and crypto data. Failed startups may remove their websites. Rejected opportunities may receive little follow-up. Delisted tokens can vanish from active screens. Successful projects, by contrast, continue accumulating funding records, employee data, press coverage and transaction histories.
A database reconstructed years later can therefore describe a cleaner and more successful market than the one investors originally faced.
The DIALECTIC dataset illustrates how much an evaluation population can change during preparation. Researchers began with 3,441 companies exported from venture-fund watchlists. After cleaning, matching, temporal controls and alignment with a historical Crunchbase snapshot, 259 companies remained. The observed success rate was 21.6% after matching current funding information, fell to 18.7% after a timing-related exclusion and reached 25.1% in the final sample.
These exclusions do not by themselves prove that the final dataset was biased. However, they show why sample attrition must be disclosed. Filtering can change the size of the dataset, the positive-class rate and the baseline against which AUC-PR, precision and lift should be interpreted.
Figure 2. Dataset attrition and changing success rates during startup-screening sample construction.
A platform should preserve the original opportunity universe, including failed businesses, rejected deals, inactive funds, delisted assets, contract migrations, renamed entities and unresolved outcomes. It should also show how many observations were removed at every stage and why.
One audit test is particularly effective. Ask the vendor to reproduce a historical score using only evidence archived on the original screening date. If the calculation requires current funding, employee, website or wallet information, the historical test contains look-ahead leakage.
Disclosure Gaps Prevent Effective Challenge
A vendor may protect its source code and proprietary feature construction. It should not withhold the evidence needed to evaluate reliability, limitations and capital exposure.
The BIS Financial Stability Institute reported in March 2026 that data privacy, quality and security remain major barriers to advanced AI use in financial services. It also warned that third-party dependencies and concentration among major service providers can intensify those risks.
In June 2026, the Financial Stability Board published a consultation proposing 12 sound practices covering organization-wide governance and different stages of AI development and deployment. The practices are intended to support boards, senior management and risk functions as they assess AI adoption. As of July 2026, the report remains a consultation rather than final guidance.
The following evidence should be requested before platform approval.
Model-risk area | Evidence required | Warning sign |
Prediction design | Defined target, population, horizon and permitted use | A vague quality or success score |
Overfitting | Experiment register and isolated final evaluation | Only the best-performing configuration is shown |
Data history | Capture dates, snapshots and revision records | Current data are used to recreate past decisions |
Sample completeness | Original opportunity universe and exclusion log | Failed or removed assets cannot be restored |
Missing information | Missingness categories and imputation rules | Blank fields are silently converted to zero |
Stability | Performance by vintage, sector and deal stage | No drift or feature-decay testing |
Human intervention | Original score, override reason and outcome | Overrides are omitted from performance reporting |
Change control | Version, deployment date and validation status | Production changes bypass validation |
Third-party reliance | Data, model and infrastructure dependencies | Upstream changes cannot be inspected |
Monitoring | Performance limits and suspension rules | No documented stop-use condition |
Independent validation should be carried out by reviewers with the technical expertise to challenge the method, sufficient separation from the development team and enough organizational authority to force corrective action. SR 26-2 describes this combination as an effective challenge.
The evidence package should include the model-development report, point-in-time data inventory, experiment register, attrition analysis, independent validation findings, override history, change log and suspension policy.
Generative and agentic systems need an additional layer of scrutiny. SR 26-2 expressly excludes generative and agentic AI from its scope, although it covers traditional quantitative models and non-generative, non-agentic AI. A platform using LLM agents should therefore combine traditional model validation with broader AI governance, data-provenance and human-oversight controls.
Where the model evaluates tokenized opportunities, its score must also be separated from the technical and compliance layers of tokenized fund deployments. Prediction accuracy does not establish legal ownership, custody protection, transfer rights, redemption priority or executable secondary liquidity.
Similarly, claimed blockchain performance requires independent forensic verification of timestamped on-chain track records. Wallet counts and transaction volumes can be calculated precisely while still misrepresenting beneficial ownership, related-wallet activity or the liquidity available for a realistic trade size.
The Decision Must Change Capital Exposure
Model-risk findings should produce a capital decision, not merely another report.
Approve the platform when its prediction target is testable, its historical sample can be reconstructed, the final evaluation was isolated from model selection and independent review has resolved every material finding.
Use restricted approval when the platform improves research triage but has a short history or limited evidence. Restrictions can include smaller allocation limits, defined sectors, mandatory analyst confirmation, tighter evidence requirements and a fixed revalidation date.
Reject or suspend the platform when its backtest cannot be reproduced, future information contaminates historical inputs, failed assets cannot be restored, material production changes remain unvalidated or overrides are missing from the audit record.
The sophistication of the interface should not influence that decision. Capital approval should depend on whether the platform can demonstrate a repeatable signal, a representative dataset and enough disclosure for objective review.
FAQs
What is the strongest evidence that a model is not overfit?
No single metric is sufficient. Strong evidence combines chronological validation, simple benchmarks, complete experiment disclosure and an untouched later-vintage evaluation.
How can survivorship bias be detected?
Reconstruct the original opportunity universe and reconcile it with the final dataset. Every removed, failed and unresolved observation should have a documented status and exclusion reason.
Which disclosure gaps should stop approval?
Approval should stop when the platform cannot define its target, reproduce its historical test, explain data exclusions, disclose material dependencies or identify who has authority to restrict and suspend the model.
Can an algorithmic score replace investment due diligence?
No. It may improve screening consistency and research prioritization, but legal rights, financial claims, management evidence, custody, valuation, liquidity and portfolio suitability still require specialist review.
Investment risk notice
Algorithmic screening outputs are research tools. They do not guarantee investment performance and are not substitutes for legal, financial, regulatory, tax or technical advice.
