Model validation
We raised performance
and mapped the limits
RenaAi builds alternative data with its own methodology and connects it to global financial markets. This page covers the “validation” step in between: generation-by-generation results for SBCR, our small-business credit scoring model built on market-area alternative data.
We compared three designs on the same sample and the same split, and wrote more about what did not work, and why, than about what did. The full validation report is provided individually to rating agencies, lenders and investors.
Validation summary
- Subject
- SBCR small-business credit scoring model
- Comparison
- Same sample · same split · out-of-sample
- Out-of-sample AUC
- 0.7470 → 0.7847 → 0.8325
- Methodology source
- Forward intensity from the NUS Risk Management Institute lineage
- Full report
- Provided individually to rating agencies, lenders and investors
Performance by generation
Three generations, measured with the same ruler
0.7470
First-generation design combining market-area concentration, sales volatility and co-movement in a logistic model.
0.7847
Transplants the forward intensity structure from the NUS Risk Management Institute lineage to produce a term structure of default probabilities at 1, 2 and 4 quarters. Adding the v1 variables back actually lowered it by 0.0058.
0.8325
Replaced the layer that collapsed the quarterly time series into fixed summaries with a flattened MLP. Up 0.0242 versus 0.8083 for the fixed summaries.
Pick at random one business that actually ran into trouble and one that did not. AUC is the share of such pairs in which the model gives the troubled one a higher risk score. 0.5 is a coin toss; the closer to 1, the better it separates them. Card issuers' retail scoring, which uses decades of payment history, falls in the 0.70–0.80 range. We confirmed 0.83 without using a single line of credit history.
Methodology source
Where the methodology comes from
The NUS Risk Management Institute lineage
- 2006RMI founded. It is a National University of Singapore (NUS) institute, established with support from the Monetary Authority of Singapore (MAS).
- 2009The Credit Research Initiative (CRI), a credit rating research unit, was launched at RMI.
- 2021NUS AIDF launched, a university-level institute jointly founded by MAS, NRF and NUS.
- TodayCRI now operates as AIDF's Deep Credit Analytics Lab.
“The Asian Institute of Digital Finance (AIDF) is a university-level institute in the National University of Singapore (NUS), jointly founded by The Monetary Authority of Singapore (MAS), the National Research Foundation (NRF) and NUS.”
RMI and CRI are university research institutes. They are not part of a supervisory authority; MAS took part as a founding and co-founding party.
Who uses what this lineage built
NUS-CRI's credit risk data and toolkits are used by academia, corporates and regulators worldwide. Among them, BuDA (Bottom-up Default Analysis) is a credit stress-testing toolkit co-developed by CRI and the IMF, a framework for assessing individual firms' probability of default under macro scenarios.
CEO Byung-Geun Choi served as Product Manager for this BuDA toolkit. The forward intensity structure transplanted into SBCR v2 belongs to the same methodological lineage.
The institutions above are users of NUS-CRI's data and toolkits, not a client list of any individual.
Researchers who share this lineage
Seyoung Park
Co-researcher · University of Nottingham
- Associate Professor, University of Nottingham
- Asset pricing and household finance theory
- Research background linked to NUS AIDF
Hyuntak Lee
Co-researcher · Gachon University
- Assistant Professor, Gachon University
- Credit risk and econometric models
- Research background linked to NUS AIDF
Byung-Geun Choi
Model design and implementation · RenaAi
- Researcher, NUS Risk Management Institute (2017–2019)
- Product Manager, BuDA toolkit (co-developed with the IMF)
- Samsung Asset Management · NH-Amundi · Korea Investment Management
- Operations Research Letters (2016), among others
The three are co-authors of “Credit Expansion Opportunities and Inclusive Household Finance,” a paper submitted to Finance Research Letters (SSCI). Hyuntak Lee and Byung-Geun Choi co-wrote the feature article “Small-business credit scoring and the use of market-area information” in Credit Finance No. 86 (Summer 2026), published by the Credit Finance Association of Korea (CREFIA).
Rejected hypotheses
We report what we rejected, too
- 1
Using spatial networks for prediction
The sales co-movement network is real; its out-of-sample persistence was confirmed at z≈116. But even with 40 neighbors as free variables, the incremental discriminatory power was a 0.0051 decline.
- 2
Attention architecture
Attention (0.8373) beat the flattened MLP (0.8325) by 0.0048, but the MLP had the higher PR-AUC (0.4376 vs. 0.4263). We judged the gain not worth the added complexity and did not adopt it.
- 3
Using a sales-decline label as a proxy for closure
Actual closure rates in the "quarterly sales down 30%" cells were no different from other cells (restaurants 0.99x, medical 1.05x). This is the nature of the indicator, not a defect, so we redefined PD as Probability of Distress.
And one was a limit of the data
Closure contagion between market areas barely showed up in quarterly data (hazard ratio 1.036). To tell whether the effect was absent or the data was too coarse, we re-measured the same phenomenon using daily closure dates, based on 135,639 closures of general restaurants in Seoul.
The key is that the effect depends on business type. Demolition or landlord issues would not discriminate by business type. And when the same data is collapsed into quarters, 1.845 drops to 1.585. Quarterly aggregation erases a third of the signal.
1.85x z=49.9
Same building · same business type
1.24x z=16.2
Same building · different business type
So the accurate statement is not “there is no contagion” but “it is not visible at that level at this resolution.”Spatial contagion is real, and it is a signal the model can use again once daily or monthly data becomes available.
Full validation report
The full validation report is provided individually
The full report, covering sample construction, the split method, layer-by-layer contribution breakdown, recalibration procedures and adjustment history, is provided individually to rating agencies, lenders and investors. CEO Byung-Geun Choi, a former fund manager, walks you through it personally.
Request the validation report