ILS-Bench: Investor Language-to-Suitability Benchmark
View the dataset on Mendeley DataThe neural-network advisor on the AI advisor page is trained on ILS-Bench, published by Marco Bonelli (Ca' Foscari University of Venice) on Mendeley Data on 16 June 2026, version 1, DOI 10.17632/w48mh2dtg5.1, licence CC BY 4.0. The dataset describes itself as a synthetic benchmark for evaluating suitability-aware robo-advisory systems that convert investor language into portfolio and escalation decisions. It contains 400 AI-assisted synthetic investor narratives, no real client data or personal information, each reviewed by four independent financial-domain experts who agreed, disagreed or revised the proposed suitability labels. The consensus of the panel is the label AdviceIT learns from.
Citation. Bonelli, M. (2026). ILS-Bench: Investor Language-to-Suitability Benchmark. Mendeley Data, V1. https://doi.org/10.17632/w48mh2dtg5.1
What each case contains
The fields of the Cases_Consensus sheet, from the dataset's codebook.
| Field | Values | Meaning |
|---|---|---|
| Investor narrative | free text, 317 to 538 characters | The investor's own description of goals, buffers, income, horizon and attitude to risk. |
| Evidence span | quoted sentences | The parts of the narrative the labels rest on. |
| Risk tolerance | Low, Moderate, High, Inconsistent | Willingness and emotional ability to accept volatility. Inconsistent when statements conflict. |
| Risk capacity | Low, Moderate, High | Financial ability to absorb losses given income, savings, debt and obligations. |
| Liquidity need | Low, Moderate, High, Urgent | Need to access the funds within a short or medium horizon. |
| Suitability risk | Low, Medium, High | Overall risk that automated advice may be unsuitable or unsafe. |
| Portfolio | Capital preservation, Conservative, Balanced, Growth, Aggressive growth, Human review | Recommended portfolio class, or a non-automated review outcome. |
| Escalation | Yes, No | Whether a human adviser should review the case before advice is finalised. Identical to Portfolio = Human review in all 400 cases. |
| Consensus | expert majority | Majority label of the four validators, Unsure votes excluded. Eight cases carry a review flag for weak consensus. |
The expert panel
- Expert 1: retired Boston portfolio manager (portfolio suitability, risk capacity, allocation realism)
- Expert 2: senior New York head trader (trading risk, downside protection, market-risk perspective)
- Expert 3: Shenzhen FinTech executive (provider-side implementation, automation risk, escalation discipline)
- Expert 4: European FinTech academic (research-design consistency, FinTech classification validity)
Across the 1600 validations, 1316 were Agree, 223 Disagree and 61 Unsure. The dataset author's own draft portfolio label agreed with the final consensus in 88.2 percent of cases, individual expert votes in 94.7 percent.
The 400 cases in numbers
Computed live from ils_bench_cases.js, the extract used for training.
How AdviceIT uses it
The procedure
The benchmark's pipeline is investor language, then structured suitability labels, then a portfolio class or escalation to a human. AdviceIT follows it. Step one, language to labels, is done either by the profile form through three documented rules, or by the in-browser language model reading a free-text description ("Read description into the form" on the advisor pages). Step two, labels to outcome, is what the network learned from the expert consensus.
From the form to the labels
- Risk tolerance: stated Low / Medium / High becomes Low / Moderate / High. Inconsistent only when a written description shows conflicting attitudes.
- Risk capacity: counts what could force selling at a loss (no emergency fund, variable income, significant debt or obligations). None: High. One: Moderate. Two or more: Low.
- Liquidity need: horizon 1 to 2 years: Urgent. 3 to 5: High. 6 to 10: Moderate. 11 or more: Low. A concrete near-term need makes it Urgent whatever the horizon.
The network
Twelve inputs (the three labels one-hot encoded plus age, which every narrative states), two hidden layers of 16 ReLU units, six outputs. Trained with Adam, cross-entropy and L2 weight decay, written from scratch in numpy in ml/train_model.py, seeded and reproducible. Probabilities calibrated with temperature scaling on out-of-fold predictions. The weights are exported to ml_weights.js and run in the browser in ml_model.js.
Human review on both advisors
The panel refused to automate almost half of the cases. The network learned that. Both advisors learned it, the network and the interpretable rule-based advisor, so both can answer Human review instead of a portfolio.
Reproduce
cd ml
pip install numpy openpyxl
python3 fetch_ils_bench.py
python3 train_model.pyTraining results
Read live from the meta block of ml_weights.js, written by the training script. Cross-validation is stratified 5-fold, repeated three times.
Benchmark the language-reading step
The in-browser language model reads each narrative into the five form fields, the form rules turn them into the three suitability labels, and the advisors turn the labels into an outcome. Each is compared with the expert consensus. Cases are drawn in a fixed order (every k-th case) so runs are comparable. Needs WebGPU and the page served over http, like the conversational condition.
Browse the cases
Each row shows the expert consensus. "Open in the AI advisor" pastes the narrative into the advisor page so the language model can read it and the advisors can be compared with the panel.
| Case | Narrative | Tolerance | Capacity | Liquidity | Outcome |
|---|