How-to Explain Model Outcomes Without Sharing Internal Logic

Please Remember...

This is a fictional example and not real documentation. The purpose is to demonstrate my technical writing.

Important to Know

Quant models don’t prove guilt. They highlight trades that look unusual based on known patterns.
Further investigation is always needed. The model is a tool to guide your attention, not make final judgments.

← tutorial: using the logistic regression formula with sample data | quant models (logistic regression formula)

If you're building a product or documentation set and need to communicate quant model behaviour without exposing proprietary details, the structure below may help.

Scenario

You (as an employee of Company-X) need to explain to a customer why a trade was flagged as potentially fraudulent, but without revealing Company-X’s proprietary logistic regression formula.

Let’s walk through how to do that.

Example Trade, Model Setup, and Outcome

This example uses the same trade data, model setup, and resulting ~99.82% fraud probability as in the previous tutorial. The only change is how we present the outcome without disclosing internal model logic.

Explaining the Outcome Without Revealing the Formula

Flagged Trade

The customer (Customer1) sent us:

Trade ID: 123456
Trade size: £250,000
Time of trade: 02:00 AM
Trader: Jo Bloggs

Trade Enrichment

We enriched this trade using internal data sources and behavioural analysis. Our system observed the following:

The trader has a high historical risk profile
The trade occurred at an unusual time for this specific trader
The volume and frequency of recent trades were notably higher than usual
The trading pattern differed from Jo Bloggs’ historical activity

Explanation

This trade triggered a high-risk alert based on several contributing signals. While we don’t disclose the full model structure, here’s what contributed to the flag:

Trader's Historical Profile

Jo Bloggs has a history of elevated risk scores in our system. This is based on aggregated behavioural patterns, including trade timing, counterparty variation, and post-trade corrections. These were collected over several months. While this may not align with your internal perception, our model is calibrated on market-wide data and historical outcomes.

Trade Timing

Although 2:00 AM may be appropriate in some trading contexts, our system flagged this time as atypical for this specific trader. Jo Bloggs typically operates during standard London trading hours. On its own, this wouldn’t raise concern, but it adds weight when combined with other signals.

Trade Frequency Spike

In the hour prior to this trade, Jo Bloggs executed 15 trades which is a notably higher frequency than their usual volume profile. Our system uses rolling time windows and dynamically adjusts thresholds. This spike indicated a short-term behavioural anomaly.

Pattern Deviation

The system also detects higher-level patterns across multiple features such as sequencing, timing gaps, instrument variety, and flow structure. The pattern of this trade differed from Jo Bloggs’ usual behaviour. While the deviation wasn’t extreme, it contributed to the overall risk signal when evaluated alongside other factors.

Flag Raised

Based on the combination of these inputs, our model assigned a fraud probability score above 99%.

Why This Matters

We understand that internal context (such as desk strategy, compliance expectations, or known trader patterns) plays a crucial role.

Our model provides a statistical signal, not a judgment, and is best used as a starting point for internal investigation and not as a conclusive determination.

What You Can Do Now

We recommend following up internally with:

Investigating further by following your internal policies
Reviewing the authorisation trail for this trade
Cross-checking with related trades in the same timeframe
Looking into Jo Bloggs’ recent activity patterns
Adding contextual data (e.g., emails, desk notes, strategy justifications)

A Note on Transparency

At Company-X, we aim to strike the right balance:

Transparent enough for you to trust the outcomes
Protected enough to safeguard proprietary models and logic