rules don't scale. here's what we learned building a behavior engine for a bank.

back in march, we finished building and deploying a behaviour intelligence platform for a public sector bank.

i have been wanting to write about this properly for a while. not the case study version, which you can read here. the actual technical version. what the problem looked like when we started, the decisions we made, what worked, what did not, and what i think about fraud detection differently now than i did before we started.

the starting point

the bank had a rules-based fraud detection system. this is the norm, not the exception. almost every financial institution that has been operating for more than a decade has a rules engine at the core of its fraud detection infrastructure.

rules engines are not stupid. they represent decades of accumulated institutional knowledge about how fraud actually happens. the rules encode real patterns that real analysts discovered by looking at real fraud cases. "flag transactions above X amount to a new beneficiary within Y hours of a login from a new device." "block if three failed PIN attempts within a session." "hold for review if the transaction location is more than Z kilometers from the registered address."

these rules work. they catch a meaningful fraction of fraud. they are auditable, explainable, and debuggable in ways that matter enormously in a regulated environment. when a rule fires incorrectly and a customer calls to complain, you can tell the customer exactly why the transaction was held. when a regulator asks for documentation of your fraud detection logic, you can hand them the ruleset.

so the question is not whether rules work. they do. the question is what they cannot do.

what rules cannot do

rules are static in an environment that is not.

fraud patterns evolve. the moment a rule is deployed, the adversarial signal that rule was designed to catch begins to adapt. not because fraudsters read your rulesets, but because the fraud patterns that get caught decrease in volume and the patterns that slip through increase. the ruleset reflects the fraud landscape at the time it was written, not the fraud landscape today.

adding new rules to keep up is possible but it does not scale well. a large rules engine can have hundreds or thousands of rules, many of which interact with each other in ways that are difficult to fully reason about. adding a new rule to catch a new pattern can inadvertently suppress another rule that was catching a different pattern. the combinatorial complexity of a large ruleset is genuinely hard to manage. the banks i have talked to consistently cite "rule interaction complexity" as one of the most significant operational costs of maintaining a rules-based system.

more fundamentally, rules are discrete. they carve the transaction space into buckets: flagged or not flagged, held or not held, blocked or allowed. fraud does not happen in discrete buckets. it happens on a continuous spectrum of suspiciousness, and the most sophisticated fraud, the kind that causes the most damage, is specifically designed to stay just below the threshold of every rule in the system.

a transaction that is 0.7 suspicious by any reasonable measure will pass through a rules engine cleanly if it does not meet the discrete threshold of any individual rule. this is not a theoretical vulnerability. it is how a significant fraction of actual financial fraud works.

what ML adds

the right frame for ML in fraud detection is not replacement. it is a different kind of signal.

rules detect known fraud patterns well. ML detects anomalous behavior relative to a learned baseline. these are different things and they are not in competition.

what a behaviour intelligence system does is learn what normal looks like for each entity in the system. not global normal. individual normal. what does this specific customer's transaction pattern look like over time. what times of day do they transact, in what amounts, to what category of beneficiary, from what devices and locations. the baseline is individual and temporal. it updates as the customer's behavior evolves.

a transaction that is entirely consistent with the global population, with no rule violations anywhere, can still be highly anomalous relative to that specific customer's historical behavior. that anomaly is a signal. it is not proof of fraud. it is a risk indicator that adds information to the decisioning process.

this is what we built for the bank. not a replacement for the rules engine. a parallel signal source that captures the behavioral dimension that rules cannot capture.

the architecture

the core of a behaviour intelligence platform is entity representation over time.

for every customer, the system maintains a set of features that describe their behavioral patterns. transaction frequency distributions, amount distributions, beneficiary diversity, channel usage, temporal patterns, device fingerprinting signals, geographic patterns. these features are not static. they are computed over rolling windows of different lengths to capture both recent behavior and longer-term patterns.

the feature engineering is the most important and least glamorous part of the work. the quality of the features determines the quality of the anomaly detection more than the choice of model architecture does. a good feature set with a simple model will outperform a sophisticated model on a poor feature set almost every time.

we made specific decisions for this deployment. we used asymmetric window lengths: short windows of 7 and 14 days to capture recent behavior changes, long windows of 90 and 180 days to capture stable baseline patterns. we weighted recent transactions more heavily than older ones within each window, using an exponential decay function rather than a uniform average. we separated the feature computation for different transaction types because transaction behavior varies significantly by type, and a model that treats all transactions as a single distribution loses signal.

the anomaly scoring itself uses an ensemble approach. isolation forest for global anomaly detection. LSTM autoencoders for sequential pattern anomaly detection. the combination gives you both point anomalies, transactions that are individually unusual, and contextual anomalies, transactions that are individually normal but unusual given the recent sequence of behavior.

both scores get passed to a lightweight gradient boosted classifier that has been trained on labeled fraud cases and produces a calibrated probability. calibration matters here. an uncalibrated model that produces a score of 0.8 for a transaction that has a 20% true fraud probability is worse than useless in a context where the score is being used to make decisions. calibration maps the model score to a meaningful probability.

the output is a single risk score and a feature contribution breakdown. the feature contribution breakdown exists for two reasons. first, explainability for the analyst reviewing flagged transactions. second, regulatory documentation for the very small number of cases that escalate to formal review.

what we got right

the feature engineering on asymmetric window lengths was the decision that paid off most clearly in production.

the conventional approach is to pick one window length, usually 30 or 90 days, and use it uniformly. the problem is that fraud often follows a behavioral warmup period, where the fraudster, or a compromised account, gradually shifts behavior before executing the high-value transaction. a 90-day window is long enough to wash out the warmup signal. a 7-day window catches it clearly.

we also got the calibration right on the first attempt, which i attribute almost entirely to having a good labeled dataset before we started. the bank's fraud team had done rigorous case documentation over the previous three years. the quality of the ground truth labels was higher than anything i have worked with at comparable institutions. good labels make calibration tractable. bad labels make it nearly impossible.

what we got wrong

the initial feature set was too heavy.

we included more features than we needed, partly from caution and partly from genuine uncertainty about which signals would carry weight. this created two problems. training was slower than it needed to be. and the model had too many opportunities to find spurious correlations in the training data that did not generalize to production.

the right approach, which we got to after the first production evaluation cycle, was to run feature importance analysis against the first month of production data and aggressively prune features below a contribution threshold. we eliminated roughly a third of the initial feature set without meaningful performance loss. the leaner model was faster, more interpretable, and generalized slightly better.

the second thing we underestimated was the distribution shift between the training data and production. this is always a problem, but it was more acute here than i expected. the labeled fraud cases in the training set were skewed toward older fraud patterns. the production environment included fraud patterns that had emerged more recently and were underrepresented in the training data. the model's recall on newer fraud types in the first month was lower than we wanted.

the fix was incremental retraining on a rolling window, incorporating new labeled cases from production on a regular cadence. this is standard practice but we underinvested in the automation for it in the initial build and had to retrofit it. that was avoidable.

what i think about differently now

before this engagement, i thought about the rules-versus-ML debate in fraud detection as a technical question. which approach is more accurate. which catches more fraud with fewer false positives.

it is not a technical question. it is an institutional question.

the constraint that determines how a fraud detection system can be designed is not the state of the art in anomaly detection. it is the willingness of the institution's risk team to stake decisions on a signal they cannot directly read. rules are readable. you can look at a rules engine and understand why it fired. a behavioral anomaly score is not readable in the same way. you can explain it in terms of feature contributions but you cannot point to the rule.

the ML system we built is objectively better at detecting certain categories of fraud than the rules engine it sits alongside. but it took significant relationship work with the bank's risk and compliance teams to get to a place where they trusted the signal enough to act on it. that work was as important as the engineering work.

the institutions that will get the most value from behaviour intelligence systems are the ones that invest in building that trust internally, not just in deploying the technology. the technology is tractable. the trust is the hard part.