Skip to main content

AI engineering

What AI engineering covers​

AI engineering covers the work required to build, deploy, and maintain AI systems in production. Reliability, governance, and auditability matter as much as model quality.

Work on the Feature Spec Generator from the Seattle Tech Hub raised a practical question: how should a regulated organisation adopt AI tools when every SDLC change has compliance implications? Treat the AI system like any other production system. Version it, test it, monitor it, provide rollback, and assign a clear owner.

AI engineering combines computer science, data science, domain knowledge, and software engineering. Sculley et al. showed in 2015 that model code is often a small part of a production ML system; data collection, feature extraction, serving, and monitoring account for much of the work and technical debt. The same applies to LLM agents: the model call is one component among tool boundaries, memory, retrieval, evaluation, orchestration, serving, and audit trails.

Data engineering​

Data engineering involves collecting, processing, and managing data for AI applications. In banking and financial services it carries additional weight, because data quality directly affects regulatory reporting, risk calculations, and customer outcomes. The main practices are data collection, data cleaning, and data transformation.

Data collection​

Data collection is the process of gathering data from the sources an AI application needs.

  • Web scraping: extracting data from websites, used cautiously in regulated environments because of licensing and data provenance requirements.
  • APIs: collecting data from external services. In banking this includes market data APIs (Bloomberg, Refinitiv), credit bureau APIs, and interbank messaging systems (SWIFT).
  • Databases: retrieving data from relational and non-relational stores. Enterprise AI systems in banking typically pull from data warehouses, data lakes, and operational data stores, often requiring cross-domain access approvals.
  • Event streams: consuming real time event data from message brokers (Kafka, Kinesis), increasingly important for fraud detection models that score transactions in milliseconds.

Data cleaning​

Data cleaning removes noise and inconsistencies so the data is accurate and reliable.

  • Missing values: imputing or removing missing data points. In financial datasets the approach must be documented because it can affect regulatory reporting; a model that imputes missing income data differently from the bank's standard methodology creates compliance risk.
  • Duplicates: transaction deduplication is non-trivial because the same economic event can appear in multiple source systems with different identifiers.
  • Outliers: in fraud detection, outliers are often the signal rather than the noise. The cleaning pipeline must be careful not to discard exactly the data points the model needs to learn from.

Data transformation​

Data transformation converts data into a format suitable for analysis.

  • Normalization: scaling features to a standard range, needed for models that combine features with different scales, such as transaction amounts (dollars) and transaction frequency (counts).
  • Aggregation: summarizing data at different levels of granularity, for example customer level for risk models, portfolio level for stress testing, and market level for trading strategies.
  • Feature extraction: creating new features from raw data. In banking, derived features like "velocity of transactions in the last 24 hours" or "ratio of international to domestic transactions" are often more predictive than raw transaction data.

Example: Data transformation in banking​

Consider a fraud detection system at a retail bank. Raw transaction data includes timestamp, amount, merchant category, and geographic coordinates. The transformation pipeline creates derived features: transaction velocity (transactions per hour for this customer), geographic velocity (distance between consecutive transactions divided by time elapsed), category deviation (how unusual this merchant category is for this customer), and amount deviation (z-score of this transaction amount relative to the customer's historical distribution). These engineered features improve model performance considerably compared with raw fields alone.

Model development​

Model development covers building and training machine learning models.

  • Feature engineering: creating features from raw data to improve model performance. This is often where domain expertise matters most; an experienced banking technologist knows which features are predictive, which are redundant, and which create regulatory risk.
  • Hyperparameter tuning: optimizing model parameters to achieve the best performance.
  • Model evaluation: assessing performance with metrics like accuracy, precision, and recall. In regulated environments, evaluation must also include fairness metrics, explainability assessments, and performance across protected demographic groups.

Feature engineering​

Feature engineering creates new variables from raw data to improve the performance of machine learning models. Examples:

  • Normalization: scaling features to a standard range.
  • Encoding categorical variables: one-hot encoding for low-cardinality features (merchant category), target encoding for high-cardinality features (merchant ID).
  • Interaction features: combining features to capture relationships between variables. In credit risk modeling, the interaction between income and debt load is more predictive than either feature alone.
  • Temporal features: day-of-week effects, seasonal patterns, and rolling aggregations. Financial data is inherently temporal, and ignoring time dynamics throws away predictive power.

Hyperparameter tuning​

Hyperparameter tuning optimizes the parameters of a machine learning model to achieve the best performance.

  • Grid search: exhaustively searching a specified subset of hyperparameters. Feasible for small parameter spaces but computationally expensive for complex models.
  • Random search: sampling hyperparameters from a specified distribution. Bengio and Bergstra (2012) showed that random search is more efficient than grid search for most practical problems because not all hyperparameters are equally important.
  • Bayesian optimization: using probabilistic models to find good hyperparameters, increasingly used for expensive-to-train models where each run costs significant compute.

Model evaluation​

Model evaluation assesses the performance of a machine learning model. Common metrics:

  • Accuracy: the proportion of correctly predicted instances out of the total instances.
  • Precision: the proportion of true positive predictions out of the total positive predictions. In fraud detection, high precision means fewer false alarms that waste investigator time.
  • Recall: the proportion of true positive predictions out of the total actual positives. High recall means fewer fraudulent transactions slip through undetected.
  • F1 score: the harmonic mean of precision and recall, useful when you need to balance both concerns.
  • AUC-ROC: area under the receiver operating characteristic curve, a threshold-independent measure of model discrimination.

Example: Model evaluation in fraud detection​

In a fraud detection model at a retail bank, the base rate of fraud is typically 0.1-0.3% of transactions. A model that predicts "not fraud" for every transaction reaches 99.7% accuracy and is useless. Evaluation therefore has to focus on the precision-recall tradeoff: at what threshold does the model catch 95% of fraud (recall) while keeping false positives manageable for the investigations team? This tradeoff is a business decision as much as a technical one, and it requires collaboration between data scientists, fraud operations, and risk management.

Model deployment​

Model deployment puts trained models into production. In regulated banking environments it is not a purely technical step: it requires model risk management (MRM) review, change advisory board approval, and documented rollback procedures. Key practices include:

  • Containerization: packaging and deploying models in containers.
  • Model monitoring: tracking model performance in production to detect issues.
  • Continuous integration/continuous deployment (CI/CD): automating model deployment with pipelines.
  • Agent runtime governance: managing tool permissions, memory writes, retrieval sources, reasoning budgets, and audit trails for systems that act rather than merely predict.

Containerization​

Containerization packages an AI model and its dependencies into a container such as Docker so the environment is consistent everywhere the model runs.

  • Portability: containers run on any platform that supports them. This matters in banking, where models may need to run in on-premises data centers, private cloud, and edge environments.
  • Scalability: containers can be scaled to match workload. Real-time fraud scoring has to follow transaction volume, which peaks during holidays and promotional events.
  • Isolation: containers separate applications and their dependencies. In a multi-model serving environment, isolation keeps a misbehaving model from affecting the others.
  • Reproducibility: the exact environment used during model validation is the one used in production, which eliminates "works on my machine" problems that are dangerous in regulated deployments.

Model monitoring​

Model monitoring tracks the performance of AI models in production.

  • Performance metrics: accuracy, precision, and recall on live traffic. Production performance often differs from validation performance because of data distribution shifts.
  • Data drift detection: identifying changes in the input data distribution. In banking, drift can come from macroeconomic changes (recession, interest rate shifts), seasonal effects, or changed customer behavior.
  • Concept drift detection: identifying changes in the relationship between input features and the target variable. A fraud model trained on pre-pandemic data may not hold up when transaction patterns shift.
  • Alerting: notifications connected to runbooks that describe the investigation and remediation process, so alerts are actionable.

Example: Model monitoring in production​

A churn model at a retail bank is monitored for precision and recall. After a competitor launches an aggressive rate promotion, recall drops from 85% to 60%: the model is missing customers who are leaving for rate-driven reasons because the training data did not include this competitive dynamic. Data drift detection catches the shift in the "competitor rate differential" feature distribution and triggers a retraining workflow.

Agent memory and retrieval​

Agentic systems introduce a new production concern: durable memory. Recent work on Agent Memory (arXiv:2606.06448) frames memory as a stateful workload with measurable costs across construction, retrieval, generation latency, and answer quality. In practice this means designing memory as infrastructure rather than an ever-growing prompt transcript.

Key practices include:

  • Memory types: separate confirmed domain facts, transient task state, historical decisions, user preferences, rejected assumptions, and incident lessons.
  • Provenance: every durable memory item needs source, timestamp, owner, confidence, and expiry metadata.
  • Retrieval evaluation: measure whether the agent retrieves the right files, policies, ADRs, and prior incidents before acting. Repository-aware benchmarks such as CORE-Bench (arXiv:2606.11864) show that local retrieval is a first-class engineering capability.
  • Forgetting and supersession: stale memory is a production risk. A superseded ADR or retired policy must not keep influencing agent behavior.

Continuous integration and deployment​

CI/CD for ML extends traditional CI/CD with ML-specific concerns. Google's MLOps whitepaper describes three levels of ML automation maturity:

  • Level 0 (manual): manual training, manual deployment. Common in early ML adoption.
  • Level 1 (pipeline automation): automated training pipelines and validation, but manual deployment approval.
  • Level 2 (CI/CD for ML): full automation of training, validation, and deployment with automated rollback. This is the target state for production ML systems.

Also worth practicing:

  • Automated testing: unit tests for feature engineering code, integration tests for the training pipeline, and model validation tests that check performance against baseline thresholds.
  • Version control: managing changes to the model, its code, its configuration, and its training data. DVC (Data Version Control) and MLflow are common tools for tracking model lineage.
  • Deployment automation: in regulated environments, automation includes evidence collection for audit trails, recording who approved the deployment, what validation results were observed, and what the rollback procedure is.

AI-assisted software development​

AI-driven development applies AI across the software development lifecycle. The Feature Spec Generator sponsored from the Seattle Tech Hub is one example, using LLMs to turn minimal requirements into executable BDD specifications. Key practices include:

  • Automated code generation: using AI to generate code snippets or entire functions. Codebase-native agents like Claude Code, alongside completion tools like GitHub Copilot and Cursor, have become daily-use tools for many engineering teams. See AI-powered SDLC with Claude Code for the full implementation pattern this site recommends.
  • Intelligent code review: using AI to review code for potential issues and suggest improvements. AI review can catch patterns human reviewers miss, particularly in large codebases with inconsistent coding standards.
  • Predictive analytics: using AI to predict project timelines, resource allocation, and risks. In large banking technology organizations, models for delivery performance can flag at-risk programs weeks before they miss milestones.
  • Automated testing: AI-driven testing tools, including generated test cases, mutation testing guided by ML, and intelligent test selection that runs only the tests most likely to catch regressions for a given change.

The empirical evidence on AI coding agents is maturing quickly. Agarwal, He, and Vasilescu (2026) ran a longitudinal causal analysis of LLM-based coding agents in open-source projects and found large, front loaded velocity gains, with two caveats. Projects already using IDE-based AI assistants showed minimal additional throughput gains, which points to diminishing returns from layered AI assistance. Static-analysis warnings and cognitive complexity rose by roughly 18% and 39% respectively across all scenarios, indicating sustained technical debt accumulation from AI-generated code. This supports the approach taken with the Feature Spec Generator: AI-generated artefacts must pass through the same quality gates (automated testing, security scanning, code review) as human-written code. The velocity gains are real, but they need quality safeguards alongside them.

Automated code generation​

Automated code generation uses AI to produce code from natural language descriptions or code patterns.

  • Code completion: suggestions for completing code statements. Modern tools provide multi-line completions that are contextually aware of the surrounding codebase.
  • Code synthesis: generating code from high-level specifications. The Feature Spec Generator is a domain-specific example, generating Gherkin BDD specifications from natural-language feature descriptions.
  • Template generation: boilerplate for common tasks. In enterprise environments this includes scaffolding for API endpoints, database access layers, and integration tests that follows organizational standards.

Intelligent code review​

Intelligent code review uses AI to find potential issues and suggest improvements.

  • Bug detection: AI reviewers can catch null pointer dereferences, resource leaks, and concurrency issues that are easy to miss in manual review.
  • Code quality: identifying code smells, suggesting refactoring opportunities, and enforcing architectural patterns.
  • Automated feedback: real time feedback to developers during coding. Adoption depends on a low false-positive rate; if the tool cries wolf too often, developers will ignore it.

The false-positive-rate problem is why an in-context reviewer beats a rules-only linter for architectural and security judgment calls. Claude Code Action run as a pre-commit review step (see the full worked GitHub Actions example in AI-powered SDLC Patterns) reads the diff alongside the actual repository, the existing ADRs, and service boundaries, instead of pattern-matching the diff in isolation. That is what keeps its false-positive rate low enough for developers to keep trusting it after the first false alarm.

Predictive analytics​

Predictive analytics forecasts project timelines, resource allocation, and risk.

  • Time series analysis: delivery velocity, defect escape rates, and infrastructure costs all exhibit temporal patterns that can be modeled.
  • Resource optimization: in banking technology organizations with thousands of engineers, even small improvements in resource allocation translate to real cost savings.
  • Risk assessment: models trained on historical delivery data can identify the characteristics of projects that are likely to miss deadlines or exceed budgets.

Automated testing​

Automated testing applies AI-driven tools to bug finding and test coverage.

  • Test case generation: AI generates test cases from code analysis. The Feature Spec Generator applies the same approach to requirements, including edge cases for human review.
  • Test optimization: intelligent test selection uses change analysis and historical failure data to run the tests most likely to catch problems first, cutting feedback time.
  • Bug detection: AI-powered fuzzing and property-based testing can find bugs that conventional test suites miss.

AI development tools​

  • Claude Code: Anthropic's codebase-native agentic CLI. It reads a repository, edits files, runs commands, and integrates with developer tools directly, rather than only completing the current line. It extends to design handoff through Claude Design and to non-developer agentic work through Claude Cowork, which shares the same agentic core. This site's default recommendation; see AI-powered SDLC with Claude Code.
  • GitHub Copilot: an AI-powered code completion tool. It is widely adopted across enterprise engineering teams, with measurable productivity improvements reported by Ziegler et al. (2024).
  • Cursor: an AI-native IDE that combines code generation, editing, and codebase understanding.
  • DeepCode / Snyk Code: an AI-based code review tool that identifies bugs and security vulnerabilities and suggests improvements. It integrates into CI/CD pipelines as an automated quality gate.
  • TabNine: an AI code completion tool that supports multiple programming languages and can be trained on proprietary codebases.
  • Snyk: a security tool that uses AI to find and fix vulnerabilities in code and dependencies, important for DevSecOps pipelines in regulated environments.
  • SonarQube: a code quality tool that uses static analysis and AI-powered rules to analyze code and surface actionable findings.

AI-driven development is moving toward multi-agent systems. Fu, Pasuksmit, and Tantithamthavorn (2024) surveyed 99 papers and identified 12 distinct security tasks in DevSecOps that AI can address, from vulnerability detection to patch generation to compliance verification. Their analysis of 65 benchmarks found that individual AI tools excel at specific tasks, while combinations of specialised agents, each covering a different phase of the SDLC, outperform monolithic assistants. OpenClaw uses this multi-agent architecture: specialised sub-agents handle coding, security analysis, testing, and deployment tasks under the coordination of a central orchestrator.

DORA metrics for AI engineering​

DORA metrics measure the throughput and stability of AI engineering work. Research by Forsgren, Humble, and Kim links stronger software delivery performance with organisational performance. The four measures:

  • Deployment frequency: how often new code is deployed to production. For ML systems this includes both application code deployments and model deployments, which may follow different cadences.
  • Lead time for changes: the time from commit to production. For ML systems this includes the time from identifying a model performance issue to deploying a retrained model.
  • Change failure rate: the percentage of changes that fail in production. For ML systems this includes model deployments that cause performance degradation, serving errors, or fairness violations.
  • Mean time to restore (MTTR): the average time to restore service after a failure. For ML systems this depends on automated rollback that can revert quickly to a previous model version.

Applying DORA metrics to AI engineering​

  1. Deployment frequency: increasing the frequency of model deployments lets teams iterate on improvements and ship features faster. At the Tier-1 bank, the target was to move from quarterly model releases to monthly releases for low-risk models, with a path to continuous deployment for models with automated validation gates.

  2. Lead time for changes: automating the data pipeline, model training, and validation cut model update lead time from weeks to days. The bottleneck then shifted from engineering to model risk management review, which prompted a parallel effort to streamline the MRM process for low-materiality model changes.

  3. Change failure rate: the team made shadow deployment (running the new model alongside the existing model and comparing outputs) standard practice, catching performance regressions before they affected customers.

  4. MTTR: automated rollback that can revert to the previous model version within minutes is essential. The team also added circuit breakers that fall back to rule-based systems when the ML model is unavailable, keeping service running.

June 2026 research update​

Recent arXiv publications add to the empirical and theoretical foundations of AI engineering. The following research informs the practices described in this document.

Agent software and repository-aware engineering​

Agentic Software (arXiv:2606.05608) argues that AI agents are restructuring the software paradigm itself: the harness, tool layer, memory layer, evaluation suite, and runtime policy become part of the production system. Dialogue SWE-Bench (arXiv:2606.13995) adds an evaluation dimension: agents should be judged on whether they can ask useful clarifying questions, in addition to whether they can emit a patch. CORE-Bench (arXiv:2606.11864) shows that repository-local retrieval remains a hard problem, which is why AI engineering teams must measure retrieval quality before trusting generated code or tests.

Automated code generation research​

Agentic coding has moved from individual tool use to autonomous development pipelines. OpenDev (arXiv:2603.05344) is a terminal-native, Rust-based autonomous coding agent for long-horizon development tasks with integrated safety controls. At production scale, Stripe's Minions programme now generates thousands of production pull requests weekly using autonomous agents, showing that agentic code generation is viable outside research settings. Both developments point the same direction as Agarwal, He, and Vasilescu (2026): teams are shifting from AI-assisted development toward AI-orchestrated development, with humans reviewing agent-generated artefacts rather than writing code directly.

Automated testing research​

The evidence for AI-generated testing has strengthened. AI-Generated Tests in Real-World Repos (arXiv:2603.13724) provides large-scale evidence that AI agents authored 16.4% of test-adding commits across real repositories. AI-generated tests show higher assertion density and lower cyclomatic complexity than human-written tests, with comparable coverage metrics. This supports investing in AI-driven test generation as a quality engineering practice, and it supports reviewing AI-generated tests with the same rigour as human-written ones.

Intelligent code review research​

Agentic Code Reasoning (arXiv:2603.01896) introduces a semi-formal reasoning methodology for LLM agents navigating codebases without execution. Grounding reasoning in static structural analysis improves patch equivalence verification and fault localization, both directly applicable to AI-powered code review pipelines. The work extends the multi-agent DevSecOps picture from Fu, Pasuksmit, and Tantithamthavorn (2024) with more rigorous semantic reasoning at the review stage.

Agent Trajectories as Programs (arXiv:2606.16988) adds another review surface: inspect the agent path, not only the final diff. Which files were read, which commands were run, which tests were skipped, and which assumptions were carried forward all become evidence for or against trusting the change.

Model deployment and LLMOps​

The Navigating MLOps framework (arXiv:2503.15577) formalises the operational requirements of deploying LLM-based agents in production: a unified MLOps lifecycle that explicitly incorporates LLM Operations (LLMOps), extending the three-level MLOps maturity model with LLM-specific concerns such as prompt versioning, chain-of-thought auditing, and agent behaviour monitoring. The paper also outlines the roles, tools, and costs associated with MLOps adoption at various maturity levels. It complements the CI/CD and model monitoring practices described in this document; consult it when designing deployment pipelines for agentic systems.

Production optimisation​

Production optimisation has shifted from generic autoscaling to LLM-specific operations research. DNN-Powered MLOps Pipeline Optimization for Large Language Models (arXiv:2501.14802) applies neural networks to automate deployment decisions and resource allocation within MLOps orchestration for LLM pipelines, reducing the manual tuning burden on platform teams. Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints (arXiv:2504.11320) derives the WAIT algorithm from a fluid-dynamics approximation of LLM serving. The newer production-serving literature, including RTP-LLM (arXiv:2605.29639), speculative decoding latency modeling (arXiv:2605.15051), and LLM Serving Needs Mathematical Optimization (arXiv:2605.01280), makes the same point more forcefully: request routing, batching, KV-cache pressure, prefill/decode separation, and fallback activation are reliability concerns, not only cost optimizations.

References​