Secure AI Engineering

From DevSecOps to MLOps: Where Should AI Security Live?

Machine learning systems sit between data science, engineering, security and risk teams, and their security often belongs to none of them. With NIST's AI Risk Management Framework taking shape and regulation on the horizon, organisations need a clear answer to 'who owns AI security?'. We propose an ownership model.

By EthicalHat Research · · 3 min read
Infographic: Who owns what in AI security

Ask five people in a typical organisation who is responsible for the security of its machine learning systems and you will get five answers. The data science team thinks security handles it. The security team thinks the ML platform is an engineering concern. The platform team provides infrastructure but does not know what the models do. Risk and compliance teams are starting to ask questions, and nobody has a complete answer.

This is the organisational problem at the centre of AI security. It is becoming more urgent: NIST released the second draft of its AI Risk Management Framework this month, the European Union's proposed AI Act would impose requirements on high-risk AI systems, and ML models are influencing ever more consequential decisions.

Why AI security falls between teams

  • Data science teams focus on model quality and often work outside the standard software delivery process.
  • Security teams understand applications and infrastructure, but often lack ML expertise.
  • Platform and MLOps teams own the tooling but not the models or data.
  • Risk teams understand compliance but not the technical details.

Each team sees part of the problem. Attacks exploit the gaps between them.

An ownership model

We recommend thinking in layers, with a clear owner for each.

Infrastructure and application security (existing teams)

The cloud accounts, networks, APIs, code and dependencies around ML systems are ordinary software. Existing infrastructure and AppSec processes should cover them: configuration reviews, dependency scanning, penetration testing and monitoring. The first step is simply making sure ML systems are in scope for those processes.

The ML platform team

The platform team should embed security in shared MLOps tooling, so every model inherits it:

  • training environments with least-privilege access to data,
  • pinned, scanned base images and dependencies,
  • a model registry with controlled promotion and integrity checks,
  • serving infrastructure with authentication, rate limiting and logging, and
  • data validation steps in pipeline templates.

This is the DevSecOps principle of paved roads applied to ML.

Model owners

Each production model needs a named owner, usually the lead data scientist or product owner, responsible for:

  • understanding the model's purpose, impact and risks,
  • the provenance and appropriateness of its training data,
  • evaluation beyond accuracy (robustness, fairness, leakage), and
  • monitoring and retraining decisions.

The security team

Security provides expertise and assurance:

  • ML-aware threat modelling for high-impact systems,
  • testing, including adversarial and privacy attacks where relevant,
  • incident response plans that cover model rollback and data compromise, and
  • training for data scientists on ML threats.

Risk and governance

Risk teams set policy and risk appetite: which uses of ML require extra review, what documentation is needed, and how regulatory requirements are met. Frameworks such as NIST's draft AI RMF, organised around governing, mapping, measuring and managing AI risks, provide a useful structure.

flowchart LR
  G[Risk & governance<br/>policy, risk tiers] --> O[Model owners<br/>risk assessment per model]
  O --> P[ML platform<br/>secure defaults in pipelines]
  S[Security team<br/>threat models, testing] --> O
  S --> P
  P --> I[Infra & AppSec<br/>cloud, APIs, code]
  O -->|high-risk models| S

Making it work

Tier your models. Not every model needs the same scrutiny. A product recommendation model and a credit decision model have very different risk profiles. Define tiers, and scale requirements accordingly.

Create a lightweight review. For higher-tier models, a short review covering purpose, data, threats, evaluation and monitoring, involving the model owner, platform and security.

Build shared vocabulary. Security teams learn enough ML to ask the right questions; data scientists learn enough security to recognise threats. A security champion in each ML team helps.

Keep an inventory. You cannot govern models you do not know about. A model registry doubles as an inventory with owners, tiers and review status.

The bottom line

AI security does not need a brand-new department. It needs explicit ownership at each layer: existing teams covering the software and infrastructure, the platform team building secure defaults, model owners owning model risks, security providing expertise and assurance, and governance setting the rules. With that in place, the gaps attackers exploit start to close.

Our CISO as a Service and risk assessment teams help organisations design AI security governance that fits their structure.

EthicalHat Research

The EthicalHat research team: penetration testers, SOC analysts and engineers writing about security, software engineering and AI.

Want an expert opinion on your own systems?

Email info@ethicalhat.com or call +1-844-838-4422. Our security engineers are happy to talk it through.