Model Extraction

Model Extraction: Reconstructing a model purely through systematic queries to its interface. The attack targets the intellectual property embodied in the model and belongs to adversarial machine learningAdversarial Machine LearningDiscipline concerning the manipulation, deception, and securing of machine learning models..

How it works and where it fits

The attacker uses the target model as an oracle: they send deliberately chosen inputs, collect the outputs, and train their own model on those pairs. The more information a response carries — confidence scores or full probability distributions rather than a single class — the fewer queries are needed. The result need not replicate the architecture, only approximate the behaviour closely enough.

Practical security relevance

The commercial damage lies in lost training investment and competitive advantage. More significant for security is that an extracted model serves as a local testing ground: attacks can be developed against it and then transferred to the original, without leaving traces in the operator’s logs. Countermeasures include rate limiting, per-account query quotas, reduced output detail, detection of atypical query patterns, and watermarking model behaviour.

Related concepts

• Adversarial Machine LearningAdversarial Machine LearningDiscipline concerning the manipulation, deception, and securing of machine learning models.: Discipline concerning the manipulation, deception, and securing of machine learning models.
• AI Penetration TestingAI Penetration TestingAuthorized security testing of AI and machine-learning systems for vulnerabilities such as prompt injection, model extraction, and training data leakage.: Authorized security testing of AI and machine-learning systems for vulnerabilities such as prompt injection, model extraction, and training data leakage.
• Data ExfiltrationData ExfiltrationUnauthorized transfer or theft of data from an organization.: Unauthorized transfer or theft of data from an organization.
• Rate LimitingRate LimitingLimitation on the number of permitted requests or actions within a specific time window.: Limitation on the number of permitted requests or actions within a specific time window.