Post-Doctoral Research Visit – From AI audits to AI security: an information gain hierarchy

Type de contrat : CDD Contexte et atouts du poste AI-based models are now core to a wide range of applications, including highly critical ones. The stakes are considerable for companies or institutions deploying them, as their training amounts to up to a billion dollars (e.g. for the training of ChatGPT). This clearly calls for defending them against attacks, like copy/extraction. In parallel, institutions such as state regulators have to ensure that these models operate according to law, in particular with regards to possible discrimination [1]. Researchers are then tasked to provide algorithms to auditors for assessing important metrics regarding deployed AI-based models. In such black-box audits, where an auditor has no access to the remotely operated model's internals, the goal is to stealthily (i.e. with a few queries only) estimate some metrics, such as fairness [6]. Interestingly, and this has not yet been mentioned in the literature, this audit setup is very close to offensive information gain, from a conceptual standpoint. Indeed, potential attackers are incentivized to try and leak information out of deployed models [4,10]. Motivations range from economic intelligence, obtaining implementation details, to simply avoiding development costs by copying deployed models. An auditor is interested in stealthy model observation [3], to avoid disrupting the audited model by using too many queries. Identically, an attacker also desire stealthiness, here to avoid being detected and cut off. In particular, auditors generally want to obtain precise property estimation, yet confined to a single feature (e.g. male/female fairness), while attackers aim at having a global picture of the model (for basic copy, or evading some parameter set). Thus, there is an avenue to devise offensive methods in between stealthy audits and global attacks, to try and leak novel model characteristics. The ambition of our group is to bridge the gap between these two critical setups: legal auditing and offensive security, in the domain of modern deployed AI models. From this unique standpoint, and from the body of work in the field of AI auditing, we expect to find new insights for attacking and defending deployed AI models, by finding novel angles. For instance, we proposed a unified way to approach model fingerprinting [2] that is of interest for an auditor to guess which model she is observing on a platform; we conjecture that leveraging such an approach to measure the evolution in time of such a model (does the model changes due to updates?) is of core interest for an attacker, as she can derive what is at play at the company hosting this model. This could provide ground for the attacker for economic intelligence, while leaking some precious information that has to be defended by the attacked company. Mission confiée • Research • Working with Ph.D. students from the group Principales activités A striking remark when looking at the current types of attacks on AI models is their quantity and apparent independence (see [10] Fig. 3): each is treated as a separate domain. In addition to this list of attacks, we claim that an audit may be viewed as the leak of a feature from a production model, and must be considered as a potential threat. In that light, clarifications in the relation between these attacks might come from a systematic study of how they relate with regards to the setup they operate in, versus the information gain they permit. We propose to work on a hierarchy of attacks, that will uncover the smallest attacks (in terms of assumptions and scope) and how they might be composed into larger attacks, and so on. This hierarchy will reveal unexplored configurations, where several simple attacks will be combined to build richer attacks. This hierarchy will provide the missing link between audits and AI security, bridging the two in a formal way. The postdoc candidate will leverage algorithmic background, to devise a hierarchy, in a parallel to the Herlihy hierarchy in algorithms. We intend to use the notion of "distinguishability" [14] as a hierarchy backbone (to assess if an attack leaks data permitting strong or weak distinguishability of models). In particular, the field of "property testing" will be related to this hierarchy. # References [1] Le Merrer, E., Pons, R., & Tredan, G. (2024). Algorithmic audits of algorithms, and the law. AI and Ethics, 4(4), 1365-1375. [2] Godinot, A., Le Merrer, E., Penzo, C., Taïani, F., & Tredan, G. (2025). Queries, Representation & Detection: The Next 100 Model Fingerprinting Schemes. In AAAI. [3] Le Merrer, E., & Tredan, G. (2020) Remote explainability faces the bouncer problem. Nature machine intelligence, 2(9), 529-539. [4] Maho, T., Furon, T., & Le Merrer, E. (2021). Surfree: a fast surrogate-free black-box attack. In CVPR. [5] Godinot, A., Le Merrer, E., Tredan, G., Penzo, C., & Taïani, F. (2024). Under manipulations, are some AI models harder to audit?. In IEEE Conference on Secur