AI models can provably hide arbitrary capabilities
The paper shows encrypted computations can be hidden in model weights and resist common elicitation, but the discussion questions whether that is a practical threat or a narrow, detectable demo.
An arXiv paper accepted at NeurIPS 2026 argues that capability evaluations assume hidden capabilities can be elicited, and shows that assumption fails under adversarial conditions.
The authors construct backdoor attacks that embed encrypted neural networks inside host model weights, decrypting and executing them only upon a secret trigger; using digital lockers, they claim the hidden circuits remain provably hard to elicit or interpret under cryptographic assumptions even with full weight access, extending prior work from hiding fixed strings to arbitrary computations.
They implement proof-of-concept constructions in PyTorch and test elicitation via fine-tuning, weight noising, gradient-based trigger search, and latent adversarial training on a small chemical reaction prediction task as a CBRN proxy, reporting resistance. The authors frame this as a limit of elicitation-based capability audits and release their models to support defenses.
Key takeaways
- Threat model: an adversary embeds an encrypted neural network in host weights; a secret trigger decrypts and executes the hidden computation.
- Digital lockers provide the cryptographic guarantee: hidden circuits are claimed hard to elicit or interpret even with full model-weight access, generalizing earlier fixed-string hiding.
- Proof-of-concept implemented in PyTorch; elicitation attempts include fine-tuning, weight noising, gradient-based trigger search, and latent adversarial training.
- Evaluation uses a small chemical reaction prediction task as a proxy for CBRN-relevant capabilities; the authors report resistance to the tested elicitation methods.
- The paper argues this exposes a limit of elicitation-based capability audits and releases the models to support defense development.
- Accepted at NeurIPS 2026; 22 pages, 6 figures, 3 tables.
What people think
Analysis details
14 comments analyzed. Analyzed .
The sample selects active discussion branches and includes available parent comments. It may omit parts of the full thread. The analysis included 14 of 19 usable stored comments because of the input limit. Selected themes do not measure community opinion or how common a view is.
Backdoors and trigger-activated behavior
The core security worry is that arbitrary strings could activate hidden capabilities, especially in open-weight models, and that reverse-engineering or obfuscating triggers remains unresolved.
Open- vs closed-weight deployment risk
Comments debate whether hidden capabilities matter more for open-weight releases, with local web-search connectors, or whether closed vendors have similar or greater incentive and opportunity.
Detection and reverse-engineering difficulty
One thread questions how hidden triggers could be discovered and how obfuscation would evolve; another calls the demo trivially detectable but does not specify a method.
Anecdotal evidence of latent memorization
A commenter reports a 1B-model experiment where an arbitrary answer vanished then spontaneously reappeared after unrelated training; others compare it to latent evolutionary precursors and vector nudging.
Sleeper agents and unaligned latent content
Concerns focus on models containing hidden, unaligned knowledge or trigger-activated behaviors that fine-tuning could reinforce, with comparisons to sleeper agents.