LLMs and Contextual Integrity
Overview
Recent research has raised concerns about the privacy risks associated with Large Language Models (LLMs) that utilize persistent memory from past interactions. A benchmark called CIMemories was introduced to evaluate how these models manage sensitive information based on context. The study revealed that leading models, like GPT-5, can leak personal attributes in inappropriate contexts, with violation rates increasing significantly as usage grows. For example, violations jumped from 0.1% to 25.1% when the same prompt was repeated multiple times. These findings indicate that current models struggle with making nuanced decisions about information sharing, suggesting a need for improved context-aware reasoning capabilities.
Key Takeaways
- Affected Systems: GPT-5 and other Large Language Models utilizing persistent memory
- Action Required: Develop contextually aware reasoning capabilities to improve information flow control.
- Timeline: Newly disclosed
Original Article Summary
I have been thinking a lot about AI and integrity. Part of that is contextual integrity. I recently found two papers on the topic. “CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs“: Abstract: Large Language Models (LLMs) increasingly use persistent memory from past interactions to enhance personalization and task performance. However, this memory introduces critical risks when sensitive information is revealed in inappropriate contexts. We present CIMemories, a benchmark for evaluating whether LLMs appropriately control information flow from memory based on task context. CIMemories uses synthetic user profiles with over 100 attributes per user, paired with diverse task contexts in which each attribute may be essential for some tasks but inappropriate for others. Our evaluation reveals that frontier models exhibit up to 69% attribute-level violations (leaking information inappropriately), with lower violation rates often coming at the cost of task utility. Violations accumulate across both tasks and runs: as usage increases from 1 to 40 tasks, GPT-5’s violations rise from 0.1% to 9.6%, reaching 25.1% when the same prompt is executed 5 times, revealing arbitrary and unstable behavior in which models leak different attributes for identical prompts. Privacy-conscious prompting does not solve this—models overgeneralize, sharing everything or nothing rather than making nuanced, context-dependent decisions. These findings reveal fundamental limitations that require contextually aware reasoning capabilities, not just better prompting or scaling...
Impact
GPT-5 and other Large Language Models utilizing persistent memory
Exploitation Status
No active exploitation has been reported at this time. However, organizations should still apply patches promptly as proof-of-concept code may exist.
Timeline
Newly disclosed
Remediation
Develop contextually aware reasoning capabilities to improve information flow control
Additional Information
This threat intelligence is aggregated from trusted cybersecurity sources. For the most up-to-date information, technical details, and official vendor guidance, please refer to the original article linked below.
Related Topics: This incident relates to Critical.