One of the world’s largest professional services and audit companies built an AI assistant into its internal tax technology platform, a central application that connects to all its clients and runs core business functions. Tax professionals could use the assistant through chat and voice to manage tasks, reminders, documents, and client work areas.
The assistant used the Model Context Protocol (MCP) to connect to tools and data sources, allowing it to access client tax data and take action on a user’s behalf.
A pilot group was already using the assistant, and the business was moving quickly on AI. Before expanding access across the company, leadership needed evidence of how the assistant and its supporting infrastructure would hold up under attack and misuse.
UltraViolet worked with the security team to define an adversarial testing scope focused on the company’s specific risks.
“Your proposal was the strongest, with detailed methodology customized to meet our requirements.”
- Associate Director, Global Professional Services and Audit Company
The testing focused on three questions:
Answering them meant looking beyond the chat window. AI platform breaches don't always start with the model. Some begin in exposed APIs and supporting infrastructure that the assistant's own interface never touches.
In a single three-week engagement, UltraViolet combined three disciplines:
Findings from attack surface discovery and cloud configuration review shaped the adversarial test scenarios. Each phase informed the next, connecting the infrastructure review to the model-level testing.
1. Attack surface discovery and cloud configuration review
UltraViolet mapped externally reachable assets tied to the assistant, including subdomains, API documentation, and exposed services. The team also reviewed supporting cloud services: storage, messaging, API management, Key Vault, and AI resources such as AI Foundry and Azure OpenAI.
Testers did not have standing access to the company’s Azure environment. Instead, they worked with the company’s DevOps team in guided sessions, using Azure command-line interface (CLI) queries to review the environment within those access constraints.
2. Adversarial scenario design
Based on the public-facing services and unclear trust boundaries identified in those reviews, testers prioritized authorization enforcement, misuse of connected tools and agents, and indirect prompt injection through uploaded content.
3. Adversarial validation
Testing covered jailbreak attempts, system prompt extraction, tool and agent invocation abuse, cross-user access attempts, and attacks through both chat and voice.
An open-source LLM vulnerability scanner supported the manual testing. The assistant's responses streamed as server-sent events that the scanner couldn't fully parse, so a flagged result by the scanner didn’t always indicate a security issue. Testers manually checked every automated result to separate confirmed issues from scanner limitations. Where a finding traced back to attack surface discovery or cloud review, the report made that connection explicit.
Authorization controls, harmful-content safeguards, and voice guardrails held during testing
Testers could not access another user’s chat history, files, reminders, or tasks. Attempts to invoke tools by naming them directly, or to make one agent invoke another outside its normal flow, also failed. The authorization model governing the assistant's actions through tools and agents held up against these attempts.
The assistant did not return synthetic personal and non-public test data planted in prompts and files during testing. Across automated and manual tests, it consistently refused prompts requesting harmful or biased content or assistance with criminal activity.
Malformed audio, high-frequency tones, and Morse code also failed to bypass the voice guardrails or reach the model as unfiltered input.
Model-level testing exposed disclosure and task-handling weaknesses
A handful of ordinary prompts exposed details of the assistant’s internal architecture in the application’s response stream. Testers could identify its central orchestrator, specialized agents for documents, workplans, and reminders, and the names of internal functions those agents called.
The assistant refused direct requests for its system prompt. However, asking for that information in JSON or XML format reliably exposed portions of internal instructions and context, including guardrail text, internal user context fields, and notes about information it wasn’t supposed to disclose.
The responses also revealed version numbers for the current and next releases, planned integrations with other firm applications, a pilot rollout timeline, and a feature still in development elsewhere in the practice.
A widely circulated roleplay jailbreak prompted the assistant to perform tasks outside its tax-related scope, including writing code and decoding an encoded message. The assistant still refused clearly harmful requests, so the jailbreak broadened the tasks it would perform without bypassing those safeguards during testing.
In another test, the assistant claimed it had created a reminder and later listed it, even though the reminder had never been created. Separately, it returned a reminder in another language that referenced an unrelated consumer app, along with a raw internal function call. Neither the reminder nor the function call was connected to the test account’s data.
Cloud configuration review identified access, encryption, and monitoring gaps
Weaknesses included:
During this engagement, testers could not exploit the Azure App Service’s direct internet exposure or the storage accounts’ use of shared-key authentication from outside the environment.
Both weaknesses were still rated high severity based on what an attacker with a compromised identity or a foothold elsewhere in the environment might be able to access. The report explained the limits of external testing and the reasoning behind those ratings.
The engagement produced 16 findings (2 high, 11 medium, 3 low, none critical), mapped to CWE identifiers and the OWASP LLM Top 10 where applicable. None of the findings gave testers a path to unauthorized access on its own. Together, they showed what an attacker could learn about the system, and which weaknesses could matter if an identity or another part of the environment were compromised.
Each finding included evidence and a specific recommended fix in the affected services. Severity ratings and their explanations helped the team prioritize remediation, while reproduction steps gave engineers a way to repeat each attack and check the fix before UltraViolet retested.
Recommended fixes included:
The company’s primary contact reviewed the final report, accepted the findings, and scheduled a retest.
"The report was a very detailed read. Every finding came with the evidence and reasoning our team needed to act on it, and that gave us confidence in the fixes we required before expanding access."
- Associate Director, Global Professional Services and Audit Company
The assessment gave the security team three types of evidence to support the company's decision about expanding access:
The company made remediation of every finding a requirement for broader rollout. Subsequent retesting confirmed that most findings had been resolved.
For leadership, the assessment and retest results clarified progress against that requirement: which fixes had been validated and which findings still needed remediation before access could expand.
UltraViolet Cyber is the only security operations partner that unifies red, blue, and purple team capabilities into one integrated program. Built by former U.S. intelligence community operators, UltraViolet brings the same offensive and defensive rigor to AI that it applies across the enterprise. Its work spans adversarial testing of models, pipelines, and agents to pre-production threat modeling and board-ready governance frameworks.
That offensive and defensive rigor showed in this engagement. Beyond testing how the assistant could be attacked, the review flagged two gaps that would limit the security team's ability to detect and investigate attacks: security alerts routed to DevOps and disabled audit logging on the Azure OpenAI resource.
Talk with our team about your AI application, access constraints, and rollout plans. We can help define what to test and the evidence your team needs to make an informed decision.