adesso Blog

Interview with Barno Kaharova | Part 2 of 2

An AI agent that reads information is one thing. An agent that can execute code, overwrite files, or change business records is another. AI security expert Barno Kaharova explains why permissions, testing and accountability matter as organisations give AI systems more autonomy.

When an agent can change the system

Barno has encountered the distinction in her own work.

While working with an agent, she found that it had overwritten records she had collected. She had a backup, but she had not authorised the change. “An agent deleted the records I had collected. I had a backup, but it just deleted them.”

The code had changed from reading the file to overwriting its contents. Barno cannot say with certainty why the agent made that change. One possibility, she says, is that context became confused during a long interaction as earlier information was compressed and important details were lost.

There is a useful analogy Barno uses in her training and webinars. Imagine hiring someone straight out of university. Your experienced staff know the processes, the compliance rules, the unwritten boundaries that matter. The new hire does not yet. No sensible organisation would hand that person full access to production systems on their first day. You would restrict their permissions, pair them with a mentor, and let autonomy grow as trust grows. “Why would you give that level of access to a model,” Barno asks, “when you often don’t know who trained it, what it was trained on, or whether it has been integrated into your own processes at all?”

The analogy sharpens on one point. A new hire, even an inexperienced one, carries an internal weighing of consequences, such as reputation, career, the quiet understanding that a wrong decision has a cost they will personally bear. That weighing is what produces careful behaviour under pressure, long before any explicit rule is consulted. A model has none of it. It will not pause before an action because there is nothing it stands to lose.

The lesson is practical: organisations need to consider what an agent is permitted to do, which actions require approval and whether a mistake can be reversed.

Why traditional penetration testing is not enough

Existing security testing remains necessary, Barno stresses. But AI systems introduce additional behaviour that conventional tests may not adequately cover.

From her consulting work, Barno sees this pattern frequently. “Companies deploy agents or chatbots and run their traditional security tests, such as penetration tests and access reviews, which they still need to do. But those tests were built around a human following a defined process. A probabilistic model that can trigger a tool doesn’t follow a defined process. A workflow that is secure when a specific person operates it is not necessarily secure when a model operates it. The model will take paths a person wouldn’t.”

Language models are probabilistic, and an agent’s actions may vary with context, prompts, retrieved information, and connected tools.

“The behaviour of models and agents is not deterministic. It is inherently probabilistic.”

In her training sessions, Barno has seen teams reconsider how they approach testing. The aim is not to discard established methods. It is to adapt them to systems that can interpret untrusted language and act on it.

Testing must also continue after deployment. “Testing happens through cycles, again and again, in a loop.” Changes to a model, data source, prompt or tool connection may introduce new failure modes.

Security says robust. Data science says accurate. Who decides?

Barno also sees an organisational challenge. AI developers and model owners are often judged on accuracy and performance. Security and compliance teams focus on robustness and risk. Improving one objective can sometimes affect another, and the right trade-off depends on the application.

She expects companies to need people who can bridge the disciplines, understand both sets of concerns and help teams agree on workable approaches.

“Data science or AI teams have to work with security teams. It will be impossible not to.”

EU AI Act requirements add another reason to bring these teams together. The specific obligations and deadlines depend on the type of AI system. For Barno, the immediate question is how organisations make sound technical and governance decisions before problems reach production.

Where to start on Monday morning

Barno’s first recommendation is an inventory. “The very first thing is to know what kind of AI systems you have, in which department, and who is using them. An inventory of AI systems. Then you at least have information about what is happening.”

Next, understand and test each system in a controlled environment before deploying it. Once it is running, monitor its behaviour and reassess it as models, tools and data change.

For the next leadership meeting, five questions follow naturally from the conversation:

  • 1. Which AI systems are in use, in which departments, and by whom?
  • 2. Which of them can change records, send messages or run code, and who approved those permissions?
  • 3. Could a malicious instruction in an email or document influence their behaviour?
  • 4. When were they last tested, and what triggers a new security assessment?
  • 5. Who helps AI, security and business teams resolve competing requirements?

Barno’s final thought is reassuring in its practicality: “It is less scary if you know it can happen, because then you can mitigate it.”

Join us on 25 November

adesso Finland hosts Steps for AI Security & Governance, an executive breakfast session for CIOs, CISOs, security leaders, AI and data leaders and business executives responsible for AI initiatives. We start at 08:00 with a delicious breakfast, and the programme begins at 08:30. AI security sits between your data team and your security team. Bring your counterpart.

When: Wednesday 25 November 2026, breakfast from 08:00 and programme from 08:30

Where: adesso Finland, Porkkalankatu 5, Helsinki

Speakers: Barno Kaharova, adesso SE | Sami Jokela, CTO, 1001 Lakes

Programme

  • • 08:00 Breakfast
  • • 08:30 Welcome
  • • 08:40 AI Security Landscape & Emerging Threats
  • • 09:00 AI Governance Best Practices
  • • 09:25 Real Project Examples and Lessons Learned
  • • 09:50 Q&A

Key takeaways

  • • Identify AI-related security risks before they become incidents
  • • Understand how AI-specific attacks work
  • • Learn best practices from real enterprise AI projects
  • • Establish practical governance for AI adoption

Register here: Registration link

Read part one: Part 1


About Barno Kaharova

Barno Kaharova is a Senior Consultant and AI Security Expert at adesso SE in Frankfurt. She works on both offensive and defensive AI security, from data pipelines and models to LLM-powered applications and autonomous agents, and delivers hands-on trainings on AI red teaming, prompt injection, RAG and vector database security, and the OWASP Top 10 for LLM and agentic AI. (Speaker profile)

About Sami Jokela

Dr. Sami Jokela is a seasoned leader with 20+ years of experience in data, technology, and strategy, including roles at Nokia, co-founding startups, and leading Accenture’s technology and information insight practices.


Picture Barno Kaharova

Author Barno Kaharova

Barno Kaharova is a Senior Consultant and AI Security Expert at adesso SE in Frankfurt. She works on both offensive and defensive AI security, from data pipelines and models to LLM-powered applications and autonomous agents, and delivers hands-on trainings on AI red teaming, prompt injection, RAG and vector database security, and the OWASP Top 10 for LLM and agentic AI.

Picture Annette Kauppinen

Author Annette Kauppinen

Annette Kauppinen is a Marketing Consultant at adesso Finland. With over 20 years of experience in digital transformation and data-driven marketing, she excels in boosting brand equity and leading cross-functional teams. Annette has successfully driven market expansion across various industries, with sustainability as a core focus in her initiatives.