Humans in the Loop, Accountability in Limbo

This October, the Future Forces exhibition and summit came to my own city, Prague, running from 21 to 23 October 2026. Czech President Petr Pavel opened the event, setting the tone for three days in which defense was treated as a meeting point for technology, policy and the people who use both. The organizers counted more than 12,500 participants from over 80 countries, including governments, armed forces, industry, research institutions and international organizations. More than 530 exhibitors from 38 countries filled the halls with new technologies, world premieres and practical solutions for defense and security. Away from the exhibition floor, over 40 conferences, panel discussions, round tables and workshops ran across the three days, with 250 international speakers. I moved between several of them, but one stayed with me longer than the rest: a panel on software for critical infrastructure.

Freelance technology journalist Fergus O’Sullivan moderated the session. Beside him sat three people who approached the problem from very different places. Michal Homza is head of programs at Helsing, the European defense AI company. Jiří Brotánek spent more than 15 years running IT for the Czech Fire Brigade and now builds large state information systems in the private sector. Col. Dušan Uhlík heads the crisis management unit at the Ministry of the Interior, the body that regulates the resilience of critical infrastructure.

Who Writes the Rules?

The conversation began with a question that sounds simple and is not: how do you set the rules by which a critical system operates?

The defense industry view started with a distinction that is often blurred in public debate, between autonomous systems and AI-enabled ones. In this view, AI exists to help people decide faster and more reliably, while the human stays at the center of the process. Every country wants full control over the systems it deploys, and responsible manufacturers carry that expectation through development, testing and deployment.

The emergency services view brought the discussion down to earth. Rules, it held, only work if they are tested in regular exercises and training, because real emergencies knock people out of their routines.

The state began elsewhere: with the law. The legal framework comes first, including the rules governing AI and cybersecurity. But Col. Uhlík argued that law alone is not enough. Systems should be built with resilience in mind, and the last step in any decision chain must belong to a human.

Where the Human Steps In

The heart of the debate came down to one question: at what point should a human step into an automated process? The panelists answered from legal, technical and state perspectives, yet they arrived at the same place. The human remains essential.

Software, as they described it, is a helper. It speeds up decisions made under time pressure and with incomplete information, but it never removes the human factor. The one exception is cybersecurity. Attacks there unfold in milliseconds, far faster than any person can react, so automated defense carries much of the load. Humans step back into the role of reviewing what happened and planning for the longer term.

In the military, international law draws the line clearly: a human must decide on targets. AI may spot and pre-select a target, such as a vehicle parked by a road, but a person must confirm that it is a legitimate military target before any strike.

The same logic holds in fire and rescue work. Software can analyze data and help set priorities, such as which people or assets to save first. Yet someone must always be able to halt the operation if the system is working from incomplete or incorrect data.

The Accountability Question

The moderator kept circling back to responsibility, and with good reason: who answers for it when an AI-enabled system fails?

One response was that failure should never be pinned on a single person. The whole chain deserves scrutiny, from hardware and software to processes, and the lessons should be turned into new routines.

The defense perspective held that the familiar rules still apply, whatever the technology. The commander who chooses to deploy a system owns its effects, must understand its limits and should not use it where the outcome is uncertain. If the AI itself errs, responsibility falls on the government, which must understand how the system works before certifying and using it. Industry’s task is to build reliable systems and to explain to governments how they work, where their uncertainties lie and how far their autonomy reaches. The final decision belongs to the government. Manufacturers, in this account, are not responsible for the end result; their duty is to make sure the government knows everything about the system.

The state’s perspective drew a further line. If a system fails because of a known limitation, responsibility rests with whoever chose to use it. If it fails because of poor quality, the supplier answers for it, just as with any piece of physical equipment.

The panel also touched on a point that rarely gets the attention it deserves: data quality. Even a well-built system is only as good as the data it runs on. In sudden emergencies, data is highly specific, needed quickly and hard to verify. State institutions sit on vast amounts of it but often do not know what they hold, while private companies struggle to reach it at all.

Consensus on Control, Uncertainty on Blame

By the end, the panel had settled one principle: AI can assist, but the human makes the final call. On responsibility, however, I left the room without a clear answer. The division of labor between governments, which must understand the systems, and manufacturers, which must explain them, sounds tidy on paper. In practice, it leaves too many questions open. The legal, private, state and other sectors still face real uncertainty over where accountability ends when an AI system fails, and I suspect that question will be waiting for us at the next edition of the summit.

Image taken by Tamar Shengelia at the Future Forces Forum

____________

By Tamar Shengelia, Lead for Cyber and Emerging Technologies, SSI