Once used mostly as just support tools, AI systems are now increasingly taking on more and more critical functions and responsibilities. How should governance and operational controls evolve in this new era of AI dependency to manage risk? Five experts discussed this in the panel titled “Agentic capital markets: when AI becomes the counterparty” at Sibos 2026.
“What we are already seeing today is that AI has been introduced, in many instances with humans still in the loop. Then, we need to be ready for the inevitable, which is autonomous AI,” says Gloria Lio, managing director, head of Enterprise Services at DTCC.
“Autonomy and trust come together,” says Armando Benitez, chief data and analytics officer (CDAO) and head of AI at BMO Capital Markets. But trust first must be earned. For AI systems, that should be achieved by “defining the right processes, and the right framework around it.”
The starting point, he believes, should be for AI to be tested in processes that are controllable and reversible, and where impact can be measured.
“Controllable and reversible” are the important keywords here. Benitez illustrated with some quick math. Imagining that there are 10,000 decisions to be made, and just one of the many processes involved is at 99.9% accuracy, then “eventually after a sequence of many of those”, there would be significant deviation from the goal. Perfect every single process, and that’s when “you can stitch many of these decisions to accomplish a particular task without losing sight of the final goal”.
Lio picked up Benitez’s point about the criticality of controllable and reversible processes. Controls need to be in place to identify when autonomous AI is “deviating from its intended authority” so that it’s possible to “stop and reverse the work that we’ve given it authority to do”.
This control has to be supported by appropriate governance. Akber Jaffer, CEO of Smartstream technologies says, “I do think there has to be a recognition that the world is moving at a much faster pace… The speed at which governance has to move now, it’s very different, because things are changing on a monthly basis.” He believes that there should be an “agility to governance”, where it is viewed more as an “enabling function” for introducing technology.
Limit autonomy
Despite its name, autonomous AI cannot be truly autonomous – at least not yet. “The collaboration of human and AI is clear. The accountability is on the human and the institution… We need to govern the decisions and the processes,” says Benitez. In the words of Jaffer, there must always be a human “on the hook”.
The level of human intervention can be decided based on risk tiering. Manoj Bohra, executive vice president and chief data and AI officer at State Street shared an example. Automating a back-end, manual process of low risk would require a very different level of governance than “an irreversible process with meaningful impact to customers and the balance sheet”.
When coordinating human and machine in a single process, Benitez advises that firms should always “start with the low impact, low risk, things that you understand”. Controls should “work at the speed of the machines, not at the speed of humans” and be “true controls” that are not just about ticking boxes.
“In the past, we learnt how to manage humans and we’re doing an okay job there. In this next stage of our society, we’re learning how to work with digital workers. Some of the lessons that we learnt from managing humans are going to translate, but it is a different dynamic,” he says.
Initial inertia
Change is inevitable, but Jaffer pointed out that there might be some inertia at the beginning. “Some of these processes have been set up for a long time… We recognise that we’re coming from a place with a lot of legacy. The question is, how do we phase that change in? Technology is just the enabler. It’s really a business change and a culture change.”
He likened autonomous AI to a new graduate that has just been recruited into the organisation. “We wouldn’t give a new graduate the decision-making power over, say, a balance sheet change of more than maybe $50,000, but we’d give the group CFO the ability to make a balance sheet decision of $10 or $20 million… when we talk about using technology to change processes, in many ways, it’s about tolerance levels.”
“How do you programme that in so that the agent is operating within the tolerance levels that you are comfortable with as an enterprise and the risk that you want to take? Ultimately it boils down to business change.”
Lio agrees, ”There has to be an assessment of the operating models that exist today and whether or not, with the introduction of autonomous AI, they need to change in the future as well. There’s no doubt that if you continue to only replace the existing processes and operating models with AI, you might get a potentially faster, but not necessarily a fundamental shift in operations.”
Sibos 2026 plays out in Miami from 28 September to 1 October. We are there, view our coverage here.