Agentic AI and Procurement (Part 4): What It Takes to Make It Work

This raises new questions about architecture, governance and change management that go beyond what most digital procurement initiatives have had to tackle to date. The Hackett Group’s recent research and report underscores this point: procurement leaders rate Gen AI (a core component of agentic AI systems) among their top transformation initiatives, yet they also describe it as the lowest maturity area across all key initiatives. This shows that the gap between intent and readiness is real.

Start with the data (and the gaps)

The Hackett Group’s study highlights that data quality is the number one concern for procurement leaders when adopting Gen AI, making data readiness a foundational issue for agentic AI.

Data quality is critical because agents rely on real-time access to contextual information: demand signals, supplier data, contract terms, risk indicators and more. However, in most organizations, this information is often fragmented across various systems, including ERPs, sourcing platforms, contract repositories and spreadsheets. So, data must be accessible, normalized and structured for action, otherwise even the most advanced agents will underperform or misfire. To get this clean data, you have to start with the people and processes that surround it.

Data for agentic AI also requires persistence, as agents need to maintain context across decision chains and steps and refine future actions based on feedback. Without structured memory and longitudinal data (information collected or measured over time to track changes and trends), agents revert to acting like smart, rule-based or scripted workflows and will not be able to evolve with the business.

Agent-ready infrastructure

Most enterprise APIs were designed for predictable, human-driven interactions: one request, one response. Agentic systems challenge and change that. They make intent-based decisions, carry memory across steps and orchestrate tools dynamically in response to real-time conditions. Supporting this requires not just API availability but APIs that can handle persistent state, asynchronous decisions and tool chaining.

An orchestration layer must be present for agents to determine which function to invoke next rather than simply executing a predefined flow. To support that, IT teams will probably need to upgrade existing interfaces or introduce middleware that can handle orchestration and maintain memory between systems. Without this adaptation, agents will constantly trip over fragmented and siloed workflows and systems.

Security, privacy and policy enforcement

Autonomous systems cannot be allowed to act without boundaries. Agents need clearly defined roles, governed by access controls and escalation logic. They must be able to distinguish between sensitive and routine data, trigger approvals and log actions for audit purposes. Boundaries are not optional, especially for agents operating in regulated environments or industries. Role-based access, tokenized data handling and dynamic consent management must be built in from the start.

Gen AI-specific risks

Agentic AI often builds on Gen AI components. Many so-called ‘agentic’ systems still operate on fragile Gen AI foundations. They may appear autonomous but lack structured reasoning or secure tool invocation, which makes their autonomy more dangerous than useful. 

Risk mitigation needs to start not just with UX guardrails but at the architectural level by constraining what the LLM can do and instrumenting the agent framework to supervise every step.

That adds a second layer of risk. Gen AI systems can fabricate content (so-called ‘hallucinations’), interpolate missing data or confidently present uncertain outputs. In procurement, where contractual language, pricing decisions and risk alerts carry financial consequences, these flaws are operational risks. Safeguards, such as retrieval-augmented generation, confidence scoring and fallback rules, must be in place before granting agents autonomy.

Governance that matches the stakes

The Hackett Group’s research also reveals that 53% of procurement leaders are concerned about unrealistic AI benefit expectations, while others cite IP leakage, privacy and lack of experience as significant risks (Figure 1). This underscores the need for robust governance.

As agents assume more responsibility, procurement teams need new mechanisms to govern their behavior. That includes performance monitoring, exception tracking and policy conformance checks. But governance also needs to account for interaction between agents (especially in multi-agent environments). 

Multi-agent systems increase complexity. Agents may interact, compete or even inadvertently conflict when goals overlap or when environmental signals trigger ambiguous responses. Overlap, drift or unintended coordination can create new failure modes. Teams must be able to trace what the agent did, why it did it and how the system responded. Governance models must be able to answer: What happened? Why? And can it happen again?

Change at the organizational level

As AI systems evolve, procurement professionals will transition away from their execution roles toward strategy, policy and governance roles. This is already reflected in the operating models of leading organizations.

Deploying agentic AI does not just reshape systems; it shifts how procurement teams operate. The function moves from managing transactions to designing agent behaviors, supervising exceptions and reviewing outcomes. That changes job descriptions, workflows and stakeholder expectations.

Skills like prompt engineering, workflow modeling and exception triage will quickly become part of the modern procurement toolkit. Some category managers will spend less time sourcing and more time curating, training and improving the agents that do such tasks. Others will shift into roles focused on governance, strategy or supplier development.

This change requires training, communication and a roadmap that shows teams where they are headed and how their roles fit into the new model.

Readiness is the real barrier

Agentic systems are advancing quickly. But the biggest hurdle is not technical; it is organizational. Data readiness, architectural flexibility, risk tolerance and cultural alignment determine whether these systems succeed or stall. 

Teams that approach implementation with a narrow, automation-focused mindset will miss the opportunity. Those that embrace agentic capabilities as part of a larger transformation will shape the next era of procurement.