As a parent of a college sophomore going into computer science, and as a programmer with over 30 years of experience, I understand both the threat and promise Artificial Intelligence (AI) poses for all sorts of people and businesses. AI is already changing how businesses operate. Some view AI as a replacement for human workers while others dismiss AI as an over-hyped technology. However, the reality is more nuanced. AI and agentic technologies can improve an organization’s performance, but only when they are used to augment experienced workers rather than replace them.
In June 2025, Marina Mancoridis and colleagues published a paper entitled “Potemkin Understanding in Large Language Models”, showing that many leading LLMs can explain concepts correctly while still applying them inconsistently in ways humans typically would not. The implication is that an AI agent may appear to understand your business rules because it performs well on demonstrations or benchmarks yet still possesses important gaps in its conceptual understanding that only become apparent in real-world use.
While LLMs have improved significantly since 2025, many of these models are still being used, and they can have failure rates of 40-50%, depending on the task. Overall, the 2025 metrics show that LLMs are reliable 94% of the time on specific benchmarks. To the untrained eye, that sounds fairly reasonable, but it is actually a red flag. In workflows where multiple AI agents work in tandem, this means that the overall reliability is disastrously low.
Consider a situation where we have 4 agents each dependent on the previous one’s output. This isn’t an uncommon situation where we have a multi-agent workflow.
The overall reliability of the process is only 78% as each agent is only 94% reliable, and those probabilities compound across the workflow. For most of us in the business world, we need overall performance to be 99.9% or better to call it reliable.
LLMs also hallucinate, confidently generating information that is false. There have been numerous reports of lawyers facing sanctions after AI-generated filings cited cases that never existed. LLMs are better at inventing information than they are at just about any other task, and they rarely even know they are hallucinating.
None of this means organizations should avoid AI. It means they should approach implementation with the same rigor they would apply to any mission-critical business system. Successful AI adoption isn’t just about choosing the right model. It’s about putting the right people, processes, and governance around it.
So how can organizations realize the benefits of AI and agentic workflows without exposing themselves to unnecessary risk? I wish I could say the answer is simple, but it isn’t. Here are some guidelines I’ve come up with from talking to experts and from my own experience.
1. Always have an experienced human in the loop on critical decision making.
You will need to empower your best employees with the responsibility to review decisions made by the AI agent and track the exceptions and times when it needs to be corrected. It sounds simple, but an experienced human will be able to spot irregularities more quickly than any AI could ever do. In all probability, something won’t “feel right” about the decision when they see it. They may ask themselves, “Why have we ordered 100 times more of product A than we usually do?” or “Why are we routing this shipment to Miami through Minot, North Dakota?”
The human in the loop should review rule exceptions and critical decisions before they are executed. These decisions should be captured through audit logs, along with the input provided to the agent and the resulting outputs. This creates a feedback loop that continuously improves prompts, instructions, and overall agent performance.
2. Establish a reviewing team that meets early and frequently when setting up an AI agent.
Initially, I’d recommend that you have a team of subject matter experts review the base prompt instructions for your agent, gather feedback from the primary human reviewer on what the agent is missing in its instructions and what should be removed. You can lengthen the time between reviews later in the development cycle as the agent is tuned correctly.
3. Create a testing sandbox for every AI agent.
You need to have a playground or a sandbox where you can ask it why it made a choice to categorize failed decisions, to take a failed action, or to not escalate a critical situation to a human. Capture example data from your human in the loop and make sure your agent can explain its “thinking” when encountering the data. Ask questions such as “Why wasn’t this escalated to our customer support?” or “Why did you categorize this order as critical when it wasn’t?” In the sandbox, you’ll need to refine your prompts by adding guardrails that guide the agent toward your organization’s desired behaviors and decision-making rules.
4. Maximize human interactions where you can.
Business is always based on trust between people and organizations. It can take years to establish a good relationship and only days to destroy one. One client we had was having to monitor WhatsApp inquiries about shipments and inventory in a variety of languages from their clients who didn’t have EDI enablement.
This kind of thing is perfect for an AI agent as they can translate to and from a variety of languages with ease and can look up information in natural language with Retrieval Augmented Generation (RAG) by querying databases or web services, all without the expense of implementing EDI. This leaves valuable staff the ability to focus on the relationship with the customer. AI should handle repetitive, information-driven interactions so your people can focus on the conversations that build trust, solve problems, and strengthen customer relationships.
5. Establish AI governance across the organization.
You will need to establish a cross-functional AI governance team. They need to establish how to prompt the agents correctly, how to measure success or failure with proper uniform metrics, and how to fall back to human decision making when these systems fail to work properly. AI agents need to be treated like any other production system and need to be managed with the same discipline, oversight, and continuous improvement as any other production system.
AI will undoubtedly be changing how all of us do business over the next few years. Organizations that are successful won’t be those that simply cut humans. Successful organizations will be the ones that augment human expertise with AI capabilities. Technology will invariably change rapidly, but good judgement, experience, and human relationships will remain the true competitive advantage for any business.
At VCO Systems, we believe successful AI adoption begins with operational expertise, thoughtful governance, and clearly defined business outcomes. By combining deep supply chain knowledge with modern AI technologies, we help organizations implement AI responsibly, improve decision-making, and create measurable business value while keeping experienced people at the center of critical operations.