When AI Agents Hallucinate IDs

May 21, 20266 min read

A Production Bug That Changed One of My Rules for Tool Calls

A customer asked a booking assistant a normal question:

Do you have availability for a haircut tomorrow?

The assistant did the right thing at first. It asked for the preferred specialist and service, then listed the available options.

The customer answered:

I want Robert, Master Fade.

At that point the assistant had enough language-level information to continue. It knew the customer wanted Robert. It knew the requested service. It knew the date.

But it did not know Robert's internal database ID.

That distinction mattered.

The assistant called the availability tool with this payload:

{
  "date": "2026-05-21",
  "service": "Tuns Master Fade",
  "staff_id": "51396750-0000-0000-0000-000000000000"
}

The staff_id was fake. The model invented it.

The tool rejected the value, the agent retried, and the customer received a generic technical error. The booking API was not down. The database was not unavailable. The model had crossed a boundary it should never have been allowed to cross.

The bug was not simply:

The model hallucinated.

The real bug was:

An untrusted model was allowed to fill a trusted identifier.

Why This Is Worse Than a Bad Chat Message

We often use "hallucination" to describe a model saying something false in text. That is bad, but this case was different.

The model did not just say a wrong sentence. It placed a fabricated identifier into a tool call.

In a tool-using system, identifiers are not harmless strings. They are handles to real objects:

a staff member;
a service;
a booking;
a customer;
a tenant;
a payment;
a document.

If the wrong identifier gets far enough, the system may check the wrong calendar, book the wrong slot, cancel the wrong appointment, or touch the wrong customer record.

That makes identifiers authority-bearing data. They decide which object the system is about to act on.

And authority-bearing data should not come from model imagination.

The Confusing Part

The same booking flow had worked in other conversations.

That made the failure easy to misread. It was not that Robert had no ID in the database. Robert did have a real ID.

The difference was provenance.

In the successful flow, the assistant first called availability without a staff ID. The booking system returned real slots with real staff data. Later, when the customer selected a slot, the assistant reused data grounded in a tool result.

In the failing flow, the assistant had only a human label: "Robert".

It did not have Robert's trusted internal identifier in context. Instead of asking the system to resolve the label, it made one up.

That is the line that matters:

The problem is not whether an ID looks valid. The problem is who supplied it.

The Bad Contract

The old contract asked the model to provide a field like this:

{
  "date": "2026-05-21",
  "service": "Tuns Master Fade",
  "staff_id": "uuid"
}

But the prompt did not expose internal staff IDs to the model. It exposed names that humans understand.

So when the customer said "Robert", the model knew the label but not the identifier. A safe system should have made the next step deterministic:

pass the staff name to the backend;
omit the staff filter;
ask a resolver to match Robert in the tenant catalog;
or ask the customer to clarify.

Instead, the model was given a field it could fill but could not know.

Validation caught this particular value, but format checks are the wrong layer to rely on. A fabricated ID can still look real. By the time the call failed, the user experience had already become a technical failure.

The Fix

The fix was to remove model-owned internal IDs from the booking tool interface.

For availability, the model now sends business-facing labels:

{
  "date": "2026-05-21",
  "service_name": "Tuns Master Fade",
  "staff_name": "Robert"
}

The backend resolves those labels against the current tenant's catalog. It can decide whether Robert exists, whether the service exists, whether Robert can perform that service, and what slots are available.

For booking confirmation, the model still sends human-readable booking details, but the tool only accepts them if they match a recently returned availability slot for the same tenant and customer:

{
  "staff_name": "Robert",
  "service_name": "Tuns Master Fade",
  "start_time": "2026-05-21T14:00:00",
  "customer_name": "Flo"
}

The important part is not that the model typed 14:00. The important part is that deterministic code checks the requested staff, service, and start time against a slot the system recently offered.

If there is no match, the tool refuses to create the booking and asks the user to choose one of the offered times.

That turns a dangerous failure into a controlled one.

An even stricter version of this pattern is to return an opaque slot_token from availability and require that token for booking. The model can choose from tokens it has been given, but it cannot invent a valid token. That is a good direction for systems with higher risk or more complex workflows.

The Rule I Took From It

LLMs can propose intent.

They should not author identity.

A practical rule:

Any identifier that selects a business object must come from a trusted system, not from the model.

This applies to:

staff IDs;
service IDs;
booking IDs;
payment IDs;
account IDs;
tenant IDs;
permission IDs;
document IDs.

If a model supplies one of these, treat it as untrusted input unless you can prove it came from a trusted tool result, a scoped catalog, or deterministic application code.

Guardrails That Actually Help

Prompt instructions are still useful. I would still tell the model not to invent IDs, not to book before availability is checked, and not to confirm anything before the booking API succeeds.

But prompt instructions are not a boundary.

The boundary has to be in the contract and in the code.

1. Accept Labels, Resolve Server-Side

Let the model pass what the user actually said:

"Robert";
"Master Fade";
"tomorrow";
"around noon".

Then let backend code resolve those labels against the tenant catalog.

If the name is unknown, ask for clarification. If it is ambiguous, show options. Do not let the model guess the underlying ID.

2. Track Provenance

When a tool call includes data that selects a business object, the system should know where that data came from.

Was it returned by a previous tool?

Does it belong to the current tenant?

Does it match the current customer or conversation?

Is it still fresh enough to use?

If not, reject it before it reaches business logic.

3. Enforce Workflow State

Booking is not one tool call. It is a workflow:

collect date, service, and staff preference;
check availability;
present returned slots;
wait for the user to choose one;
create a booking only for a returned slot;
confirm only after the booking API succeeds.

The model should not be able to jump from "the user mentioned Robert" directly to "create a booking with arbitrary identifiers".

4. Ground the Assistant's Output

The assistant should not claim that:

a staff member exists unless the system found that staff member;
a service exists unless the system found that service;
a time is available unless availability returned it;
a booking is confirmed unless booking creation succeeded.

This is separate from tool input validation. You also need to validate what the assistant says back to the user.

5. Fail in Business Language

The safe failure is not:

Something technical went wrong.

The safe failure is closer to:

I cannot confirm that specialist and service from the available slots. Please choose one of the times I sent.

That message is less dramatic and much more useful.

What I Would Watch For in Other Agent Systems

This bug showed up in a salon booking flow, but the pattern is not specific to bookings.

The same mistake can appear in MCP tools, RAG systems, admin agents, CRM automations, payment workflows, support bots, and internal copilots.

Common smell:

The tool schema asks the model for an internal ID that the model did not receive from a trusted source.

Examples:

customer_id in a support action;
document_id in a RAG retrieval or deletion flow;
account_id in a billing tool;
tenant_id in a multi-tenant admin tool;
ticket_id in a helpdesk action;
payment_id in a refund tool.

The fix is rarely "write a longer prompt".

The fix is to move identity resolution and authorization back into deterministic code.

Takeaway

Useful agents need tools. Tools need boundaries.

The model can understand language, extract intent, and help the user move through a workflow. But when the system is about to act on a real business object, identity must come from the system.

Let the model say:

The user wants Robert for Master Fade tomorrow.

Do not let it decide:

This UUID is Robert.

That one boundary prevents a small hallucination from becoming a production action.

References

OWASP Top 10 for LLM Applications 2025: https://owasp.org/www-project-top-10-for-large-language-model-applications/
OpenAI Function Calling and Structured Outputs: https://help.openai.com/en/articles/8555517