Idempotent Tool Calls: Safe Retries for AI Agents

Suppose an agent is responsible for one small piece of billing: when an invoice is ready, it calls a send_invoice_email tool. One afternoon the mail API is slow, the call times out, and the agent does the reasonable thing and tries again. The customer now has two copies of the same invoice in their inbox, and nothing in the agent’s own logs looks wrong.

This post is about why that happens and how to prevent it. We will reproduce the failure with a real agent execution and look at the trace, then fix it with two mechanisms that work together: an idempotency key, which lets the receiving API recognize a repeated request, and a recorded result, which stops the agent from re-running work that already finished. Along the way we will look at actual execution traces from Agentspan, because this is one of those problems that is much easier to understand when you can see each attempt laid out as a step.

Why retries duplicate side effects in AI agents

Start with the uncomfortable fact underneath all of this: when a tool call times out, the caller has no way to know what actually happened. Maybe the request never reached the mail API. Maybe it arrived and the email went out, but the response got lost on the way back. From the agent’s side those two situations are indistinguishable, and only one of them is safe to retry.

Since giving up on every timeout would make the agent useless, every practical system chooses to retry. That choice has a name: delivery is at least once. The call is guaranteed to happen one or more times, and the “or more” is exactly where duplicates come from. Retries are not a bug here. They are the correct behavior, colliding with a tool whose side effect happens on every execution.

Here is that collision as a real execution trace. The agent below calls send_invoice_email, gets back a timeout, and retries with the same arguments, exactly as instructed. Reading the trace top to bottom: the first tool call returns status: timeout, a second turn begins, and the second tool call returns status: sent with a message id.

The execution completes with a clean summary and a message id, and every node in the trace is green. What the trace cannot show is the mail provider’s outbox, which now contains two messages: the first send actually worked, and only its response was lost. This is the defining trait of the duplicate-side-effect problem — from where the agent sits, everything looks correct.

The timeout is only the first of two windows where this can happen. The second one does not involve the network at all: the tool call succeeds, and then the process dies after the side effect but before the result is recorded anywhere. When the run is restarted, there is no record that the step ever ran, so any sane recovery logic runs it again. We wrote about the losing-work half of this problem in why AI agents lose work when they crash; this is the same failure with the sign flipped. Instead of work disappearing, work happens twice.

Retry loops, crash recovery, and a human rerunning a failed job are all, mechanically, the same thing: executing a tool call whose earlier execution may have already taken effect. So instead of trying to prevent re-execution — which at-least-once delivery makes impossible — the fix is to make re-execution harmless.

What makes a tool call idempotent

An operation is idempotent when doing it twice leaves the world in the same state as doing it once. An elevator call button is idempotent: pressing it five times gets you one elevator. Some tool operations already work this way. Setting a record’s status to paid, upserting a row by its order id, or PUT-ing a document to a fixed URL can all be repeated freely, because they describe a target state rather than an action, and reapplying the same state changes nothing.

The tools that get agents into trouble are the ones that describe actions: send an email, create a ticket, charge a card. Run those twice and the action happens twice, because the receiving system has no way to tell a retry from a genuinely new request. Both arrive as POST /v1/send with the same fields.

That framing points directly at the fix. If the problem is that the second request looks new, give both requests a shared identity that says “these are the same attempt at the same thing.”

How idempotency keys deduplicate retries

An idempotency key is a caller-chosen identifier for the intent of a request rather than for the individual attempt. The first try and the retry must carry the same key, and that requirement tells you what the key can be made from. It cannot include a timestamp, a retry counter, or anything random, because those change between attempts. It should be derived from what makes the operation itself unique:

import hashlib, json

def idempotency_key(execution_id: str, step_id: str, args: dict) -> str:
    canonical = json.dumps(args, sort_keys=True)   # same args, same string, every time
    raw = f"{execution_id}:{step_id}:{canonical}"
    return "ik_" + hashlib.sha256(raw.encode()).hexdigest()[:8]

Each part has a job. The execution id scopes the key to this run, so that next month’s invoice email is not mistaken for a repeat of this one. The step id separates two sends inside the same run. And the canonicalized arguments (sorted keys, stable formatting) make sure the same logical request always hashes to the same key, while a genuinely different request never does.

The receiving API’s half of the contract is simple, and it is the same one Stripe documents for its API: the first time a key arrives, execute the request and store the response under that key; every time the same key arrives again, return the stored response and execute nothing.

We gave our demo tool exactly that behavior and ran the same scenario again — same timeout on the first call, same retry. The difference shows up when you open the retried tool call in the trace and look at its output:

Read the output carefully, because every field is doing work. The status is sent and the agent is satisfied. The message_id is the first attempt’s id — no new message was created. The idempotency_key is the same key both attempts carried. And deduplicated: true is the mail API saying, in effect, “I have seen this request before, here is what happened the first time.” One email in the outbox, and the retry cost nothing.

Recording tool results so restarts don’t repeat work

Keys handle the retry-after-timeout case, but think back to the second window: the process dies after the tool succeeded and before anything was written down. If the memory of “step 4 is done” lived only inside that process, then after a restart the whole step runs again. The key store on the API side might still catch the repeat, but you would rather not depend on an external provider’s retention policy for your own correctness.

The complementary mechanism is to record each step’s result somewhere durable the moment it completes, and to check that record before executing anything. If the record says the step already ran, you return the recorded result instead of executing the tool again. The record has different names in different systems — a journal, a write-ahead log, an execution history — but the discipline is always the same: a step is not finished when the tool returns; it is finished when the result is durably recorded.

To show this one honestly, we ran a two-step agent — send the invoice email, then record it in a CRM — and killed the process between the steps, after the email had gone out but with the CRM write still pending. Then we reconnected to the same execution and let it finish. Here is the completed trace:

Two details in this screenshot carry the argument. First, the 53-second duration on an execution whose work takes about two: most of that time is the gap where no process was running at all, and the execution simply waited. Second, send_invoice_email appears exactly once. The email step’s result was recorded on the server before the crash, so when the run resumed, that step was replayed from the record rather than re-executed — and this time the tool had deliberately no dedupe of its own, so a re-execution would have shown up as a second message. The outbox still holds one email, and the CRM record points at the same message id.

It is worth being precise about how the two mechanisms divide the work, because neither is sufficient alone. The idempotency key protects the window between request and response, when the caller cannot know what happened. The recorded result protects the window between response and record, when the caller knows but hasn’t durably written it down yet. Together they make the entire step safe to run again from any point of failure — which is the property at-least-once delivery demands.

Best practices for idempotent agent tools

The demo compresses a handful of rules that generalize well beyond invoice emails:

• Key the intent, not the attempt. Execution id, step id, canonical arguments. Never time, randomness, or a retry counter — those differ between attempts of the same operation.
• Canonicalize before hashing. json.dumps(args, sort_keys=True) exists because two dicts with the same content can serialize differently, and a key that shifts with formatting is no key at all.
• Prefer server-side deduplication when you control the API. Store key → response, replay on repeat, and keep keys at least as long as your longest retry horizon.
• Lean on natural idempotency when you don’t control the API. Upserts, conditional writes such as If-Match, and create-with-client-supplied-id endpoints make repeats harmless without any key infrastructure.
• Record results before advancing. The step is done when its result is written down, not when the function returns.
• Separate reads from writes. Retrying a read is always safe, so keeping side-effecting tools distinct shrinks the surface that needs this care at all.
• Let unsafe tools fail loudly. If a call genuinely cannot be made safe to repeat, mark it so the agent loop stops and escalates instead of retrying on faith.

Exactly-once execution vs. at-least-once delivery

You will sometimes see systems advertise exactly-once delivery, and it is worth knowing what that can and cannot mean. No transport can deliver a request exactly once to a receiver that is allowed to crash — that is a consequence of the same ambiguity we started with, not an engineering shortfall. What well-built systems actually provide is at-least-once delivery combined with idempotent effects, which is indistinguishable from exactly-once when observed from outside. That combination is the contract durable execution is built on: the runtime guarantees every step runs at least once and gets recorded, and idempotent tools guarantee that “at least once” reads as “once.”

What a durable runtime handles for you

The traces above came from a durable runtime, and it is doing one half of this work automatically: every tool call’s arguments and result are persisted as steps in the execution history, which is why the crashed run could resume without re-sending, the mechanism described in crash recovery. If you run agents on Agentspan, that recording discipline comes with the platform, and start() even accepts an idempotency key for the run itself, so a double-submitted execution deduplicates the same way a double-submitted request does.

What no runtime can do is make your tools’ side effects idempotent for you. The moment a call leaves the runtime’s boundary for a mail API, a payment processor, or a database, the key discipline in this post is your half of the contract. The tool definition docs cover where to attach it.

Common questions

Are AI agent tool calls idempotent by default? No. A tool call is a function invocation. Unless the function is a pure read, or writes to a fixed target state, each invocation performs its side effect again. Retry safety has to be designed in, which is what the key and the recorded result are for.

What should go into an idempotency key? A stable identity for the logical operation: the execution id, the step id, and a canonical hash of the arguments. The test is simple — the key must come out identical across retries of the same step, and different for any genuinely different operation.

How long should an API keep idempotency keys? Longer than the longest gap across which a retry of the same operation could plausibly arrive. Stripe keeps keys for 24 hours. Agent runs that pause for human approval can stay open for days, so if your agents wait on people, size the retention window to match.

Run the example yourself

The traces in this post came from live runs against a local Agentspan server, but the mechanics don’t need a server to understand. The script below simulates the mail API, the lost response, the crash, and both fixes in about a hundred lines of standard-library Python, with the service state kept in a local file so crashes and reruns behave like calls to something external. Save it as email_tool.py in an empty directory — it needs Python 3.8 or newer and nothing else:

import hashlib, json, os, sys

STATE = os.path.join(os.path.dirname(__file__), "state.json")

def load():
    return json.load(open(STATE)) if os.path.exists(STATE) else {"outbox": [], "keys": {}, "journal": {}}

def save(s):
    json.dump(s, open(STATE, "w"), indent=1)

def msg_id(payload, attempt):
    return "msg_" + hashlib.sha256(f"{payload}{attempt}".encode()).hexdigest()[:6]

ARGS = {"to": "jordan@acme.dev", "subject": "Your June invoice", "body": "…"}
EXECUTION_ID = "exec_01H8"
STEP = "step-4"

def ikey():
    canon = json.dumps(ARGS, sort_keys=True)
    return "ik_" + hashlib.sha256(f"{EXECUTION_ID}:{STEP}:{canon}".encode()).hexdigest()[:8]

def mail_api_send(state, payload, attempt, idem_key=None):
    """The 'server side'. Delivers mail. With a key, dedupes."""
    if idem_key and idem_key in state["keys"]:
        prior = state["keys"][idem_key]
        print(f"[mail-api]  duplicate Idempotency-Key {idem_key} — returning original result, no send")
        return prior, True
    mid = msg_id(payload, attempt)
    state["outbox"].append({"id": mid, "to": ARGS["to"]})
    print(f"[mail-api]  delivered id={mid} to {ARGS['to']}")
    if idem_key:
        state["keys"][idem_key] = mid
    save(state)
    return mid, False

def outbox(state):
    n = len(state["outbox"])
    flag = "   ← duplicate send" if n > 1 else ""
    print(f"[mail-api]  outbox for {ARGS['to']}: {n} message{'s' if n != 1 else ''}{flag}")

def banner():
    print(f"[agent]     {STEP}: send_email(to={ARGS['to']}, subject=\"{ARGS['subject']}\")")

def mode_naive(state):
    banner()
    print("[tool]      POST /v1/send")
    mail_api_send(state, "invoice-jun", attempt=1)
    print("[tool]      ✖ network timeout after 10s — response lost in transit")
    print("[agent]     tool call failed, retrying (attempt 2/3)")
    print("[tool]      POST /v1/send")
    mid, _ = mail_api_send(state, "invoice-jun", attempt=2)
    print(f"[tool]      ✓ 200 OK id={mid}")
    print(f"[agent]     {STEP} recorded as complete")
    print()
    outbox(state)

def mode_crash(state):
    banner()
    print("[tool]      POST /v1/send")
    mail_api_send(state, "invoice-jun", attempt=1)
    print("[agent]     ✖ killed (exit 137) — delivery happened, result never recorded")
    sys.stdout.flush()
    os._exit(137)

def mode_rerun(state):
    print(f"[agent]     restarted — no record that {STEP} ran, executing it again")
    banner()
    print("[tool]      POST /v1/send")
    mid, _ = mail_api_send(state, "invoice-jun", attempt=2)
    print(f"[tool]      ✓ 200 OK id={mid}")
    print()
    outbox(state)

def mode_safe(state):
    key = ikey()
    if STEP in state["journal"]:
        rec = state["journal"][STEP]
        print(f"[journal]   {STEP} already has a recorded result ({rec}) — skipping tool call")
        print()
        outbox(state)
        return
    banner()
    print(f"[tool]      POST /v1/send  Idempotency-Key: {key}")
    mail_api_send(state, "invoice-jun", attempt=1, idem_key=key)
    print("[tool]      ✖ network timeout after 10s — response lost in transit")
    print("[agent]     tool call failed, retrying (attempt 2/3)")
    print(f"[tool]      POST /v1/send  Idempotency-Key: {key}")
    mid, deduped = mail_api_send(state, "invoice-jun", attempt=2, idem_key=key)
    print(f"[tool]      ✓ 200 OK id={mid}")
    state["journal"][STEP] = mid
    save(state)
    print(f"[journal]   recorded {STEP} = {mid}")
    print()
    outbox(state)

if __name__ == "__main__":
    mode = sys.argv[1]
    if mode == "reset" or not os.path.exists(STATE):
        save({"outbox": [], "keys": {}, "journal": {}})
        if mode == "reset":
            sys.exit(0)
    state = load()
    {"naive": mode_naive, "crash": mode_crash, "rerun": mode_rerun, "safe": mode_safe}[mode](state)

The script takes one argument, the scenario to run, and reset wipes the fake mail service between experiments. A full session that reproduces everything in this post looks like this:

python email_tool.py reset
python email_tool.py naive     # window 1: response lost, client retries → duplicate

python email_tool.py reset
python email_tool.py crash     # window 2: dies after the send, before recording
python email_tool.py rerun     # the restart re-executes the step → duplicate

python email_tool.py reset
python email_tool.py safe      # same timeout and retry, now with a key → one send
python email_tool.py safe      # a full rerun after that — the journal skips the step

Each stage prints the mail API’s view at the end, so you can check your output against what should happen. The first experiment ends with the failure this whole post is about:

[mail-api]  outbox for jordan@acme.dev: 2 messages   ← duplicate send

The crash experiment shows you both sides of the second window — crash stops at ✖ killed (exit 137) — delivery happened, result never recorded, and rerun opens with restarted — no record that step-4 ran, executing it again before producing the same two-message outbox. Then the safe runs show each defense doing its specific job. During the retry you will see the server-side key absorb the repeat:

[mail-api]  duplicate Idempotency-Key ik_a94a0b1d — returning original result, no send

and on the second safe invocation, which stands in for a full process restart, the journal answers before the tool is ever called:

[journal]   step-4 already has a recorded result (msg_41bcc7) — skipping tool call
[mail-api]  outbox for jordan@acme.dev: 1 message

However many times you run safe, the outbox stays at one message. That is the whole contract in miniature: retries and reruns keep happening, and the customer keeps receiving exactly one email. From there, two good next steps: try deleting the journal entry from state.json by hand and rerunning safe to convince yourself the key alone still holds the line, and then wrap a tool of your own — the ~15 lines of mode_safe are the pattern to copy.

If you want to go deeper, the system design guide places retry policy in the context of a full agent architecture, and the idempotency glossary entry is the short reference version of everything above.