Retries & Timeouts

Use a retry policy when an action may succeed on another attempt. Add a timeout when a slow or stuck call should stop holding up the workflow. Both settings apply to an individual action call.

Default behavior

Without a retry policy or timeout:

• A bare await some_action(...) runs the action once. If it raises, the failure propagates out of the workflow. No automatic retries.
• There is no default timeout. An action runs until it completes or its worker dies. You opt into a deadline per call with timeout=.

When a failure exhausts its retries (or has none), the workflow fails and the await workflow.run(...) call raises a WorkflowFailedError carrying the failure - unless a try/except in the workflow body catches the exception first. See Exceptions.

Configuring with run_action

To override the defaults, wrap the action call in self.run_action(...) and supply a RetryPolicy:

from waymark import RetryPolicy, Workflow, workflow

@workflow
class FlakyApiWorkflow(Workflow):
    async def run(self, payload: dict) -> dict:
        return await self.run_action(
            call_third_party_api(payload),
            retry=RetryPolicy(attempts=5),
        )

RetryPolicy(attempts=N) is total executions - five attempts means up to four retries after the initial try. RetryPolicy() with no attempts defaults to 100 retries (101 attempts in all). Set attempts explicitly when you need a smaller budget.

Filtering by exception

You usually don't want to retry on every exception - a ValueError from a bad input is permanent, but a dropped connection is worth another shot. Pass exception_types to scope retries to specific exceptions, by name, since the workflow body doesn't import them.

Each name is matched against the raised exception's class and its base classes - type(exc).__name__ and the names along its MRO, without modules - the way an except clause matches: "ConnectionError" retries a ConnectionResetError or a ConnectionRefusedError, since both derive from it, and "OSError" covers all three. A library's exceptions go by their own class names and bases - httpx's failed connections are ConnectError, deriving from TransportError, for example. Only names are compared, so two unrelated classes both called ValidationError are the same to a filter, and a module path in a name is rejected at compile time.

await self.run_action(
    call_third_party_api(payload),
    retry=RetryPolicy(
        attempts=5,
        exception_types=["ConnectionError", "TimeoutError"],
    ),
)

If the action raises something not in the list, the policy doesn't match - the failure propagates immediately. Each run_action call takes exactly one RetryPolicy; to treat different exceptions differently, list them all in exception_types or split the work into separate actions with their own policies.

With no exception_types, the policy retries on Exception: every ordinary exception the action raises. An explicit empty list retries nothing. "Exception" listed next to other names still covers everything deriving from it, so ["Exception", "ActionTimeout"] retries ordinary exceptions and timeouts both.

The runtime's own exceptions sit outside Exception. ActionTimeout (see Timeouts) and ActionExecutionLost, raised when the worker process running the call dies or is recycled, derive from BaseException directly, because the attempt may still be running or may have run to completion, so a retry can end up running it twice. A policy on Exception does not retry them; list them by name, or "BaseException", to opt in. ActionExecutionNotStarted, raised when a lost worker process provably never started the call, derives from Exception and is retried like any other. The three are spelled out as classes in waymark.vm_exceptions for except clauses; see Exceptions. A node dying is a different case: the calls in flight on it are redelivered by the node that resumes the workflow, as Production Deployment explains.

Besides a policy, the workflow can catch ActionTimeout, ActionExecutionLost or ActionExecutionNotStarted in a try/except around the call and retry it with code of its own.

An action itself should raise only Exception subclasses. The worker reports only those back, so a BaseException the action raises never reaches the workflow, and neither a policy nor an except sees it. The call fails some other way instead: a SystemExit or KeyboardInterrupt stops the worker process, so the call fails with ActionExecutionLost; any other BaseException leaves the call waiting for its timeout, which fails it with ActionTimeout - and with no timeout, until its worker process dies or is recycled, which fails it with ActionExecutionLost.

Timeouts

A per-call timeout caps how long an individual action attempt may take. The clock starts once the call has been issued, before any worker picks it up, so time spent waiting for a worker counts; a sleep of that length is raced against the attempt, and when it wins the attempt fails with an ActionTimeout exception. The timed-out attempt itself is not cancelled: it may still be running on its worker, which is why retrying timeouts is an explicit opt-in (below). The value is a number of seconds, fractions included, or a timedelta built from seconds, minutes, hours or days keywords; it must be more than zero.

from datetime import timedelta

await self.run_action(
    slow_summary(report_id),
    retry=RetryPolicy(attempts=3),
    timeout=timedelta(minutes=2),
)

await self.run_action(
    slow_summary(report_id),
    retry=RetryPolicy(attempts=3, exception_types=["ActionTimeout"]),
    timeout=timedelta(minutes=2),
)

Because a timeout doesn't cancel the original attempt, this policy can create overlapping attempts. The action must tolerate being called more than once.

How retries and timeouts combine

The timeout is raced against each attempt separately, and backoff_seconds is slept before each retry, so a retried action waits the backoff and then runs again with a fresh timeout. Do not add your own sleep inside the action for the backoff: it would add to the wait and count against the timeout. backoff_seconds is slept as given, fractions included; zero means no backoff.

Timeouts and backoff are durable sleeps, woken by the node's sleep poller, so each one can wake up to one poll interval late - WAYMARK_SLEEP_POLL_INTERVAL_MS, 250 ms by default. Lower it if that matters for short timeouts or backoffs.

Every RetryPolicy field and every timeout= must be a literal in the workflow body, a module-level constant, or the whole policy or the timeout a self. attribute assigned a literal once, at the top level of __init__. A constant compiles as the value it holds when the workflow compiles, so AGENT_ATTEMPTS = 3 shared across call sites, or a DEFAULT_RETRY = RetryPolicy(...) passed as retry=DEFAULT_RETRY, both work. The compiler rejects anything it cannot read at compile time with UnsupportedPatternError. The Control Flow guide names the rejected forms.

API reference

For the precise signatures and field types, see the Python RetryPolicy reference.