Errors and retries

Every provider translates its native SDK exceptions into one portable error model, so failures are handled the same way on every cloud.

The error model#

All failures are unchecked subtypes of CloudException. The hierarchy is sealed: providers can't add types, so a switch over it is exhaustive and stays correct as providers are added.

Every CloudException also tells you where it came from:

try {
    store.get(key);
} catch (CloudException e) {
    e.isRetryable();          // true for ThrottledException and UnavailableException
    e.provider();             // Optional: "aws", "azure", "gcp", ...
    e.providerErrorCode();    // Optional: the native code, such as "NoSuchKey" or "BlobNotFound"
    e.getCause();             // the native SDK exception
}

Handling errors portably#

The same handler works on every cloud:

try (Blob blob = store.get(key)) {
    return Optional.of(blob.readAllBytes());
} catch (NotFoundException e) {
    return Optional.empty();                       // NoSuchKey, BlobNotFound or 404, whichever cloud
}

try {
    orders.update(key, update, WriteOptions.ifVersion(seen));
} catch (PreconditionFailedException e) {
    throw new ConflictException("Order changed since it was read");
}

With pattern matching over the sealed hierarchy:

String describe(CloudException e) {
    return switch (e) {
        case NotFoundException nf -> "missing";
        case PreconditionFailedException pf -> "conflict";
        case ThrottledException t -> "slow down" + t.retryAfter().map(d -> " for " + d).orElse("");
        case AccessDeniedException ad -> "forbidden";
        default -> "failed: " + e.getMessage();
    };
}

Retries#

The cloud SDKs already retry transient errors such as network failures, 5xx responses and throttling. On top of that, Halyx offers an opt-in, provider-neutral RetryInterceptor:

Halyx halyx = Halyx.builder()
        .register("aws", ProviderConfig.of("aws").region("eu-west-2"))
        .interceptor(new RetryInterceptor(new RetryPolicy(
                4,                          // attempts, including the first
                Duration.ofMillis(200),     // initial backoff
                Duration.ofSeconds(5),      // maximum backoff
                2.0)))                      // multiplier
        .build();

• It retries retryable errors (ThrottledException, UnavailableException) on idempotent operations only, with exponential backoff and full jitter. RetryPolicy.defaults() makes 3 attempts, starting at 100 ms and capped at 2 seconds.
• When the provider says how long to wait (Retry-After, or Azure's x-ms-retry-after-ms), it waits at least that long. If that is longer than maxBackoff, it rethrows at once instead of retrying early.
• Writes that aren't idempotent are never retried automatically. Where retrying a write is safe, such as rewriting a secret that's still being deleted, the application decides.

AWS clients use the SDK's adaptive retry mode by default (aws.retry-mode: adaptive, standard or legacy). It adds a client-side rate limiter that backs off while S3 returns 503 SlowDown. The limiter applies per provider connection; choose standard if a throttled bucket mustn't slow calls to others.

Interceptors#

Interceptors wrap every operation on object storage, NoSQL, messaging, caches, and parameters and secrets (SQL goes straight through JDBC). That makes them the place for cross-cutting behaviour such as retries, metrics, logging and auditing:

class SlowCallLogger implements Interceptor {
    @Override
    public <T> T intercept(OperationContext context, Invocation<T> invocation) {
        long start = System.nanoTime();
        try {
            return invocation.proceed();
        } finally {
            long millis = (System.nanoTime() - start) / 1_000_000;
            if (millis > 500) {
                log.warn("{} {} on {} took {} ms", context.service(), context.operation(), context.resource(), millis);
            }
        }
    }
}

Halyx halyx = Halyx.builder()
        .register("aws", ProviderConfig.of("aws"))
        .interceptor(new SlowCallLogger())
        .interceptor(new RetryInterceptor(RetryPolicy.defaults()))
        .build();

OperationContext describes the call: service() (storage, nosql, messaging, cache or parameters), operation() (such as put), provider(), resource() (the bucket, table, topic and so on) and idempotent(). Interceptors run in registration order, outermost first. Errors reaching an interceptor are already translated into CloudExceptions.

Asynchronous calls, such as BlobStore.async() and the reactive views, go through the same interceptors. interceptAsync defaults to running intercept on a virtual thread around the call, so every interceptor applies unchanged; override it to wrap the returned CompletableFuture instead, as RetryInterceptor (retrying on a timer) and ObservationInterceptor (observing until the future completes) do.

In Spring Boot, every Interceptor bean is applied in @Order order, and Micrometer observations are added for you. See Spring Boot.

Observations and tracing#

halyx-micrometer brings Micrometer to any application, with or without Spring. Add it first, so it observes retries and every other interceptor:

Halyx halyx = Halyx.builder()
        .register("aws", ProviderConfig.of("aws"))
        .interceptor(new ObservationInterceptor(observationRegistry))   // a halyx.operation timer and span per call
        .interceptor(new RetryInterceptor(RetryPolicy.defaults()))
        .build();

With Micrometer Tracing, HalyxTracing carries the trace from senders to listeners: publishers add it to message attributes, event publishers to the CloudEvents traceparent and tracestate extensions, stream producers to record headers, and function invokers to invocation headers (on Lambda, in the ClientContext). Wrapped listeners run each delivery in a consumer span that continues it:

HalyxTracing tracing = new HalyxTracing(tracer, propagator);
Publisher orders = tracing.publisher(context.publisher("orders"));
EventPublisher events = tracing.eventPublisher(context.eventPublisher("orders"));
FunctionInvoker resize = tracing.functionInvoker(context.functionInvoker("resize-image"));
subscription.listen(tracing.listener("billing", message -> bill(message)), ConsumerOptions.DEFAULTS);
consumer.listen(tracing.recordListener("ledger", record -> post(record)), ListenerOptions.DEFAULTS);

The Spring Boot starter applies all of these to its beans and @HalyxListener methods.