Testing on real clouds

Every Halyx driver passes its conformance kit against emulators on every build (see Design, build and test). Emulators are fast and free, but they don't model everything that matters in production: IAM and Entra ID logins, eventual consistency, asynchronous deletes, throttling, private certificate authorities, or exactly how a service rejects a request. The real-cloud runs close that gap by running the same conformance kits against real AWS, Google Cloud and Azure services.

Results at a glance#

These are the full regression before 1.1.0, run on 2026-11-01 at 40d4101. Tier 1 now includes function invocation (Lambda, Cloud Run functions and Azure Functions), and on Azure the key rotation tests: a function key, a Cosmos DB key and an Event Grid key changed in Key Vault while the clients stay open.

The skipped tests are features a service doesn't offer, such as per-message delays on SNS or Pub/Sub, and the driver declares them as missing capabilities. Azure tier 4's two skips are the claim-check tests instead: that run has no blob store for the claim check. The Spring skips on AWS and Google Cloud are the Key Vault rotation test, which runs on Azure only.

What is tested#

Each cloud is tested in up to five tiers. The tiers are grouped by how expensive and slow the resources are to create, and by how they are reached.

The conformance kits are the same ones the emulator builds run. A real-cloud run doesn't relax them. Where a cloud is eventually consistent, the provider's test class declares it explicitly (for example listingLag() and deleteLag() for AWS Secrets Manager and Azure Key Vault), and the library itself never waits on the caller's behalf. Those runs found the bugs listed on each provider's page, under "What real … found".

How a run works#

One script per cloud runs a tier from start to finish:

cloud-testing/scripts/aws-real-cloud.sh                 # TIER=1 by default; TIER=2, 3 or 4 for the others
cloud-testing/scripts/gcp-real-cloud.sh                 # TIER=1 to 4
cloud-testing/scripts/azure-real-cloud.sh               # TIER=1 to 4
IT_TEST='RealAwsIT$SecretsManager' cloud-testing/scripts/aws-real-cloud.sh     # a single suite
IT_TEST='RealAwsIT$Lambda' SPRING_IT_TEST=RealSpringFunctionIT cloud-testing/scripts/aws-real-cloud.sh   # a suite, plus Spring ones

1. Provision. Terraform in cloud-testing/infra/<cloud>/tier<n>/ creates the run's resources. Every name includes a fresh run id (r<yyMMddHHmm><hex>), and everything is tagged or labelled halyx-test=<run id>. Passwords and SSH keys, where a tier needs them, are generated for the run and passed through the environment only.

2. Test. A Real…IT test class (RealAwsIT, RealGcpSqlIT, RealAzureCacheIT and so on) runs the kits. It is skipped in normal builds and enabled only by a system property the script sets. Tests that need entities Terraform doesn't own, such as topics, queues or secrets, create them with run-specific names.

3. Tear down, always. On exit, failure or Ctrl-C, the script:

   • sweeps anything the tests left behind
   • runs terraform destroy
   • deletes Terraform state that held generated passwords
   • lists every kind of resource the tier creates, and reports nothing left or what remains

   A listing that itself fails is reported as CHECK FAILED, never taken as "nothing left".

Every run in the published history ended with nothing left behind.

Identities and permissions#

The tests never run as an administrator. Each cloud has a dedicated test identity, holding only what the tiers need:

The exact policies and roles are on each provider's page. Credentials never enter the repository.

Running the tests yourself#

You need:

• Terraform 1.5 or later, Maven 3.9 or later and Java 21 or later
• the cloud's CLI, signed in
• for the tunnelled tier 3 runs and the tier 4 runners, ssh and scp

A tier 1 run takes 10 to 30 minutes, and tiers 2 and 3 take 15 to 30, most of it creating and deleting managed services. Tier 4 takes about 10 minutes on Azure, and about 45 on Google Cloud, where creating the Kafka cluster takes 20 to 25. The resources are the smallest tiers available, so a run costs cents.

Portable microservice#

The portable orders service runs one end-to-end suite (PortableOrdersSuite, 10 tests) against every deployment, on Spring Boot 4 and 3.5. A feature a deployment doesn't offer is skipped; the last test checks that a feature that's off has no endpoint, so it is skipped wherever every feature is on.

One image on three clusters#

cloud-testing/scripts/build-portable-orders-images.sh builds the two images once. eks-real-cloud.sh, gke-real-cloud.sh and aks-real-cloud.sh push them and deploy them with one Helm chart. Each cluster is throwaway, logs in with its platform's workload identity, and uses only its own cloud's services. Each run records the digest it pushed and the digest its pods ran, and fails if they differ. In the latest runs, all three registries got the same digests, and every pod ran them:

Each run takes 25 to 45 minutes, most of it creating the managed services; GKE's Managed Kafka cluster alone takes about 15 minutes.

What the runs found:

• Numbers lose their scale. DynamoDB, Cosmos DB and Firestore keep a decimal's value but not its scale, so a price of 3.50 reads back as 3.5. The service formats amounts itself; the in-memory store hid this.
• A halyx.* key is a bean name. Classes called Fulfilment, Receipts and Ledger clashed with the fulfilment subscription, the receipts blob store and the ledger database. Startup failed, naming both.
• Parameter Manager has no emulator. The Google Cloud emulator run reads its settings from Secret Manager instead; the real-cloud run reads them from Parameter Manager.
• A least-privilege policy must cover the health check. The secrets store's health indicator reads a probe secret, <prefix>/halyx-ping, and expects "not found". On EKS the pod's role allowed only the signing key, so the probe got "access denied" and health was DOWN. The role now allows reads under the run's prefix.
• Buildpack images cap direct memory at 10 MB. On AKS the Azure SDK's Netty client ran out (OutOfDirectMemoryError) and a request hung. The chart sets -XX:MaxDirectMemorySize=64M in JAVA_TOOL_OPTIONS, and the memory calculator sizes the heap around it.
• A database user that owns objects can't be dropped. Cloud SQL refused to delete the IAM user, which owns the ledger table, so the first GKE teardown left the instance behind (deleted by hand). Terraform now abandons the user and the database, which go with the instance, and the scripts uninstall the release before destroying.
• AKS details. kube_admin_config is empty on clusters without Entra ID RBAC (kube_config holds the credentials), and Service Bus and Event Hubs namespaces share one DNS zone, so their names must differ. The subscription offers Standard_B2s_v2 for AKS but not Standard_B2s.

Not yet tested on a real cloud#