Microsoft Azure: real-cloud results

Results of running the Halyx conformance kits against real Microsoft Azure services in australiaeast (Sydney), acting as the halyx-test service principal. Every run creates its own throwaway resources and deletes them afterwards. See Cloud testing for how the runs work.

Tier 1: latest result#

cloud-testing/scripts/azure-setup-test-principal.sh      # once, as a subscription Owner
cloud-testing/scripts/azure-real-cloud.sh                # AZURE_LOCATION=australiaeast by default
IT_TEST='RealAzureIT$Cosmos' cloud-testing/scripts/azure-real-cloud.sh     # one suite only

Identity. Terraform and the tests act as a dedicated halyx-test service principal. Its credentials live in ~/.azure/halyx-test.env, outside the repository and readable only by its owner. The setup script:

• registers the resource providers
• creates the principal with Contributor on the subscription
• gives it Role Based Access Control Administrator with a condition that lets it assign and remove only the data-plane roles the tests use (listed under IAM)

The tests reach every service through DefaultAzureCredential's environment-variable path, as a deployed application would.

Provision. Terraform (cloud-testing/infra/azure/tier1) creates a resource group of the run's own, tagged halyx-test=<run id>, with:

• a StorageV2 account
• a Standard Service Bus namespace
• a serverless Cosmos DB account with the items and items-sorted containers
• a Key Vault in RBAC mode
• a free App Configuration store
• a Standard Event Grid namespace with an orders topic and an inbox pull (queue) subscription; azurerm has no resources for namespace topics, so those two go through the azapi provider
• the function kit's function app (the emulator tests' Python app) on a Basic plan's single instance, so the runs it counts in memory hold for every invocation; the tests call the function with its function key. Its host has a storage account of its own, which the azurerm provider requires, reached with the account key: the principal may not give the app's managed identity storage roles. Nothing under test uses that account

The kits log in with Entra ID only: shared keys and local auth are off, except on the function host's own storage account (see below), and on the Cosmos DB account and the Event Grid namespace, whose keys only the key rotation tests use. Those tests (RealAzureIT$KeyRotation, and aFunctionKeyFromKeyVaultFollowsARotation in RealSpringFunctionIT) keep a key in the run's Key Vault and change it there while the client stays open: a function key read on every call, and the Cosmos DB and Event Grid keys switched from the primary to a wrong key and then to the secondary. The script passes the keys through the environment. Cache and Kafka password rotation runs on the emulators only (Valkey, and Kafka 3.9 and 4.2 brokers): the clouds' caches and Kafka endpoints here use Entra ID or IAM. The principal gets Storage Blob Data Contributor, Service Bus Data Owner, Key Vault Secrets Officer, App Configuration Data Owner, EventGrid Data Sender and Receiver, and Cosmos DB's built-in Data Contributor. The script waits until the blob role reaches the data plane (one attempt this run) and gives the rest 30 seconds more.

Test. RealAzureIT creates Service Bus topics (session-enabled with duplicate detection for the ordering, deduplication and delay tests), subscriptions with a 5-second lock, and a queue.

Tear down. terraform destroy deletes the resource group and purges the Key Vault and App Configuration store rather than leaving them soft-deleted. A check then lists resource groups, tagged resources, and deleted vaults and stores; a failed listing is reported as CHECK FAILED.

Skipped tests, both expected:

• Blob presignWithoutCapabilityIsRejected. This test only applies to providers that can't presign; Azure can.
• App Configuration earlierVersionsCanBeRead. App Configuration keeps no readable history of a key, so the driver doesn't declare VERSIONS.

Service Bus passes every messaging test, including the ones other providers skip: duplicate detection and delayed delivery.

Key Vault timing. The driver no longer waits for Key Vault's long-running deletes: deletedNamesCanBeReused took 3.5s, the kit retrying while the name was still being deleted (see the findings).

The repository kit and the Spring Boot repository test run on the items and items-sorted containers, through halyx-spring-data-document. Cosmos DB needs no index beyond its default indexing.

Per-test results#

Tier 2: latest result#

TIER=2 cloud-testing/scripts/azure-real-cloud.sh

The script first registers Microsoft.DBforPostgreSQL and Microsoft.Sql, as the service principal through the REST API. Terraform (cloud-testing/infra/azure/tier2) then creates:

• PostgreSQL: a flexible server (16, B_Standard_B1ms) with Entra and password auth. The service principal is its Entra admin (role halyx-test), and a password admin halyx_admin gets a password generated for the run and passed through the environment only.
• Azure SQL: a Basic database on a server that allows Entra ID only, with the service principal as its admin. SQL logins are refused.

Both accept only the test machine's address. Teardown deletes the Terraform state, which held the password.

MySQL is not run. An Entra admin on Azure Database for MySQL needs a managed identity that can read the directory (Graph User.Read.All, GroupMember.Read.All and Application.Read.All, or Directory Readers), which only a directory admin can grant. Left out by choice for now.

On Azure SQL the service principal's session runs as <application id>@<tenant id> (SUSER_SNAME()). The token is the driver's accessToken property, with no username.

Login latency, from run r26101122362e30, the last one that timed logins:

Tier 3: latest result#

TIER=3 cloud-testing/scripts/azure-real-cloud.sh

The script registers Microsoft.Cache, then Terraform (cloud-testing/infra/azure/tier3) creates an Azure Managed Redis cache (Balanced_B0, no high availability) with:

• access keys off, so only Entra ID works
• the EnterpriseCluster clustering policy: one endpoint, with no redirects to node addresses
• encrypted connections on the public endpoint (port 10000)

The service principal gets the cache's default access policy. This subscription doesn't offer classic Azure Cache for Redis in Australia East, and Managed Redis is its successor. The username is the principal's object id, which Halyx reads from the token's oid claim.

Tier 4: latest result#

TIER=4 cloud-testing/scripts/azure-real-cloud.sh

The script registers Microsoft.EventHub, then Terraform (cloud-testing/infra/azure/tier4) creates three Event Hubs namespaces (Standard, one throughput unit) with shared access keys off, so only Entra ID works, and makes the principal their Azure Event Hubs Data Owner. RealAzureStreamIT runs the stream kit on the namespaces' Kafka endpoints (port 9093), logging in with Entra ID tokens through SASL OAUTHBEARER. It creates each test's topics (event hubs) through Kafka's admin API, logged in the same way, and deletes older ones as it goes. Before each creation it logs how many topics the namespace lists.

After the kit, RealSpringStreamIT runs on Spring Boot 3.5 and on Spring Boot 4, with the Spring for Apache Kafka and Kafka clients each manages, against the namespace the kit used least. In one Spring application it sends and receives records through the starter's StreamProducer bean and @HalyxListener, and through Spring for Apache Kafka's KafkaTemplate and @KafkaListener, which log in through halyx.spring-kafka.cluster. It creates its two topics with Kafka's admin client, logged in with kafkaClientProperties.

KAFKA_CLIENTS=4.2.1 TIER=4 cloud-testing/scripts/azure-real-cloud.sh runs the same tests on Kafka 4 clients.

Three namespaces, because a Standard namespace refused new event hubs with PolicyViolationException after about twenty had been created in a run, even with only one or two alive (see the run history). The test moves to the next namespace before one has created twelve.

The same kit, with idempotence on, also passed 24/25 on Kafka 4.2.1 clients (r2610161121e9c0).

Topics per namespace in the passing run: 11, 11 and 4, with one listed before each creation, so deletes do take effect.

Tier 5: latest result#

TIER=5 AZURE_FEDERATED_GOOGLE_SA=<Google Cloud test service account> cloud-testing/scripts/azure-real-cloud.sh

Terraform (cloud-testing/infra/azure/tier5) creates a storage account with shared keys off, and gives the principal Storage Blob Data Contributor on it. For the federated login, the script gets a Google ID token for the Google Cloud test service account, with the audience api://AzureADTokenExchange, which the halyx-test application trusts through a federated identity credential (set up once). That is code on Google Cloud logging in to Azure with no stored secret.

RealAzureLoginIT names the principal explicitly:

Then Spring Boot 3.5 and 4, each twice: credentials.type: service-principal with secret (from the environment), and with token-file. 4/4; Spring 4/4, run r26102311158b75, nothing left after teardown.

Run history#

What real Azure found#

The emulators (Floci-az and Microsoft's Service Bus emulator) missed all of these:

Event Hubs' Kafka endpoint found these:

IAM#

Tiers 2 and 3 need nothing beyond tier 1's setup: Contributor covers registering their resource providers, creating the servers and caches, and naming the principal as their Entra admin or access-policy holder.

Tier 5 needs nothing beyond tier 1's setup for its secret login. Its federated login needs a federated identity credential on the halyx-test application, created once, that trusts Google ID tokens for the Google Cloud test service account:

az ad app federated-credential create --id <application id> --parameters '{
  "name": "google-halyx-test",
  "issuer": "https://accounts.google.com",
  "subject": "<the service account unique id: gcloud iam service-accounts describe <email> --format=value(uniqueId)>",
  "audiences": ["api://AzureADTokenExchange"]
}'

The run then gets a token with gcloud auth print-identity-token --impersonate-service-account, for the service account in AZURE_FEDERATED_GOOGLE_SA, so the person running it needs roles/iam.serviceAccountTokenCreator on that account, as for the Google Cloud runs. Without AZURE_FEDERATED_GOOGLE_SA, the federated test is skipped.

IAM for the AKS run#

cloud-testing/scripts/aks-real-cloud.sh runs the portable orders service on AKS, and needs nothing beyond tier 1's setup. Contributor creates the cluster, its registry and a user-assigned managed identity, and federates the identity to the service's Kubernetes service account. The identity gets the same data-plane roles the principal may assign: Storage Blob Data Contributor, Azure Service Bus Data Owner, Key Vault Secrets Officer, App Configuration Data Owner, Azure Event Hubs Data Owner, and EventGrid Data Sender and Receiver. It also gets Cosmos DB Built-in Data Contributor, the Managed Redis default access policy, and the PostgreSQL server's Entra admin. The pods pull images with the registry's admin credentials, because AcrPull isn't among the roles the principal may assign.