Google Cloud: real-cloud results

Results of running the Halyx conformance kits against real Google Cloud services in australia-southeast1 (Sydney), acting as the halyx-test service account, impersonated through Application Default Credentials. Every run creates its own throwaway resources and deletes them afterwards. See Cloud testing for how the runs work.

Tier 1: latest result#

GCP_PROJECT=<project> GCP_REGION=australia-southeast1 cloud-testing/scripts/gcp-real-cloud.sh
IT_TEST='RealGcpIT$Firestore' cloud-testing/scripts/gcp-real-cloud.sh     # one suite only

Identity. Terraform and the tests act as a dedicated halyx-test service account, through Application Default Credentials that impersonate it:

gcloud auth application-default login --impersonate-service-account=halyx-test@<project>.iam.gserviceaccount.com

Provision. Terraform (cloud-testing/infra/gcp/tier1) creates the following, labelled halyx-test=<run id>:

• a Cloud Storage bucket, which also holds the messaging kit's claim-check payloads
• a Firestore database of the run's own, so the project's (default) database is never touched
• the composite indexes the document store's index queries need: status with placedAt ascending and descending, on both test tables, plus a collection-group index on owner
• an Eventarc Advanced message bus, an enrollment matching every event, and a pipeline that publishes them to a Pub/Sub topic with an OAuth token for halyx-test, and the topic's inbox subscription
• the function kit's Cloud Run function (the emulator tests' main.py, on python312), with one instance taking one request at a time so the runs it counts in memory hold for every invocation, its source bucket, and roles/run.invoker on it for the test service account only

Test. RealGcpIT creates Pub/Sub topics and subscriptions named after the run, plus tck-* secrets and parameters (Parameter Manager both global and regional).

Tear down. A sweep deletes what the tests left behind, then terraform destroy runs, then a check lists every kind of resource. A listing that fails is reported as CHECK FAILED, never taken as empty.

Skipped tests, all expected:

• Cloud Storage presignWithoutCapabilityIsRejected. This test only applies to providers that can't presign; GCS can.
• Pub/Sub repeatedDeduplicationIdIsDeliveredOnce and delayedMessageIsHeldBack. Pub/Sub has no publish-side deduplication and no per-message delay, so the driver declares neither capability.
• Eventarc eventsWithoutDataArrive. Pub/Sub refuses a pipeline's publish of an event without data, so the event never arrives; declared as a known gap (see what real GCP found).

Pub/Sub's time is mostly the redelivery tests, which wait for the 10s ack deadline to pass.

The repository kit and the Spring Boot repository test run on the run's items and items-sorted collections and their status/placedAt composite indexes, through halyx-spring-data-document.

Per-test results#

Tier 2: latest result#

TIER=2 GCP_PROJECT=<project> GCP_REGION=australia-southeast1 cloud-testing/scripts/gcp-real-cloud.sh

Terraform (cloud-testing/infra/gcp/tier2) creates Cloud SQL for PostgreSQL 17 and for MySQL 8.4:

• shared-core db-f1-micro on the ENTERPRISE edition (newer versions default to Enterprise Plus, which has no shared-core tiers)
• IAM authentication on, encrypted connections only, no backups, no deletion protection
• a public IP that accepts only the test machine's /32; the Cloud SQL Java connector needs no authorized network

Each instance has three users:

• halyx_admin, a built-in user with a password generated for the run
• the test service account as an IAM user: halyx-test@<project>.iam on PostgreSQL and halyx-test on MySQL
• halyx_password_only, a built-in user that an IAM token must not log in

On MySQL the test then grants the IAM user SELECT on the app database, as the admin. IAM users start with no privileges there; see the findings below.

Login latency#

From run r2610111859cad1, the last one that timed logins.

Times per new connection, in the order they ran, on newly created instances:

• PostgreSQL: a token login costs about 140 ms more than a password login. There's no slow first login like RDS's 5–6s.
• MySQL on db-f1-micro: slow for every login, passwords included. The shared-core instance is the bottleneck, not IAM or Halyx, and a larger tier would bring it close to PostgreSQL. Its warm token logins (up to 2.3s) are near the test's 3s limit, so that limit could become flaky on this tier.
• The connector costs the same as a direct connection once warm.

Tier 3: latest result#

TIER=3 GCP_PROJECT=<project> GCP_REGION=australia-southeast1 cloud-testing/scripts/gcp-real-cloud.sh

Terraform (cloud-testing/infra/gcp/tier3) creates the following, in a VPC and subnet of the run's own:

• Cache: a Memorystore for Valkey 8.0 instance: cluster mode disabled, SHARED_CORE_NANO, IAM authorization, and TLS (SERVER_AUTHENTICATION) with a server certificate from Memorystore's private CA. Cluster mode is off so the client gets one endpoint and no redirects to node IPs a tunnel can't reach.
• Private endpoint: a service-connection policy (gcp-memorystore), under which Memorystore creates its Private Service Connect endpoint in the subnet.
• Bastion: an e2-micro VM with no service account. A firewall rule allows SSH only from the test machine's /32, with a key generated for each run.

The script then:

1. fetches the instance's CA (gcloud memorystore instances get-certificate-authority)
2. forwards 127.0.0.1:16380 through the bastion to the private endpoint
3. runs RealGcpCacheIT

The certificate names the private endpoint, not the tunnel's 127.0.0.1, so the caches use redis.verify-peer=ca. That checks the chain against the instance's CA, but not the name.

This is the first time Memorystore IAM authentication has been verified end to end. Before this, MemorystoreIamTokens was only unit-tested, against a plain Valkey container with a password.

Unlike AWS tier 3, Memorystore IAM has no per-user access: the token's principal needs memorystore.instances.connect (roles/memorystore.dbConnectionUser), and there are no database users to mix up. So there is no "denied user" test. The rejection tests cover a missing and a forged token.

Tier 4: latest result#

TIER=4 GCP_PROJECT=<project> GCP_REGION=australia-southeast1 cloud-testing/scripts/gcp-real-cloud.sh

Terraform (cloud-testing/infra/gcp/tier4) creates the following, in a VPC and subnet of the run's own:

• Cluster: a Managed Service for Apache Kafka cluster with the smallest capacity, 3 vCPUs and 3 GiB. Its brokers get Private Service Connect addresses in the subnet, and the service adds their DNS names to the VPC. There is no public endpoint.
• Runner: an e2-small Ubuntu 24.04 VM that runs as the test service account. A firewall rule allows SSH only from the test machine's /32, with a key generated for each run. Its startup script installs Java 21.
• Claim checks: a Cloud Storage bucket, so the claim-check test runs too.

Kafka clients connect to the bootstrap address, then to each broker by the name the cluster advertises, so an SSH tunnel to one address, as in tier 3, can't work. The tests run on the runner instead. The script builds them on the test machine, copies them with their classpath (cloud-testing/scripts/in-vpc-runner.sh, 172 jars, 141 MB), and runs RealGcpStreamIT there with the JUnit console launcher. The reports come back into halyx-gcp/target/failsafe-reports. No credentials leave the test machine: the runner gets tokens from its metadata server, as an application on Compute Engine would.

RealGcpStreamIT logs in with Google IAM through SASL OAUTHBEARER, the token wrapping the service account's access token. It creates each test's topics through Kafka's admin API, logged in the same way, and deletes older ones as it goes. Then RealSpringStreamIT runs on the runner on Spring Boot 3.5 and on Spring Boot 4: in one Spring application, the starter's StreamProducer bean and @HalyxListener, and Spring for Apache Kafka's KafkaTemplate and @KafkaListener logging in through halyx.spring-kafka.cluster. The script stages the three test modules' jars into one directory on the runner and sends each jar once (313 MB in all).

The first tier 4 run (r261016095774dd) was the first time Managed Kafka's IAM login had been checked end to end: until then its tokens had only been unit-tested (ManagedKafkaTokensTest). It passed first time and found nothing to fix.

Tier 5: latest result#

TIER=5 GCP_PROJECT=<project> cloud-testing/scripts/gcp-real-cloud.sh

Terraform (cloud-testing/infra/gcp/tier5) creates a bucket, and grants roles/storage.objectUser on it to the login target, halyx-login-target: a standing service account with no other roles, which the test service account may impersonate (set up once).

RealGcpLoginIT builds Halyx with Credentials.impersonate(target), starting from Application Default Credentials:

Then Spring Boot 3.5 and 4: a provider with credentials.type: impersonate and a BlobStore bean that uses it. 3/3; Spring 2/2, run r2610231129846c, nothing left after teardown.

Run history#

What real GCP found#

The emulators (Floci-gcp, and Google's Pub/Sub and Firestore emulators) missed all of these:

IAM#

The project also needs these APIs enabled: the IAM Service Account Credentials API (iamcredentials.googleapis.com) for impersonation, Eventarc and Eventarc Publishing (eventarc.googleapis.com, eventarcpublishing.googleapis.com) for tier 1's event bus, and the Cloud SQL Admin API (sqladmin.googleapis.com) for tier 2, and Compute Engine, Memorystore, Network Connectivity and Service Consumer Management (compute, memorystore, networkconnectivity, serviceconsumermanagement) for tier 3, and Managed Service for Apache Kafka (managedkafka) for tier 4.

Tier 5 needs a standing service account to impersonate, created once:

gcloud iam service-accounts create halyx-login-target --display-name="Halyx tier 5: impersonation target"
gcloud iam service-accounts add-iam-policy-binding halyx-login-target@<project>.iam.gserviceaccount.com \
    --member=serviceAccount:halyx-test@<project>.iam.gserviceaccount.com --role=roles/iam.serviceAccountTokenCreator

Its refusal test impersonates the Compute Engine default service account (<project number>-compute@developer.gserviceaccount.com), on which halyx-test has no rights; set GCP_LOGIN_DENIED to use another.

IAM for the GKE run#

cloud-testing/scripts/gke-real-cloud.sh runs the portable orders service on GKE. The nodes and the pods run as halyx-test, which already holds every data-plane role above, so the service needs nothing new. Creating the cluster and its registry does:

The project also needs the Kubernetes Engine and Artifact Registry APIs (container, artifactregistry). The workload identity binding can only be made once the project's workload identity pool exists, which happens when the first cluster with Workload Identity is created. So the script makes the binding itself, right after the cluster is created, as the person running it; on later runs it changes nothing.

gcloud services enable container.googleapis.com artifactregistry.googleapis.com
for role in roles/container.admin roles/artifactregistry.admin; do
  gcloud projects add-iam-policy-binding <project> --member=serviceAccount:halyx-test@<project>.iam.gserviceaccount.com \
      --role=$role --condition=None
done