Parameters and secrets: ParameterStore

One API for configuration values and secrets on every cloud and on HashiCorp Vault.

Reading and writing#

ParameterStore secrets = halyx.provider("aws").parameterStore(
        ParameterStoreConfig.secrets().prefix("orders-svc").cacheTtl(Duration.ofMinutes(5)).build());

String password = secrets.require("db/password");          // Secrets Manager secret "orders-svc/db/password"
Optional<Parameter> key = secrets.get("api/key");          // empty if missing
Map<String, Parameter> db = secrets.getAll("db");           // everything at or below orders-svc/db
secrets.put("api/key", newKey);                            // creates, or adds a new version
secrets.delete("api/key");                                  // idempotent

ParameterStore config = halyx.provider("aws").parameterStore(ParameterStoreConfig.parameters().build());
Optional<Integer> poolSize = config.get("db/pool/size", Integer::valueOf);

require throws NotFoundException for a missing name; get returns an empty Optional. A Parameter carries the name, value, version and lastModified time.

Store options:

Portable names#

Names are segments of letters, digits and single dashes separated by /, such as orders/db/password. Each backend stores them in its own naming scheme:

getAll returns everything at or below a prefix, and never matches partial segments: db doesn't match dbx/....

JSON secrets#

Read fields of JSON object values, such as RDS and Aurora managed secrets or multi-field Vault secrets:

String rdsPassword = secrets.require("db/rds", "password");              // one field
Parameter credentials = secrets.get("db/rds").orElseThrow();
Optional<String> user = credentials.field("username");
Map<String, String> all = credentials.fields();                          // flattened: db.host, hosts[0]

Versions#

Where the backend keeps versions, earlier ones are readable:

Parameter current = secrets.get("api/key").orElseThrow();
Optional<Parameter> previous = secrets.get("api/key", previousVersionId);

capabilities().supports(ParameterCapability.VERSIONS) says whether a store keeps versions. App Configuration doesn't.

Asynchronous calls#

async() gives the store's asynchronous view: get, require, getAll, put, delete and ping as CompletableFutures, through the same read cache, statistics and interceptors as the blocking calls. A cached read completes at once; the rest go to the backend with no thread waiting, on every backend: SSM Parameter Store, Secrets Manager, App Configuration, Key Vault, Secret Manager, Parameter Manager and Vault. Three calls still block, on a virtual thread: Key Vault deletes and the recovery a conflicting Key Vault write starts (the SDK starts its long-running operations only through a poller), and Vault's login, when the token is missing or about to expire.

secrets.async().require("api/key").thenAccept(client::authenticate);

Watching for changes#

watch polls a prefix and reports added, updated and removed values, which suits credential rotation:

try (AutoCloseable watch = secrets.watch("db", Duration.ofMinutes(1), changes ->
        changes.updated().forEach((name, p) -> pool.rotateCredentials(p.fields())))) {
    ...
}

One daemon thread serves all watchers. Each change a listing shows is confirmed with a read by name before it's reported, since some listings briefly lag or go back and forth after a write: a change is reported once, when the read agrees, and never undone by a stale listing. A detected change also refreshes the store's cache with the values the watch saw. Failed polls are logged and retried. statistics() counts cache hits, backend reads, detected changes and watch failures.

In Spring Boot, spring.config.import=halyx:<store> loads a store into the Environment and can reload it. See Spring Boot.

Feature flags#

A parameter store makes a portable home for feature flags: the same flags, managed the same way, on SSM Parameter Store, App Configuration, Parameter Manager or Vault. Halyx stores them; a flag library evaluates them. Any library with a pluggable source works, such as flipjack's PollingFeatureToggles, a Togglz StateRepository, an FF4J FeatureStore or an OpenFeature provider. This example uses flipjack, which routes calls between an implementation and its fallback so business code has no flag checks:

@Bean
OverridableFeatureToggles featureToggles(@Qualifier("settings") ParameterStore settings) {
    return PollingFeatureToggles.builder(() -> {
                Map<String, String> flags = new LinkedHashMap<>();
                settings.getAll("features").forEach((name, p) ->           // features/bulk-discount -> bulk-discount
                        flags.put(name.substring(name.lastIndexOf('/') + 1), p.value()));
                return flags;
            })
            .interval(Duration.ofSeconds(30))
            .requireInitialLoad()
            .build()
            .withOverrides();
}

@Component
@Toggleable(value = "bulk-discount", fallback = StandardPricing.class)
class BulkDiscountPricing implements OrderPricing { ... }

• getAll returns names relative to the store's prefix, the same on every backend, so features/bulk-discount under the prefix orders-svc is the flag bulk-discount.
• Flipjack polls, so a change takes effect within the poll interval plus the store's cacheTtl. Give a flags store no cache, or a short one.
• A flag that is missing from the store is off. Features start off until the store turns them on, and deleting a flag turns it off at the next poll.
• requireInitialLoad() stops the start when the store can't be read, rather than starting with every flag off.
• withOverrides() adds kill switches: toggles.override("bulk-discount", false) turns a feature off at once, even while the store is unreachable, and clearOverride hands it back to the store.

A parameter store holds on/off flags per deployment. Per-user targeting and percentage rollouts need a flag platform such as flagd, Unleash or LaunchDarkly; flipjack's flipjack-openfeature module takes any OpenFeature provider as its source in place of PollingFeatureToggles, and the routing stays the same.

Without Spring, build the same PollingFeatureToggles and wrap beans with ToggledFeature.of(...). The portable orders sample runs this on every cloud.

Keys for other services#

A store can hold the keys and passwords that other services log in with, read as they're used so a rotation takes effect without a restart: a SQL password (SqlAuth.password(user, supplier)), a cache password (CacheAuth.password(supplier)), a Kafka password (StreamAuth.password(mechanism, user, supplier)), an Azure function key, a Cosmos DB account key or an Event Grid access key (see Provider configuration). Give the store a cacheTtl, which also bounds how long a rotation takes to land. In Spring Boot, these are auth.secret and secrets entries; see Spring Boot.

A store on another cloud#

A parameter store names its own provider, so a service on one cloud can read its keys from a store on another, or from Vault: a function on Azure whose key is in AWS Secrets Manager, for example.

ParameterStore shared = halyx.provider("aws").parameterStore(
        ParameterStoreConfig.secrets().cacheTtl(Duration.ofMinutes(5)).build());

FunctionInvoker quote = halyx.provider("azure").functionInvoker(FunctionConfig.builder("rate-quote")
        .property(AzureProperties.FUNCTIONS_ENDPOINT, "https://pricing.azurewebsites.net")
        .property(AzureProperties.FUNCTIONS_KEY, () -> shared.require("rate-quote-key"))
        .build());

What to plan for:

• Login to the store's cloud. The service needs an identity there as well as on its own cloud. Workload identity federation avoids a stored key: the store's provider uses the default chain, which logs in with a token the cluster issues (on AWS, a role trusting the cluster's issuer, through AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE; on Google Cloud, a workload identity pool's credential configuration). A static key for the store's cloud would itself need rotating.
• Availability. The service depends on a second cloud. Keys are read when clients are built, so a start fails while the store is unreachable; the read cache rides out shorter outages once running.
• Latency and egress. A read after the cache expires crosses clouds. A cacheTtl of minutes makes reads rare.
• Bootstrap credentials stay each cloud's workload identity, since they can't come from the store.

Native names#

ParameterStoreConfig.nativeNames(true) uses the backend's own names as they are, with no validation and no separator translation; the prefix is then a literal string prefix. Use it for existing values like /app/db_password, or AWS-managed rds!db-... secrets.

Backend behaviour#

• AWS Systems Manager Parameter Store: getAll (GetParametersByPath) can return an older view for a second or two after a write, including one older than a listing just before it; reads by name are current. watch confirms changes by name, so it isn't affected.
• AWS Secrets Manager: getAll (and so watch and config reloads) is eventually consistent, so a secret written moments ago may not be listed yet. A read by name can briefly lag a write. Deletes finish asynchronously (up to about 20 seconds measured): until then a deleted secret may still be read, and writing its name throws the retryable UnavailableException. Deletes skip the recovery window. The IAM policy needs secretsmanager:ListSecrets for getAll, plus GetSecretValue, PutSecretValue, CreateSecret and DeleteSecret. Granting BatchGetSecretValue on "*" lets getAll read 20 secrets per call; without it Halyx reads them one at a time and logs a warning.
• Azure Key Vault: deletes are soft deletes that finish asynchronously, and delete doesn't wait. Writing a name that is soft-deleted, or still being deleted or recovered, throws the retryable UnavailableException. If the name was soft-deleted, the write also starts its recovery, so a retry stores the new value without needing the purge permission.
• Google Parameter Manager: there's no "latest" alias and the caller names each version. Halyx writes version ids that sort by creation time and reads the newest enabled version, which is also correct for versions written by other tools. A read lists versions and then fetches one, so give these stores a cacheTtl. Deletes remove all versions, then the parameter.
• HashiCorp Vault: deletes remove all versions. Secrets written by other tools without a value field are read as their JSON data.

HashiCorp Vault#

The halyx-vault module supports:

• token auth, or VAULT_TOKEN from the environment
• a token file (a Vault Agent sink), read again whenever Vault rejects the token
• AppRole, Kubernetes, GCP and Azure login
• Vault namespaces
• a private CA (vault.ca-certificate)

It renews tokens before they expire. It uses the JDK HTTP client and a built-in JSON codec, so it adds no dependencies that could clash with the cloud SDKs. See Provider configuration.

Native clients#