Invalid API Key

Diagnose authentication failures that prevent SDK synchronization.

Problem

The SDK cannot authenticate with SOASAP Cloud.

Typical observations:

HTTP 401 during synchronization
authentication repeatedly fails
SDK never receives configuration
reconnect attempts continue but fail
application uses default values
cache restores but never refreshes
new deployments stop synchronizing
only one environment is affected

Authentication, not evaluation. An invalid API key prevents synchronization. It does not affect local evaluation of an already restored snapshot.

Application Requests → Local Evaluation → Continues Working

↓

SOASAP Cloud → Authentication → Failed → No New Configuration

This page is an authentication troubleshooting runbook. For credential lifecycle and storage practices, see API Keys and Security.

What an API key does

SDK → SOASAP Cloud

↓

API key authenticates synchronization

API keys authenticate SDK-to-Cloud synchronization. They do not:

authenticate end users
authorize application requests
replace application authentication
secure business operations
control feature visibility by themselves

Treat “invalid API key” as a sync-authentication failure, not as an application authorization failure.

Expected behavior

Healthy SDK:

authenticates successfully
synchronizes configuration
maintains real-time updates

Invalid API key:

synchronization is rejected
configuration is not refreshed
existing local snapshot or restored cache may continue working
explicit defaults are used if no usable snapshot exists

See Offline Operation and Persistent Cache.

Quick diagnostic checklist

Is the API key present in the running process?
Was the correct environment used?
Was the correct project used?
Was the deployment secret updated?
Was the key rotated or regenerated recently?
Was the previous key invalidated before all deployments migrated?
Did processes reload or restart after the secret changed?
Does only one environment fail?
Does only one replica or region fail?
Was the key copied or injected correctly (no truncation or whitespace)?
Is authentication the only failing component (not DNS, TLS, or proxy)?

Never log the full key. Prefer presence checks, non-secret identifiers, or truncated previews already shown by approved tooling.

Understand the symptom

First deployment

No synchronization ever succeeded. Likely missing key, wrong key, wrong environment, or secret not injected.

Existing deployment suddenly stops

Possible regenerated/invalidated key, secret overwritten, incomplete rotation rollout, deployment rollback, environment mismatch, or wrong secret reference.

Cache works, authentication fails

Application continues evaluating cached values while freshness stops advancing. Fix credentials; do not delete cache. See Cache Not Restored.

Defaults returned

Authentication failed and no usable snapshot is available (no prior sync, no cache, or cache not restored).

1. Verify the API key is present

Confirm the running application actually received a non-empty key.

Possible mistakes:

missing environment variable
empty secret
deployment typo in secret name
configuration provider not loaded
wrong configuration file or profile
secret scoped to another namespace or account

Log only presence or a non-secret identifier where supported — never the complete key.

2. Verify the correct environment

Each environment has its own SDK key. Production keys belong to Production; Development keys belong to Development.

Production deployment → Development key
or
Production deployment → Staging key

Environment mismatches often appear identical to invalid credentials from the application’s perspective: synchronization fails or returns unexpected configuration scope. Confirm both the dashboard environment and the secret injected into the running deployment.

3. Verify the project

A key from another SOASAP project does not authenticate the intended configuration scope. Check project, organization, and deployment configuration against the consuming service’s documented ownership.

4. Recent key regeneration

In SOASAP, rotating an environment key regenerates it and permanently invalidates the previous value. There is one active SDK key per environment.

Ask:

Was a key regenerated recently?
Was every consuming deployment updated with the new value?
Did processes reload the new secret?
Were workers, jobs, inactive slots, and other regions included?

Coordinated sequence for routine rotation:

Inventory All Consumers

↓

Prepare Secret Updates

↓

Regenerate Environment Key

↓

Deploy New Key Across Fleet

↓

Verify Authentication and Synchronization

For compromise response, regenerate immediately even if some instances are not ready — accept temporary sync interruption for unmigrated consumers. Details: API Keys.

5. Deployment secrets

Verify the value actually injected by the deployment platform:

Kubernetes Secret
Docker / Compose secret
environment variable
cloud secret manager
CI/CD deployment secret
platform-managed configuration provider

Dashboard shows current key
↓
Deployment still injects previous secret
↓
Running process authenticates with invalidated value
↓
401 / sync rejected

Confirm the secret reference name, account/namespace, version/alias if used, and that the deployment pipeline targets the intended environment.

6. Kubernetes

Updating a Kubernetes Secret alone does not necessarily reload credentials already held in memory by running processes.

Secret updated
↓
Pods not restarted / not reloaded
↓
Old key still loaded
↓
Authentication continues to fail

Check:

Secret exists in the correct namespace
Pod mounts or envFrom reference the intended Secret
rollout or restart occurred after the Secret change
all Deployments, StatefulSets, Jobs, and CronJobs that need the key were updated
one Deployment was missed while another succeeded

7. Containers

Possible causes:

old image with baked configuration (anti-pattern)
stale Compose or stack file
wrong service environment block
container not recreated after secret change
host env overriding container env unexpectedly

Prefer runtime secret injection over embedding credentials in images.

8. Serverless

Warm instances may retain older environment configuration while cold instances receive the new secret, producing mixed authentication outcomes.

Warm instance → Old environment value → 401
Cold instance → New environment value → Authenticated

Confirm the deployment completed, aliases or traffic shifts finished, and long-lived warm environments were recycled according to platform behavior.

9. Multiple replicas and regions

Replica A → Authenticated
Replica B → 401
Replica C → Authenticated

Possible causes: stale secret on one rollout slice, failed rollout, wrong environment on one Deployment, node-specific config, or a missed region.

One healthy Pod does not prove the entire fleet migrated. Compare by Deployment, region, cluster, start time, and secret version. See Outages for partial-fleet patterns.

10. Verify authentication is the only failure

Do not treat every sync failure as an invalid key.

Signal	Likely layer	Next page
HTTP 401 / authentication rejected	Credentials / scope	This runbook
Timeouts, intermittent disconnects	Networking / proxy / SSE	SSE Disconnected
DNS resolution failure	Connectivity	SSE Disconnected
TLS handshake / trust failure	Certificates / inspection	SSE Disconnected
Auth succeeds; value still wrong	Flag key / app cache / logic	Flag Not Updating

Copy and injection errors

Keys may fail authentication because of truncation, leading/trailing whitespace, newline characters from paste, wrong secret field, base64 decode mistakes in manifests, or quoting issues in shell injection. Prefer secret-manager references over manual paste into tickets or chat.

Should you restart?

Restart (or reload) helps after the secret was updated but the process still holds the previous in-memory value, or after deployment configuration was fixed.

Restart does not fix:

an invalidated or wrong key still injected by the platform
environment or project mismatch
empty or missing secret
networking, DNS, TLS, or proxy problems

Restart one canary after correcting the secret. Do not restart the entire fleet hoping authentication will heal itself.

Should you delete cache?

No — not for authentication failures. Cache integrity is unrelated to whether the current key is accepted. Deleting cache during auth failure usually makes the situation worse by removing the last usable snapshot and forcing defaults until sync succeeds.

Observability

Recommend monitoring:

authentication failure rate
synchronization failure rate
401 rate where exposed
successful authentication / sync rate
time since last successful synchronization
deployment version / release
key regeneration / rotation change events (operational calendar)
per-replica and per-region auth outcomes

Local evaluation success is not proof that authentication is healthy.

Alerting

Informational — single failed authentication on a known misconfigured test instance.
Warning — repeated authentication failures on a subset of replicas.
Critical — entire Production fleet returns 401 / cannot synchronize while configuration changes are expected.

Failure-scenario table

Scenario	Symptoms	Root Cause	Resolution
Missing key	No sync; defaults or no snapshot	Secret not injected	Add secret; verify process env
Wrong environment	401 or unexpected scope	Staging/Dev key in Production (or reverse)	Inject environment-correct key
Wrong project	Auth fails or wrong inventory	Key from another project	Use intended project key
Regenerated / invalidated key	Sudden fleet or partial 401	Previous key no longer valid	Deploy current key; reload processes
Old deployment / secret not updated	Dashboard key new; apps still 401	Deployment lag	Update secret reference and roll out
Pod restart missing	Secret updated; still 401	In-memory old credential	Rollout / restart consumers
Partial rollout	Some replicas OK, some 401	Incomplete migration	Finish rollout; verify every group
Copy / injection error	Consistent 401 on new deploys	Truncation, whitespace, wrong field	Re-inject from dashboard/secret manager
Expired / stale deployment configuration	Only one pipeline target fails	Wrong secret alias or old manifest	Correct pipeline config; redeploy
Serverless warm instances	Mixed auth after secret change	Old env retained	Complete deploy; recycle warm envs
One region missed	Regional 401	Regional secret lag	Update that region; verify sync

Safe recovery procedure

Verify the expected project.
Verify the expected environment.
Verify the current environment key in the dashboard (Owners/Admins).
Verify the deployment secret reference and value presence (not the full key in logs).
Verify deployment version and rollout status across replicas and regions.
Update the secret where needed, then restart or reload updated instances.
Confirm successful authentication on a canary instance.
Confirm synchronization connects and a known non-sensitive flag updates.
Confirm remaining replicas and regions converge.
Remove obsolete secret versions from deployment systems according to policy.

Verify recovery

✓ authentication succeeds
✓ synchronization connected
✓ dashboard updates arrive
✓ all replicas synchronized
✓ all regions synchronized
✓ no repeated 401
✓ last successful sync is recent
✓ defaults no longer used unexpectedly

Recovery is incomplete until fleet convergence is confirmed, not merely one successful canary.

Common misconceptions

"401 means networking is broken."

False. 401 indicates rejected authentication. Timeouts and disconnects point to transport problems instead.

"Restart always fixes invalid keys."

False. Restart only helps after the correct secret is already available to inject.

"Deleting cache fixes authentication."

False. Cache is unrelated and may remove the last usable snapshot.

"API keys authenticate users."

False. They authenticate SDK synchronization to SOASAP Cloud.

"One working Pod means deployment succeeded."

False. Partial rollouts leave other replicas on invalidated credentials.

"Environment does not matter."

False. Each environment has its own key and configuration scope.

"Rotating a key instantly updates every deployment."

False. Regeneration invalidates the previous key immediately; deployments must receive and load the new value.

"If flags still evaluate, the key must be valid."

False. Local snapshots and cache can continue while authentication fails.

Prevention

✓ One key per environment — isolate Development, Staging, and Production.
✓ Store Production keys in an approved secret manager — inject at deploy time.
✓ Controlled rollout after regeneration — inventory consumers before routine rotation.
✓ Verify rollout before considering rotation complete — check every replica and region.
✓ Monitor authentication failures separately from request health.
✓ Document secret ownership and consumers — required for safe regeneration.
✓ Test secret reload behavior in staging — platforms differ on restart requirements.

Production troubleshooting runbook

Incident owner:
Application:
Project:
Environment:
SDK language / version:
Application version:
Deployment platform:
First seen:
401 / auth failure confirmed?
Last successful synchronization:
Cache restored at startup?
Defaults returned?
Key regeneration recently performed?
Secret store / reference name:
Secret updated?
Processes restarted or reloaded?
Affected replicas:
Affected regions:
Healthy replicas also present?
Networking / DNS / TLS ruled out?
Root cause:
Resolution:
Canary auth verified?
Fleet sync verified?
Completion time:
Retrospective link:

Do not include full API keys, authorization headers, or secret values.

Relationship to other documentation

Concept	Relationship
API Keys	How to create, store, regenerate, and inventory credentials
Security	Broader secret-handling and environment isolation
SSE Disconnected	Transport failures that look like sync loss without 401
Cache Not Restored	Startup defaults when no snapshot exists alongside auth failure
Flag Not Updating	Broader stale-value diagnosis after auth is healthy
Outages	Extended sync loss and fleet impact
Offline Operation	Local evaluation while sync cannot authenticate