Outages

Introduction

SOASAP separates local feature flag evaluation from remote configuration delivery. A control-plane or network outage does not automatically stop flag evaluation inside a running application.

When synchronization becomes unavailable:

• the SDK retains its current in-memory snapshot
• existing flags continue evaluating locally
• application requests do not wait for SOASAP Cloud
• dashboard changes stop propagating temporarily
• background reconnection continues

Application behavior during an outage depends on:

• whether a valid snapshot exists
• whether persistent cache is available after restart
• which explicit defaults are defined
• how long stale flag values remain operationally safe

For the architectural model behind this behavior, see High Availability and Offline Operation.

What counts as an outage?

An outage can occur in several parts of the synchronization path. Scenarios may look similar from the SDK perspective, but they can require different operator responses.

SOASAP Cloud outage

The SOASAP control plane is temporarily unavailable.

Network outage

The application cannot reach SOASAP Cloud because of routing, firewall, proxy, or Internet connectivity problems.

DNS failure

The application cannot resolve the SOASAP endpoint.

TLS or connection failure

A secure connection cannot be established.

SSE disconnection

The real-time synchronization connection is interrupted. See SSE Disconnected.

Regional or cluster isolation

Only part of the application fleet loses connectivity.

Credential failure

An invalid, expired, revoked, or incorrectly configured API key prevents synchronization. See Invalid API Key.

Behavior during an outage

While the application process remains running and a snapshot exists:

• evaluations continue from memory
• evaluation latency remains unchanged
• no remote request is introduced into the evaluation path
• existing flag values remain stable
• new dashboard changes are delayed
• defaults are not substituted for valid snapshot values

The last known snapshot becomes temporarily frozen.

What continues to work

During an outage, the following remain available when a valid local snapshot exists:

• ✓ Boolean flag evaluation — on/off reads from the frozen snapshot
• ✓ String flag evaluation — variants and mode strings remain readable
• ✓ Number flag evaluation — limits, timeouts, and thresholds remain readable
• ✓ JSON flag evaluation — structured configuration remains available in memory
• ✓ Local in-memory lookups — O(1) evaluation with no network I/O
• ✓ Existing environment-specific values — previously synchronized environment state remains usable
• ✓ Code-defined defaults — fill gaps for missing or invalid values
• ✓ Application request handling — request paths are not blocked on SOASAP connectivity
• ✓ Previously loaded kill switches — retain their last synchronized value
• ✓ Cached snapshot restoration — after restart, where supported

What is delayed or unavailable

The following depend on a functioning synchronization path:

• newly created flags
• updated flag values
• deleted flag state
• dashboard-driven kill-switch changes
• environment changes
• API key changes
• configuration propagation
• first synchronization for a new application instance without cache
• confirmation that every application instance has received the latest snapshot

Reads from the existing snapshot continue. Changes to the snapshot are delayed.

Configuration freshness during an outage

During normal operation:

During an outage:

Snapshot age is the amount of time since the SDK last received a valid synchronized configuration. It is an operational signal: a larger snapshot age does not necessarily mean evaluation has failed. It means the application may be using increasingly stale configuration.

Frozen-flag behavior

A frozen flag is a flag whose local value cannot change because synchronization is unavailable. Teams should identify how each important flag should behave if frozen for several minutes, several hours, an entire business day, or an extended regional incident.

Release flags

Often safe to remain at the last known value temporarily.

Kill switches

May be unsafe if operators need to change them during the outage. Critical kill switches should:

• be created before dependent code is deployed
• have explicit safe defaults
• be tested in both enabled and disabled states

Operational configuration

May become unsafe when stale. Examples include rate limits, queue controls, integration endpoints, maintenance modes, and workload throttles.

Experiment flags

Usually lower operational risk, but may produce inconsistent exposure across instances during partial outages.

Document for important flags:

• flag owner
• purpose
• safe default
• acceptable stale duration
• outage behavior
• rollback path
• removal date where applicable

Running process vs process restart

A running-process outage and a restart-during-outage are different failure scenarios and must both be tested.

Running process during an outage

The process continues using the last synchronized snapshot.

Restart with persistent cache

Persistent cache allows the SDK to recover the last persisted snapshot without waiting for the network. See Persistent Cache and Cache Strategy.

Restart without persistent cache

Only code-defined defaults are available until synchronization succeeds. See Cache Not Restored.

First deployment during an outage

A newly deployed application instance may have no in-memory snapshot, no persistent cache, and no previous successful synchronization.

In this state the instance can still start, but flag-dependent behavior is limited to code defaults until the first successful sync. Operational implications:

• use explicit defaults for every evaluation
• avoid deploying completely new flag-dependent behavior during a control-plane incident
• ensure critical flags are synchronized before deploying dependent code
• verify cache persistence across replacement instances where applicable

See Non-Blocking Startup and Default Values.

Partial outages

Not every outage affects the entire application fleet. Examples include one pod losing outbound connectivity, one availability zone with DNS issues, one region unable to reach SOASAP Cloud, a proxy affecting only part of the fleet, or some instances reconnecting before others.

Different instances may temporarily evaluate different snapshot versions. This is eventual convergence, not strong consistency. After connectivity returns, instances synchronize toward the latest available state. Monitor synchronization by instance, region, cluster, or deployment group. See Scaling.

Outage detection

Detect outages through synchronization observability rather than application failures alone.

Recommended signals:

• SSE connection state
• duration of the current disconnection
• timestamp of last successful synchronization
• snapshot age
• reconnect attempts
• authentication errors
• DNS or TLS failures
• cache restoration failures
• evaluations using code defaults
• snapshot version where exposed
• percentage of disconnected instances
• regional synchronization differences

Health check behavior

SOASAP connectivity should normally be monitored separately from application liveness.

Liveness

Should indicate whether the application process is functioning. A synchronization outage should not usually make liveness fail.

Readiness

Should indicate whether the application can safely serve traffic. When a valid snapshot or safe defaults exist, temporary disconnection should not automatically make the application unready.

Diagnostic health

Should expose SOASAP synchronization status separately. Suggested states:

• synchronized
• reconnecting
• offline with snapshot
• offline without snapshot
• authentication failed
• cache restore failed
• stale beyond configured threshold

Applications with strict freshness requirements may choose stricter readiness rules, but this is an application-level policy.

Incident response

Use a practical response sequence during a synchronization outage.

Step 1: Confirm the scope

Determine:

• whether the issue affects SOASAP Cloud, networking, credentials, DNS, or SSE
• whether all instances are affected
• which environments are affected
• when the last successful synchronization occurred

Step 2: Confirm evaluation continuity

Verify:

• application requests continue
• existing flags are evaluated
• no request-time network dependency has appeared
• fallback-to-default rates are not unexpectedly increasing

Step 3: Identify frozen critical flags

Review:

• kill switches
• operational controls
• incident-related flags
• flags with short acceptable stale durations

Step 4: Avoid risky configuration assumptions

Do not assume dashboard changes have reached disconnected instances. Do not treat a successful dashboard update as proof of fleet-wide propagation. See Flag Not Updating.

Step 5: Protect the application

Use application-level mitigations when necessary:

• traffic reduction
• deployment rollback
• disabling dependent jobs through existing controls
• isolating affected regions
• reverting application code
• using infrastructure-level controls

Step 6: Monitor recovery

Track:

• reconnect progress
• last synchronization timestamps
• snapshot convergence
• authentication errors
• regional recovery

Step 7: Verify configuration after recovery

Confirm that intended values reached all relevant application instances.

Rollbacks during an outage

A dashboard rollback cannot reach an SDK that remains disconnected. Operators should not rely exclusively on remote flag changes during a synchronization outage.

Fallback rollback paths include:

• application deployment rollback
• infrastructure traffic controls
• disabling workloads outside SOASAP
• queue pausing
• external integration shutdown
• database or service-level controls
• regional failover

Once synchronization returns, dashboard changes can propagate normally. Document both flag-based rollback and non-flag rollback for critical production features.

Kill switches during outages

A kill switch only provides remote operational control while configuration updates can reach the SDK.

During an outage:

• an already synchronized kill switch retains its last value
• changing it in the dashboard may not affect disconnected instances
• a newly created kill switch cannot protect instances that have never received it
• an explicit default applies only when no valid snapshot value exists

Recommend:

• create kill switches before deploying risky functionality
• default critical kill switches to the safest code behavior
• test kill switches before production rollout
• maintain a non-SOASAP emergency rollback path
• document how long the current value may safely remain frozen

See Production Safety.

Reconnection and recovery

Recovery should not require application restart, SDK recreation, manual cache deletion, or request-handler changes. See Real-Time Synchronization.

After reconnection:

• the SDK refreshes its local state
• subsequent evaluations use the updated snapshot
• different instances may recover at different times
• operators should verify fleet convergence

Post-recovery validation

Service recovery is not complete until configuration state has been validated across the relevant fleet.

• ✓ SSE connections have recovered
• ✓ Snapshot age has returned to normal
• ✓ Authentication errors have cleared
• ✓ Intended dashboard changes have propagated
• ✓ Critical flags have the expected values
• ✓ All regions and instances have converged
• ✓ Default fallback rates are normal
• ✓ Cache persistence is functioning
• ✓ No temporary mitigation remains active unintentionally
• ✓ Application metrics have returned to baseline

Failure-scenario table

Testing outage behavior

Validate failure behavior in staging before production incidents expose gaps.

Test 1: Block outbound SOASAP traffic

Verify:

• evaluations continue
• application requests remain healthy
• synchronization status becomes degraded
• reconnect attempts are observable

Test 2: Disconnect SSE only

Verify:

• the snapshot remains active
• evaluation latency does not change
• reconnection occurs automatically

Test 3: Restart with network blocked and cache present

Verify:

• the snapshot restores successfully
• the application becomes ready
• values remain available without synchronization

Test 4: Restart with network blocked and no cache

Verify:

• explicit defaults are used
• startup does not block indefinitely
• health status clearly reports the missing snapshot

Test 5: Change a flag while an instance is offline

Verify:

• connected instances receive the change
• the disconnected instance retains its previous value
• the disconnected instance converges after recovery

Test 6: Simulate an extended outage

Verify:

• frozen critical flags remain safe
• monitoring alerts at the expected threshold
• operators can follow documented rollback paths

Test 7: Invalidate an API key

Verify:

• authentication failure is distinguishable from temporary network loss
• evaluation continues from the existing snapshot
• operators can identify and correct the credential issue

Common misconceptions

"If SOASAP Cloud is down, application flag reads stop."

False. Existing values continue evaluating from the local snapshot.

"Defaults replace all values during an outage."

False. Valid snapshot values remain active. Defaults apply when no valid synchronized value exists.

"A dashboard change is an immediate rollback during every outage."

False. Disconnected SDKs cannot receive the update until connectivity returns.

"An SSE disconnect means the application is down."

False. It normally means configuration freshness is degraded.

"Every application instance will hold the same value throughout a partial outage."

False. Instances may temporarily use different snapshot versions.

"Persistent cache guarantees the latest configuration."

False. It restores the latest successfully persisted snapshot, which may be stale.

"Recovery is complete as soon as SOASAP Cloud is reachable."

False. Operators should verify reconnection and configuration convergence across the fleet.

Production recommendations

Outage runbook template

Copy the following into an internal runbook and fill it during an incident.

Incident:
Affected environment:
Affected regions or clusters:
Start time:
Last successful synchronization:
Current snapshot age:
Percentage of disconnected instances:
Critical frozen flags:
Application impact:
Current mitigations:
Non-flag rollback options:
Recovery status:
Post-recovery validation:
Incident owner:
Link to retrospective:

Relationship to other SOASAP concepts

Related documentation

• High Availability — availability boundaries and evaluation independence
• Cache Strategy — production guidance for persistent snapshots
• Scaling — multi-instance and fleet considerations
• Security — API keys, environments, and access boundaries
• Production Safety — kill switches, defaults, and safe rollout practices
• Offline Operation — evaluation while sync is unavailable
• Real-Time Synchronization — SSE delivery and reconnect behavior
• Persistent Cache — snapshot restoration across restarts
• Non-Blocking Startup — readiness without waiting on first sync
• Default Values — code-defined fallbacks
• SSE Disconnected — diagnosing sync interruptions
• Cache Not Restored — restart resilience failures
• Invalid API Key — credential failures that block sync
• Flag Not Updating — diagnosing propagation and convergence issues