SOASAP
Sign In Sign Up Free

Production Safety

Use kill switches, explicit defaults, environment isolation, and controlled rollouts to reduce deployment risk and improve operational resilience.

Introduction

SOASAP is designed to support safe production releases. Feature flags are not only a delivery mechanism — they are operational controls that allow teams to reduce deployment risk, limit blast radius, disable problematic behavior without redeploying, and separate deployment from activation.

Production safety comes from combining multiple mechanisms:

  • Kill switches — Boolean flags that disable risky paths from the dashboard in near real time
  • Explicit defaults — code-defined fallbacks that fail safe during incidents and startup gaps
  • Environment isolation — separate API keys and flag values per environment
  • Local evaluation — predictable O(1) reads that do not depend on live connectivity
  • Offline operation — continued evaluation from the last snapshot during outages

No single mechanism is sufficient alone. Safe production operation requires intentional flag design, architecture-aware operations, and runbooks that account for synchronization limits during extended disconnects.

Kill switches

A kill switch is a Boolean flag used to disable functionality immediately from the SOASAP dashboard — without redeploying application code or restarting processes. 

if (flags.GetBool("enable-payments", defaultValue: false))
{
    ProcessPayment();
}

When a production issue occurs:

Connected SDKs receive the update over SSE and apply it atomically to the in-memory snapshot. Subsequent evaluations return false. No redeploy is required. No restart is required.

Common kill switch use cases:

  • Payment processing — stop charging when a downstream processor fails
  • Background jobs — pause workers that amplify errors or load
  • Integrations — disable third-party API calls while partners recover
  • Expensive workloads — halt CPU-, memory-, or cost-intensive code paths

Default kill switches to false in code so never-synced or first-start instances fail safe. Name flags after the enabled state (enable-payments) or the disable action (disable-payments) consistently across services.

Offline limitation. Kill switch changes do not reach SDKs while SSE is disconnected. During extended outages, instances continue serving the last synchronized value. Plan safe defaults and alternative operational controls for disconnect scenarios. See Offline Operation.

See Boolean Flags.

Explicit defaults

Every flag evaluation must provide an explicit default value defined in application code. Defaults are production behavior — not optional parameters. 

flags.GetBool("new-checkout", defaultValue: false);

Defaults matter during:

  • Missing flags — key never created or absent from snapshot
  • Deleted flags — flag removed from dashboard after last sync
  • Startup synchronization — first SSE payload not yet received
  • Cache restoration — key absent from persisted snapshot
  • Outages — new keys cannot sync; existing keys use snapshot values

Safe defaults reduce unexpected behavior during incidents. Feature gates should default to false; limits should default to conservative values; JSON configs should default to fully constructed safe objects.

See Default Values.

Environment isolation

Development, Staging, and Production must remain isolated. Each environment binds to a separate SOASAP environment with its own API key, flag values, and rollout decisions.

Each environment should use:

  • Separate API keys — never share Production keys with lower environments
  • Separate flag values — Staging exercises new values before Production changes
  • Separate rollout decisions — enabling a flag in Development does not affect Production

Isolation reduces operational risk: a Staging experiment cannot accidentally toggle Production behavior; credential leaks in non-production environments do not expose Production configuration; operators can validate changes in Staging with production-like code paths before cutover.

Store API keys in environment-specific secret stores — not in source control. Inject keys at deploy time through your platform's configuration mechanism.

See Environment Separation.

Safe rollouts

Avoid enabling new functionality for all Production users immediately. Use feature flags to decouple code deployment from feature activation and limit blast radius during validation.

Recommended progression:

  1. Deploy code with the feature disabled (false default, Production flag off)
  2. Validate in Development and Staging with flag enabled
  3. Enable in Production for internal users or a subset (when supported by application logic)
  4. Expand exposure after metrics and error rates remain stable
  5. Remove the flag and dead code after the rollout is complete

Feature flags reduce blast radius: a bad rollout affects only instances that evaluate the flag as enabled. A kill switch or revert disables the feature for connected SDKs without rolling back the deployment artifact.

SOASAP provides environment-scoped flag values. Percentage rollouts and user targeting require application-side logic or future platform capabilities — design explicit branching in code when gradual exposure is required.

See Deployment Strategies.

Failure scenarios

Feature causes errors

Disable the kill switch or feature flag in the Production dashboard. Connected SDKs receive the update over SSE within typical sub-second propagation on healthy networks. Monitor error rates while propagation completes across the fleet.

Bad configuration

Revert the flag value in the dashboard to the last known good setting. Audit logs record who changed the value and when. Validate the corrected value in Staging before re-applying experimental changes to Production.

Synchronization delay

During the window between dashboard save and SDK receipt, instances serve the previous snapshot value. Code defaults apply only for keys not in the snapshot. Design defaults so delayed propagation fails safe — not open.

SOASAP Cloud outage

Existing snapshots continue evaluating locally. Applications remain operational with frozen configuration. Kill switch activations during the outage do not reach disconnected SDKs until sync resumes. See Outages.

Deployment failure

Flags remain independent from deployment artifacts. Rolling back a bad deploy reverts code; flag state in SOASAP persists across deploys. After rollback, verify flag values still match intended operational state — a prior deploy does not reset dashboard configuration.

Relationship to other architecture components

Production safety depends on SOASAP's local-first architecture:

Component Production safety role
Local Evaluation Predictable O(1) reads — flag checks do not fail when SOASAP Cloud is slow or unreachable
Persistent Cache Startup durability — restarts during incidents restore last snapshot from disk
Offline Operation Runtime resilience — evaluation continues during control-plane and network failures
Default Values Safe fallback — fail-safe behavior when configuration is missing or invalid
Real-Time Synchronization Operational control — dashboard changes reach connected SDKs over SSE in near real time

Operators control configuration in the dashboard. Synchronization delivers changes to SDKs. Local evaluation applies configuration on every request without network dependency. Defaults cover gaps when synchronization is incomplete or data is absent.

Common mistakes

Using unsafe defaults

Defaulting feature gates to true or limits to unbounded values means missing flags silently enable risky behavior. Defaults must fail safe under outage and first-start conditions.

Sharing API keys between environments

A Staging toggle affects Production when keys are shared. Credential rotation becomes all-or-nothing. Use one API key per SOASAP environment.

Leaving stale rollout flags indefinitely

Long-lived rollout flags accumulate dead code paths, inconsistent defaults across services, and operational confusion about which path is canonical. Remove flags after rollout completes.

Enabling large features for all users at once

Full immediate activation maximizes blast radius. Deploy with flag off; enable progressively after Staging validation and Production smoke tests.

Treating feature flags as permanent configuration

Flags are operational controls and rollout tools — not a substitute for environment variables, secrets, or durable application config. Long-lived flags should migrate to static configuration once behavior stabilizes.

Production checklist

  • Use explicit defaults — pass a safe default on every getter; review defaults in code review as production behavior.
  • Use separate API keys per environment — Development, Staging, and Production each bind to a distinct SOASAP environment.
  • Create kill switches for critical functionality — payments, outbound integrations, and expensive workloads should be disable-able from the dashboard.
  • Test rollbacks regularly — practice disabling flags in Staging; verify propagation time and application behavior after revert.
  • Validate in Staging before Production — enable new values in Staging with production-like traffic patterns before Production changes.
  • Monitor synchronization health — alert on prolonged SSE disconnect; application health does not imply configuration freshness.
  • Document operational flags — maintain a runbook listing kill switches, owners, expected defaults, and safe-to-freeze behavior during outages.
  • Remove obsolete rollout flags — delete flags and dead code after rollout completes to reduce long-term operational debt.

Incident response

Feature flags reduce recovery time by allowing operators to disable behavior without redeploying. They do not replace root-cause analysis or fix underlying defects.

During an incident:

  1. Identify the affected feature — correlate errors with recent deploys, dashboard changes, and flag keys in application code
  2. Disable the flag if appropriate — toggle the kill switch or feature flag in Production; prefer disable over enable when uncertain
  3. Verify propagation — confirm connected SDKs receive the update; account for SSE delay and disconnected instances during outages
  4. Monitor recovery — watch error rates, latency, and business metrics after the flag change; propagation is not instantaneous across all edge cases
  5. Investigate the underlying issue — flags mitigate impact; fix the root cause in code, configuration, or dependencies before re-enabling

Document post-incident actions: whether the flag default was safe, whether offline instances blocked kill switch effect, and whether the flag should remain or be removed after fix deployment.

Related documentation