Cache Not Restored

Problem

After an application, process, container, Pod, host, or mobile application restarts, the SOASAP SDK does not appear to restore the previously synchronized configuration snapshot.

Common observations include:

• flag evaluations return application-defined defaults
• previously available flag values disappear after restart
• the application works while online but starts with defaults when offline
• an old value appears instead of the latest dashboard value
• the cache file or storage entry exists but appears to be ignored
• cache restoration works on one replica but fails on another
• a process restart preserves values, but a container or Pod replacement does not
• a mobile application loses its values after reinstall, data clearing, or storage-provider failure
• newly created replicas have no cached snapshot
• the cache restores successfully, but no newer updates arrive

Persistent cache is a startup durability mechanism. It allows an SDK to reconstruct its local in-memory snapshot after a restart without requiring an immediate successful connection to SOASAP Cloud. See Persistent Cache.

Expected startup flow

Cache restoration and cloud synchronization are independent. A successful cache restoration means local state is available after restart. It does not prove that SSE is connected, that authentication succeeds, that the snapshot is current, or that recent dashboard changes were received. A failed restoration does not necessarily prevent the SDK from receiving a new snapshot later.

Critical mental model

Persistent cache provides durability. Real-time synchronization provides freshness. An old value after restart may mean the cache worked correctly but synchronization has not refreshed it. A default value after restart may mean no usable cached snapshot was available. Do not treat both symptoms as the same failure.

Cache state model

Operators must distinguish these states:

• Cache disabled — the SDK was not configured to persist snapshots.
• No cache written — persistence was enabled, but no valid snapshot was ever synchronized and stored.
• Cache missing — a cache existed previously, but it is absent from the location used by the restarted process.
• Cache not found — the cache may still exist, but the current process resolves a different path, namespace, volume, user profile, or storage provider.
• Cache unreadable — storage exists, but the process cannot access it because of permissions, ownership, mounting, locking, sandboxing, or storage-provider failure.
• Cache empty or incomplete — a previous write may have been interrupted or failed.
• Cache rejected — the SDK found data but could not safely restore it (invalid data, corruption, incompatible format, wrong scope, or unsupported provider state, depending on SDK behavior).
• Cache restored but stale — an older valid snapshot was restored that predates the expected dashboard change.
• Cache restored successfully — the expected snapshot is available for local evaluation after restart.

Quick diagnostic checklist

1. Confirm persistent cache is enabled.
2. Confirm the previous SDK instance completed synchronization.
3. Confirm a valid snapshot existed before shutdown.
4. Confirm the snapshot was successfully persisted.
5. Confirm the restarted process uses the same cache provider.
6. Confirm the effective cache path or namespace has not changed.
7. Confirm the storage survived the lifecycle event.
8. Confirm the runtime identity can read the cache.
9. Confirm the runtime can also write and replace cache data.
10. Confirm sufficient disk capacity, quota, and inode availability.
11. Confirm the cache belongs to the same project and environment.
12. Check for corruption or documented incompatibility.
13. Confirm the application is not evaluating before restoration completes.
14. On mobile, confirm the configured storage provider is available.
15. Verify authentication and live synchronization separately.
16. Delete or quarantine cache only after identifying a cache-specific failure.

Identify the exact symptom

Defaults immediately after restart, then correct values later

Likely no cache was restored, the application evaluated before restoration completed, or fresh synchronization later supplied the snapshot.

Defaults persist indefinitely

Likely cache restoration failed and synchronization also failed, or the API key, flag key, type, or SDK client is wrong.

Old values restore, then become current

Cache restoration succeeded. The cache was older than the cloud snapshot and was refreshed after synchronization.

Old values persist indefinitely

Cache restoration probably succeeded, but synchronization is disconnected, blocked, or unauthenticated. See SSE Disconnected.

Only new replicas return defaults

Existing instances may have local cache storage that new replicas do not inherit.

Process restart works, Pod replacement fails

The cache is likely stored in process-local, container-local, Pod-local, or node-local storage.

Cache exists but is ignored

Possible wrong path, permission failure, invalid or incompatible data, wrong environment identity, or another SDK client evaluating.

Mobile cache disappears after reinstall

Application-private storage may have been removed with the application.

1. Confirm persistent cache is enabled

Do not assume every SDK enables durable persistence automatically. Verify the actual SDK configuration: persistence enabled, supported provider or adapter configured, configured cache instance passed to the SDK client, production settings not disabling persistence, evaluation through the same SDK client that owns the cache, and no environment-specific override.

SDK Client A → Persistent Cache Enabled
Application Evaluation → SDK Client B → Persistent Cache Disabled

Use one long-lived SDK client per required project and environment scope where supported. See SDK Installation and platform cache docs such as .NET Persistent Cache.

2. Confirm a valid snapshot existed

A cache cannot restore a snapshot that was never obtained. Before the previous shutdown, confirm the API key was valid, initial synchronization completed, the expected environment was synchronized, evaluations returned synchronized values, a valid in-memory snapshot existed, the process remained active long enough for persistence, and no cache-write failure was reported.

Configuring a cache provider does not guarantee that any cache data has been written.

3. Confirm the cache was written

Receiving a snapshot in memory and persisting it to storage are separate events. Possible write failures include missing or non-writable directory, read-only filesystem, full disk, exhausted inodes, exceeded quota, unavailable volume, process termination during write, mobile storage rejection, concurrent writers, I/O errors, and failed atomic replace/rename.

Check:

• SDK diagnostics where available
• application logs
• operating-system storage errors
• orchestrator events
• cache entry or file existence
• modification timestamp
• non-zero size where filesystem storage is used
• storage-provider errors
• process termination history
• node disk-pressure events

Do not manually edit cached data.

4. Check the effective cache location

The restarted SDK must resolve the same effective cache location used by the previous instance. Location changes commonly come from relative paths, environment variables, deployment manifests, runtime user changes, new data directories, container filesystems, volume mounts, host replacement, versioned release directories, package identity changes, mobile build variants, or test versus production configuration.

Record the resolved path or namespace in diagnostics where safe. Do not log raw cached content. See Cache Strategy.

Relative paths

Configured: ./soasap-cache

Deployment A cwd: /app        → /app/soasap-cache
Deployment B cwd: /srv/service → /srv/service/soasap-cache

The second process does not find the cache written by the first. Prefer explicit absolute paths on server platforms where the SDK supports them. Mobile storage providers may use logical namespaces instead of filesystem paths.

Directory existence, permissions, and ownership

The configured parent directory may not exist in a new runtime. Check existence, parent creation, volume mounts, cleanup scripts, temporary-directory cleanup, mobile storage initialization, and user profile. Do not assume every SDK creates missing directories — follow SDK-specific behavior.

The runtime may need permission to read, list, create temporary files, write, replace, rename, delete invalid data, and update metadata. A readable file inside a non-writable directory may restore once but fail to persist subsequent updates.

Previous Deployment → Runs as root → Cache owned by root
New Deployment → Runs as non-root → Cache read or replacement fails

Use a stable least-privileged runtime identity, configure volume ownership deliberately, and test with the same identity as the application.

Read-only filesystems and capacity

A read-only filesystem may allow reading an existing cache while preventing newer writes, temporary files, atomic replacement, or removal of invalid data. In containers with a read-only root filesystem, configure an explicitly writable cache mount without disabling security controls globally.

Check free disk space, inodes, quotas, ephemeral-storage limits, PVC capacity, mobile free storage, and node disk pressure. A full disk may create a stale-cache incident rather than a missing-cache incident.

Understand storage lifetime

“Restart” is ambiguous. Cache durability must match the actual lifecycle event:

• Application or process restart — usually reuses the same filesystem and user profile.
• Container process restart — may retain the current container’s writable layer.
• Container recreation / image update — often loses container-local state.
• Container restart inside the same Pod — may retain Pod-level storage such as emptyDir.
• Pod replacement / rescheduling — removes Pod-local storage; may lose node-local state.
• Host or VM restart — may preserve durable disks but not temporary disks.
• VM replacement — may start from a clean machine image.
• Serverless cold start — may use a new execution environment with no prior local state.
• Mobile force stop — normally differs from clearing application data.
• Mobile data clear / reinstall — commonly removes application-owned persistent data.

Containers, Docker, and Kubernetes

A cache written into a container’s writable layer is tied to that container’s lifecycle. Mount a writable volume when cache must survive container replacement, verify the mounted path matches SDK configuration, and do not bake runtime cache into images.

Docker storage choices differ:

• Writable layer — may survive process restart; removed with the container.
• Bind mount — lifetime follows the host directory; check path, ownership, host replacement.
• Named volume — may survive recreation if the same volume is reattached.
• Temporary filesystem — intentionally non-durable.

Kubernetes models:

• Container writable layer — not durable across Pod replacement.
• emptyDir — typically survives container restarts in the same Pod; removed when the Pod is deleted.
• Host-local storage — binds cache to a node; may fail after rescheduling.
• PersistentVolumeClaim — may preserve storage across Pod replacement depending on binding, access mode, reclaim policy, attachment, mount path, and storage class.

Check mount path, PVC bound state, attachment, security context, read-only mounts, disk pressure, and whether a new release uses a different claim. Do not use Kubernetes Secrets or ConfigMaps as snapshot-cache storage.

Multiple replicas and shared cache

Every SDK instance owns its own in-memory snapshot. Persistent cache should normally also be isolated per active SDK instance unless the SDK explicitly supports safe shared access.

Unsafe:
Replica A ↘
           One Shared Mutable Cache File
Replica B ↗

Preferred:
Replica A → Cache A
Replica B → Cache B
Replica C → Cache C

A shared filesystem does not automatically make a shared mutable cache safe. Concurrent writes, locking, stale overwrites, partial writes, and corruption are possible. StatefulSets can simplify per-replica volumes but are not mandatory for every SOASAP deployment.

VMs, bare metal, and serverless

On VMs and physical servers, check new release directories, replacement VMs, temporary disks, service-user changes, immutable image rollouts, cleanup scripts, home-directory changes, disk failure, host moves, and restored machine snapshots containing older cache. Place operational cache outside disposable release directories when durability is required.

On serverless platforms, local storage may be reused during warm execution but should not generally be treated as durable across environment replacement. Design startup to tolerate missing local cache, use safe defaults, and test cold starts without cache.

Mobile storage model

Mobile cache restoration depends on the storage provider used by the SDK or application. Possible abstractions include AsyncStorage, application-private filesystem, platform preferences, key-value storage, SDK-provided adapters, and custom application storage. Not every mobile SDK uses AsyncStorage.

Potential causes:

• storage provider not installed or not initialized
• adapter not passed to the SDK
• application data cleared or application reinstalled
• device or simulator reset
• storage permission or availability changed
• application identifier or build variant changed
• restoration attempted before storage was ready
• device capacity prevented writes
• user changed devices without restoring that storage

React Native and AsyncStorage

For React Native deployments that use AsyncStorage, verify the dependency and native integration, that the application can read and write values, that the intended adapter is supplied to the SOASAP SDK, that storage initializes before restoration is required, that production and development use the intended provider, and that application data or emulator state has not been reset. See React Native AsyncStorage Cache.

Android and iOS considerations

Uninstall, clear storage, different application IDs or bundle identifiers, work profiles, emulator/simulator reset, build variants, device restore limitations, locked-device protected storage, and low free storage can all affect persistence. Exact SOASAP SDK behavior depends on the configured provider — confirm in platform documentation such as Kotlin Persistent Cache.

Desktop applications

Check other user accounts, portable versus installed data directories, package-identity changes, sandbox path changes, cleanup software, roaming profiles restoring older cache, multiple instances writing one file, and post-install permissions. Prefer the platform’s standard application-data location where supported.

Project, environment, and API key scope

A persisted snapshot belongs to a specific configuration scope. Do not let one project or environment become the startup state for another.

Previous Deployment → Production API Key → Production Cache
New Deployment → Staging API Key → Same Cache Path

Possible outcomes include rejection, unexpected restoration if isolation is not enforced, or subsequent writes overwriting another environment’s cache. Isolate cache locations or namespaces per project and environment. After API key rotation, verify the replacement key targets the same intended scope, the application loaded the new secret, and the SDK supports restoring existing cache under the replacement key where documented. See API Keys.

Conceptual namespacing example (not a required SOASAP path):

/cache/soasap/<application>/<environment>/<instance>/

Corrupted or incompatible cache

Corruption may come from interrupted writes, abrupt termination, storage failure, manual modification, partial copies, multiple writers, filesystem corruption, provider errors, or unsupported migration. Symptoms include rejection, parse/validation failure, repeated defaults, single-replica impact, or recovery after a clean synchronization.

An SDK upgrade, provider change, or platform migration may make existing data unusable. Check release notes and migration guidance; test restoration during every SDK upgrade rollout.

Safe corruption recovery

1. Confirm that the cache is the likely failing component.
2. Verify the correct project, environment, and API key.
3. Verify network connectivity for fresh synchronization.
4. Record cache location, modification time, size, ownership, and affected instance.
5. Stop or isolate the affected instance.
6. Quarantine the suspected cache rather than deleting all evidence.
7. Start one instance without the suspected cache.
8. Confirm authentication succeeds and a fresh snapshot is received.
9. Confirm expected flag values and a successful new cache write.
10. Restart again with network access blocked and confirm restoration.
11. Remove the quarantined cache after investigation according to policy.

Stale cache vs failed restoration

A stale cache is not a failed cache. Restoration behaved correctly; it does not contain the later update. Restore synchronization and allow the current cloud snapshot to replace the restored state. Do not delete a valid stale cache merely because it is old. Define a stale-data objective per production system based on flag criticality and acceptable update delay.

Initialization order and non-blocking startup

An application may evaluate flags before restoration has completed, depending on SDK initialization semantics. Follow the SDK-specific pattern, await initialization where documented, keep explicit defaults safe, and distinguish first evaluation from later evaluations without blocking indefinitely on cloud connectivity. See Non-Blocking Startup and Default Values.

A healthy application process does not prove that cache restoration succeeded. Defaults can mean no cache, unreadable or rejected cache, incomplete restoration, missing snapshot, wrong key or type, another client, or intentional override.

Multiple SDK clients and concurrent processes

Client A → Correct Project → Cache Enabled → Snapshot Restored
Client B → Different Configuration → No Cache → Default Returned

Check dependency injection, singleton versus transient lifetime, static clients, background services, mocks, credentials, and accidental recreation. Independent active writers should not share one mutable cache location unless explicitly supported.

Cleanup, temporary storage, and security policies

Cache may have been stored in disposable locations: temporary directories, release directories, scheduled cleanup, container GC, ephemeral storage, node cleanup, OS disk cleanup, mobile cache-directory eviction, CI workspaces, or user-profile cleanup. Prefer a location whose lifecycle matches required durability.

Security controls (SELinux, AppArmor, sandboxes, antivirus, controlled-folder access, Kubernetes policies, mobile sandboxes) may block access. Check policy events before weakening protections; prefer an approved writable location.

Should you restart again?

Repeated restarts do not correct storage configuration. A controlled restart may help after path, mount, permissions, identity, provider, corruption quarantine, initialization, or deployment fixes. Restart does not fix missing durable storage, wrong lifecycle assumptions, full disk, wrong environment, unavailable storage provider, invalid API key, blocked synchronization, a cache that was never written, or unsafe shared writers. Restart one canary after correcting the root cause.

Should you delete the cache?

Delete or quarantine only when corruption is confirmed, the data belongs to the wrong project or environment, documented incompatibility exists, the SDK repeatedly rejects it, controlled clean-start testing requires it, or support/SDK guidance recommends it. Deleting a valid cache during an outage may remove the last usable snapshot.

Diagnostic procedure

1. Record the symptom — expected/actual value, default?, start time, lifecycle event, affected instance.
2. Verify configuration scope — project, environment, API key reference, flag key, type, SDK client.
3. Verify cache configuration — enabled, provider, configured and effective path/namespace, client lifetime, app version.
4. Verify the previous snapshot — prior auth, sync, expected values, no write error.
5. Verify storage existence — entry/directory/volume/provider, mtime, size.
6. Verify access — identity, ownership, read/write/replace, security-policy events, read-only mounts.
7. Verify capacity and health — disk, inodes, quota, device storage, filesystem/volume health.
8. Verify storage lifetime — process, container, Pod, node, VM, cold start, data clear, reinstall.
9. Verify scope and compatibility — project/env, key rotation, SDK version, provider change, concurrent writers.
10. Verify synchronization separately — auth, DNS/network, SSE, newer snapshot, subsequent cache write.
11. Correct the root cause — path, mount, permissions, capacity, provider, lifecycle, isolation, corruption.
12. Prove recovery — online start + sync + write, then offline restart + restored evaluation, then reconnect.

Reproduction test

1. Configure persistent cache.
2. Start with SOASAP Cloud reachable; confirm auth and synchronization.
3. Evaluate a dedicated test flag; confirm persistence succeeds.
4. Stop cleanly; block SOASAP network access; restart with the same storage.
5. Confirm the cached value is restored.
6. Restore network; change the test flag; confirm update and cache rewrite.
7. Stop again; block network; restart; confirm the newer value restores.
8. Repeat using the actual production lifecycle (container recreation, Pod replacement, rescheduling, VM replacement, cold start).
9. For mobile, also test force stop, device restart, offline launch, data clear, reinstall, simulator reset, and provider unavailability.

Verifying the fix

• ✓ persistent cache is enabled
• ✓ intended provider is configured
• ✓ effective cache location is correct
• ✓ storage survives the required lifecycle
• ✓ directory or namespace exists
• ✓ runtime identity can read and write/replace
• ✓ sufficient disk, quota, and inodes
• ✓ valid snapshot existed before shutdown
• ✓ cache writes complete successfully
• ✓ restart restores the expected snapshot
• ✓ application does not evaluate too early unintentionally
• ✓ project and environment match
• ✓ active writers use isolated locations
• ✓ synchronization reconnects after startup
• ✓ newer snapshots replace stale cache
• ✓ second offline restart restores the newer value

Observability and alerting

Where supported, collect cache enabled state, provider type, resolved path/namespace, restoration attempt/result, failure category, restored snapshot timestamp, cache age, last write time, write success/failure, capacity, volume health, process start time, initialization status, last successful sync, SSE state, and active value source (synchronized snapshot, persistent cache, preload, or explicit default). Combine SDK telemetry with application logs, filesystem metrics, Kubernetes events, cloud storage events, and mobile storage diagnostics.

• Informational — no cache on a known first start.
• Warning — restoration fails but sync succeeds quickly with safe defaults; or cache age exceeds freshness objective.
• High — Production starts on defaults with neither cache nor sync; multi-replica restoration failure; PVC inaccessible.
• Critical — cache loss plus control-plane/network outage with unsafe defaults.

Failure-scenario table

Root-cause decision tree

Defaults appear after restart
↓
Was persistent cache enabled?
  No  → Configure cache or rely on safe defaults + live sync.
  Yes ↓
Did the previous instance obtain a valid snapshot?
  No  → Complete successful synchronization first.
  Yes ↓
Was the snapshot successfully persisted?
  No  → Diagnose write permissions, capacity, provider, shutdown.
  Yes ↓
Does the cache exist at the currently resolved location?
  No  → Diagnose path changes and storage lifetime.
  Yes ↓
Can the current process read it?
  No  → Fix permissions, ownership, mount, sandbox, provider.
  Yes ↓
Was it rejected as corrupted, incompatible, or wrong-scope?
  Yes → Controlled quarantine + fresh synchronization.
  No  ↓
Did the application evaluate before restoration completed?
  Yes → Correct initialization ordering.
  No  ↓
Was an old value restored?
  Yes → Restoration succeeded; diagnose synchronization freshness.
  No  ↓
Does another SDK client perform the evaluation?
  Yes → Correct client registration and lifecycle.
  No  → Collect sanitized diagnostics; do not delete fleet-wide cache.

Common misconceptions

• “If a cache path is configured, a valid cache must exist.” False — a snapshot must first be synchronized and persisted.
• “If a cache file exists, restoration succeeded.” False — it may be unreadable, invalid, incompatible, or wrong-scope.
• “An old value proves the cache was not restored.” False — often restoration succeeded and sync has not refreshed.
• “Process restart and Pod replacement have the same storage behavior.” False.
• “Container-local storage is persistent storage.” False for most replacement lifecycles.
• “Deleting the cache is the first troubleshooting step.” False — removes evidence and may remove the last usable snapshot.
• “Read permission is sufficient.” Not necessarily — writes/replace/rename may also be required.
• “All replicas should share one cache.” False — isolate active writers.
• “Restarting repeatedly repairs a missing volume.” False.
• “AsyncStorage is always available in React Native.” False — depends on deps, native integration, and init.
• “A mobile reinstall should preserve the cache.” False — app-private storage is commonly removed.
• “Persistent cache guarantees the latest dashboard value.” False — it restores the last persisted snapshot.
• “If the application starts successfully, cache restoration succeeded.” False — it may have started on defaults.

Prevention

Production troubleshooting runbook

Incident owner:
Application:
SDK language / version:
Application version:
Deployment platform:
Project / Environment:
Flag key / expected type / expected value / actual value:
Is actual value an application default?
Process start time / lifecycle event:
Persistent cache enabled?
Cache provider / configured path or namespace / resolved path or namespace:
Storage type:
Cache exists? / modification time / size:
Directory readable / writable?
Runtime user or service account / owner:
Read-only filesystem?
Free disk / inodes / quota:
Volume mounted? / PVC bound? / Pod rescheduled?
Previous synchronization succeeded? / last successful synchronization:
Previous cache write succeeded?
Cache restoration attempted / result / cache age:
SSE connected after startup?
API key recently rotated? / SDK recently upgraded?
Multiple writers use this location?
Mobile storage provider available? / app data recently cleared?
Root cause / Resolution:
Online synchronization verified?
Offline restart verified? / second restart verified?
All affected replicas verified?
Completion time / Retrospective link:

Do not include full API keys, authorization headers, raw cache contents, or sensitive JSON flag values.

Escalation data

Collect SDK language/version, application version, OS/mobile platform, container/orchestrator, cache provider, configured and resolved location, lifecycle event, runtime identity, sanitized permissions, cache metadata, capacity/quota, volume events, restoration and write diagnostics, startup timeline, last successful sync, SSE state, whether a default was returned, recent deployment/SDK/key changes, and results of controlled online and offline restart tests. Never include full API keys, auth headers, unrelated customer data, sensitive payloads, or raw cache data unless explicitly requested through an approved secure process.

Relationship to other SOASAP concepts

Related documentation

• Persistent Cache
• Cache Strategy
• Non-Blocking Startup
• Default Values
• Offline Operation
• Local Evaluation
• Real-Time Synchronization
• Flag Not Updating
• SSE Disconnected
• API Keys
• Invalid API Key
• Outages
• High Availability
• .NET Persistent Cache
• Node.js Cache
• Python Cache
• React Native SDK
• React Native AsyncStorage Cache
• Kotlin Persistent Cache