SSE Disconnected
Diagnose why an SDK lost its real-time synchronization connection and how to restore configuration updates safely.
Problem
The SDK is no longer receiving real-time configuration updates from SOASAP Cloud.
Possible observations:
- dashboard changes never arrive
- flag values remain stale
- synchronization status reports disconnected
- reconnect attempts continue
- application traffic remains healthy
- cached snapshot continues working
- only one region or replica is affected
Disconnected synchronization is primarily a freshness problem, not a request-processing problem. For application-layer causes of stale values, see Flag Not Updating.
How SSE fits into the architecture
Normal flow:
If SSE disconnects:
The SDK evaluates locally. The SSE connection only distributes configuration updates. See Real-Time Synchronization and Local Evaluation.
Expected behavior
Healthy SDK:
- maintains long-lived synchronization
- receives configuration updates
- reconnects after transient network failures
- continues evaluating locally
Disconnected SDK:
- continues evaluating the last snapshot
- stops receiving new changes
- may attempt reconnection
- should not require application restart under ordinary circumstances
Local evaluation and synchronization are independent. See Offline Operation and High Availability.
Quick diagnostic checklist
Check in this order:
- Is the application healthy?
- Is only synchronization affected?
- Does the SDK still evaluate flags?
- Is the API key valid?
- Is outbound Internet connectivity available?
- Does DNS resolve correctly?
- Does TLS succeed?
- Is a firewall blocking outbound traffic?
- Is a proxy buffering or terminating SSE?
- Is only one replica affected?
- Is only one region affected?
- Has a recent deployment changed networking?
Understand the symptom
Application healthy, flags stale
Likely a synchronization or transport issue — continue this runbook.
Application healthy, defaults returned
Possible invalid key, no snapshot, missing cache, or initialization failure. Also check Invalid API Key and Cache Not Restored.
Only one replica stale
Likely per-instance networking, proxy affinity, node issue, wrong deployment secret, or local configuration drift.
Entire fleet stale
Likely firewall, DNS, proxy, authentication, or an organization-wide network issue. See Outages.
1. Verify the SDK is actually disconnected
Do not assume stale values automatically mean lost SSE. An application-side cache may mimic a synchronization problem.
Check:
- synchronization status exposed by the SDK or application diagnostics
- whether recent configuration updates arrived
- time of last successful synchronization
- application logs related to sync errors
- diagnostics endpoint, if available
- SDK health reporting, where supported
If direct SDK evaluation returns the new value while the product UI remains old, the transport path is likely healthy — switch to Flag Not Updating.
2. Check last successful synchronization
Compare current time with the last successful synchronization.
- Recent synchronization — investigate application logic or wrong scope rather than transport.
- Old synchronization — investigate networking, proxies, TLS, firewalls, and credentials.
- No synchronization ever completed — investigate initialization, authentication, or first-connect path.
3. Verify API key
Authentication failures often appear as synchronization failures.
Verify:
- key exists
- key belongs to the expected environment
- key was not regenerated without updating deployments
- deployment secret was updated
- application loaded the latest secret value
See API Keys and Invalid API Key.
4. Check network connectivity
The SDK requires outbound connectivity to SOASAP Cloud for synchronization.
Check:
- outbound routing
- Internet access
- NAT
- gateway health
- service mesh egress
- VPN paths
- private networking and cloud egress controls
5. DNS
Incorrect DNS resolution may prevent connection establishment or reconnection.
Verify:
- hostname resolution from the affected host or Pod
- DNS servers used by the runtime
- split DNS and corporate DNS filtering
- stale resolver cache
Possible symptoms:
- reconnect loops
- timeouts
- hostname not found
6. TLS
TLS negotiation failures prevent synchronization.
Possible causes:
- expired certificates
- custom trust store missing required roots
- outbound TLS inspection
- corporate SSL interception
- incorrect system clock
Verify certificate trust, clock synchronization, and any proxy-issued certificates installed on the host or container image.
7. Firewalls
Outbound firewall rules may block the synchronization connection while other application traffic remains allowed.
Check:
- cloud firewall / security groups
- corporate firewall
- Kubernetes NetworkPolicy
- outbound ACLs
- egress gateway policies
Allow outbound access required for SOASAP synchronization. Refer to official connectivity guidance for current endpoints; do not hardcode undocumented hostnames from this page.
8. Proxies
Long-lived HTTP connections behave differently from short REST requests. Proxies are a common root cause of SSE disconnects that leave ordinary HTTPS looking healthy.
Possible proxy problems:
- idle timeout closing quiet streams
- response buffering delaying or blocking event delivery
- forced reconnect or connection recycling
- content transformation of the response body
- request or connection limits
- gateway timeout
- transparent proxy interception
Diagnostic questions:
- Does the disconnect duration align with a known proxy idle timeout?
- Does initial sync succeed while live updates stop?
- Are only hosts behind a specific egress proxy affected?
- Did a proxy or gateway upgrade precede the incident?
9. Load balancers and gateways
Reverse proxies, ingress controllers, API gateways, service meshes, and cloud load balancers can interrupt long-lived streams.
Potential issues:
- idle timeout
- response buffering
- forced connection recycling
- header rewriting that breaks streaming
- connection limits
Review idle timeout and buffering settings for any intermediary on the outbound or outbound-via- proxy path. Exact vendor knobs vary; validate against your platform documentation.
10. Service mesh
Sidecars may influence streaming connections through retry policies, timeout policies, protocol handling, buffering, and connection limits.
Review mesh egress policies for the application workload. Confirm that long-lived outbound connections are not terminated by mesh timeouts that are appropriate for short RPC calls but inappropriate for streaming.
11. Kubernetes
Relevant Kubernetes surfaces:
- NetworkPolicy
- Ingress / gateway (when used for egress or shared proxies)
- service mesh sidecars
- Pod and node networking
- cluster DNS
- cluster egress / NAT
- per-node CNI or host-route failures
Only Pods on one node may disconnect. Compare healthy and unhealthy Pods by node, CNI status, DNSConfig, NetworkPolicy, and secret mounts before blaming the entire cluster.
12. Cloud networking
In AWS, Azure, Google Cloud, and similar platforms, review VPC egress, NAT gateways, outbound route tables, private networking restrictions, and any private endpoint or egress filter that may affect SOASAP Cloud reachability.
A NAT or regional egress failure can disconnect an entire availability zone or region while other regions remain synchronized.
13. Corporate networks
Enterprise environments frequently introduce proxy inspection, SSL interception, outbound filtering, idle termination, and transparent gateways. These may interrupt streaming while ordinary short HTTPS requests still succeed.
Reproduce from a controlled host outside the corporate path when operationally allowed, then compare with an affected production host.
14. Only one replica disconnected
Possible causes:
- node or host networking issue
- unhealthy Pod / process
- stale or missing secret on that instance
- proxy affinity or local proxy agent
- DNS resolver difference
- firewall exception missing on that path
- corrupted local state that blocks reconnect (less common than transport)
Compare the disconnected replica with a healthy peer: node, IP path, DNS answers, secret hash, application version, and sync error category.
15. Only one region disconnected
Regional routing differences may isolate one deployment.
Investigate:
- regional firewall / security group drift
- DNS views that differ by region
- NAT or egress gateway health
- routing table changes
- cloud-region outages affecting outbound Internet
- partial rollout of network policy
One healthy region does not prove global recovery.
Reconnect behavior
SDKs generally attempt to recover after transient failures. Temporary disconnects are expected on imperfect networks. Long-running disconnects, reconnect storms, or disconnects that never recover require investigation.
Exact reconnect intervals and backoff algorithms depend on the SDK and are not specified on this page. Follow SDK-specific documentation for observable reconnect signals.
Should you restart?
Restart is rarely the first fix for an SSE disconnect.
Restart may help when:
- deployment configuration changed and the process must reload it
- the secret changed and the runtime does not hot-reload credentials
- SDK initialization failed and left the client in a bad state
- infrastructure was repaired and a stuck client needs a clean start
Restart does not fix:
- firewall blocks
- proxy buffering or idle termination
- DNS failures
- TLS trust problems
- wrong or invalidated API key
Should you clear cache?
Persistent cache does not control synchronization. Deleting cache does not repair networking, proxies, or authentication.
Cache deletion is unrelated to most SSE incidents. Clearing cache during a disconnect can remove the last valid snapshot and force defaults until connectivity returns. See Persistent Cache and Cache Strategy.
Observability
Recommend monitoring:
- synchronization state (connected / reconnecting / offline)
- reconnect count
- disconnect duration
- time since last successful synchronization
- fleet convergence across replicas
- region health
- per-instance status
- authentication failure rate
Application request success rate alone is not a synchronization health signal.
Alerting
Informational
Short disconnect that recovers within the normal transient window for your platform.
Warning
Disconnect lasting beyond expected transient recovery, or elevated reconnect rates on a subset of instances.
Critical
- entire fleet disconnected
- Production disconnected while configuration changes are expected or in progress
- regional disconnect during an incident that requires kill-switch propagation
Exact thresholds depend on acceptable stale duration for your critical flags.
Failure-scenario table
| Scenario | Application | Synchronization | Root Cause | Resolution |
|---|---|---|---|---|
| Firewall blocks outbound path | Healthy; local eval continues | Disconnected / reconnecting | Egress ACL / NetworkPolicy | Allow required outbound access; verify |
| Proxy buffering | Healthy | Updates delayed or stream dies | Streaming buffered or idle-closed | Disable buffering; raise idle timeout |
| DNS failure | Healthy if snapshot exists | Cannot establish / reconnect | Resolver or hostname failure | Fix DNS; flush stale resolver cache |
| TLS failure | Healthy if snapshot exists | Handshake fails | Trust store, MITM, clock skew | Fix trust chain and clock |
| Invalid API key | May use snapshot or defaults | Auth rejected | Wrong / regenerated key | Deploy current key; reload process |
| Corporate proxy / SSL inspection | Healthy | Stream interrupted | Inspection gateway | Allowlist streaming; install proxy CA |
| One replica disconnected | Mixed results under LB | Per-instance disconnect | Node, Pod, secret, local path | Compare peers; heal or replace instance |
| One region disconnected | Regional split | Regional disconnect | Regional egress / DNS / NAT | Repair regional path; verify convergence |
| Entire fleet disconnected | Usually still serving | Fleet-wide stale | Shared egress, auth, or control plane | Treat as outage; restore shared path |
| SOASAP Cloud / path outage | Healthy with last snapshot | Cannot refresh | Upstream or transit failure | Wait/recover; use non-flag mitigations |
| Service mesh timeout | Healthy | Periodic disconnect | Sidecar RPC timeouts on stream | Adjust mesh timeout/egress for streams |
| Node networking | Healthy on other nodes | Node-local disconnects | CNI / host route / DNS | Drain or repair node; verify Pods |
| NAT failure | Healthy inside VPC | No outbound to Cloud | NAT / egress gateway | Restore NAT; confirm outbound |
Safe recovery procedure
- Confirm application request handling remains healthy.
- Record last successful synchronization and disconnect duration.
- Verify API key scope and validity.
- Verify DNS from an affected instance.
- Verify outbound networking and NAT / egress.
- Verify firewall and NetworkPolicy allow the sync path.
- Verify proxy and gateway idle timeout / buffering settings.
- Verify TLS trust and system clock.
- Compare one healthy replica with one disconnected replica.
- Restore networking or credentials as indicated.
- Confirm synchronization resumes without a mass restart.
- Confirm a known non-sensitive flag change propagates.
- Verify every region and deployment group.
Verify recovery
- ✓ synchronization connected
- ✓ dashboard updates received
- ✓ new snapshot applied
- ✓ expected flag updated
- ✓ all replicas synchronized
- ✓ all regions synchronized
- ✓ reconnect storm ended
- ✓ last successful sync is recent
Recovery is incomplete until fleet convergence is confirmed, not merely one reconnect event.
Common misconceptions
"SSE disconnect means the application is down."
False. Request handling can continue from the local snapshot.
"Restarting fixes every synchronization issue."
False. Firewall, proxy, DNS, TLS, and credential problems persist across restart.
"Deleting cache repairs SSE."
False. Cache is unrelated to most transport failures and may remove the last valid snapshot.
"Ordinary HTTPS working proves SSE works."
False. Idle timeouts and buffering affect long-lived streams differently.
"If one Pod reconnects, the fleet recovered."
False. Each instance has its own connection and must be verified.
"Disconnected SDK stops evaluating flags."
False. Evaluation continues from the last known snapshot when available.
"Proxy configuration cannot affect SSE."
False. Proxies are a frequent cause of streaming disconnects.
"Long-lived HTTP behaves like short REST requests."
False. Idle timeout, buffering, and connection recycling matter for streams.
Prevention
- ✓ Allow outbound streaming — ensure proxies and gateways permit long-lived responses on the sync path.
- ✓ Monitor disconnect duration — duration matters more than a single transient drop.
- ✓ Monitor last successful synchronization — primary freshness signal.
- ✓ Validate proxy configuration — especially after gateway or mesh upgrades.
- ✓ Document firewall and egress rules — make required outbound access explicit.
- ✓ Periodically test failover and blocked egress in staging — confirm stale- snapshot behavior is understood.
- ✓ Test proxy upgrades before production — idle timeout and buffering regressions are common.
- ✓ Isolate and alert on regional failures — do not wait for global symptoms.
- ✓ Monitor fleet convergence — one healthy replica is not recovery.
Relationship to other documentation
| Concept | Relationship to SSE disconnect |
|---|---|
| Real-Time Synchronization | How SSE delivers configuration while connected |
| Offline Operation | Evaluation continues from the last snapshot while disconnected |
| High Availability | Why request handling stays independent of the control plane |
| Flag Not Updating | Broader runbook starting at scope and application layers |
| Invalid API Key | Authentication failures that look like sync loss |
| API Keys | Credential lifecycle and rotation impact on reconnect |
| Outages | Extended sync loss and incident response |
| Persistent Cache | Restart behavior while the stream remains down |
| Cache Strategy | Why cache clear is not an SSE fix |