Noise Detection

When you’re debugging a web application, a huge portion of the network traffic you see isn’t meaningful — it’s background noise. Browsers constantly poll for notifications, analytics scripts send tracking data every few seconds, CDN servers fetch content chunks in bursts, and heartbeat endpoints ping the server to check if it’s alive. If Ghost stored and displayed every single one of these repetitive requests, your traffic list would be flooded with thousands of identical entries, making it nearly impossible to find the actual API calls you care about.

Ghost’s noise detector watches for these repetitive patterns and automatically suppresses them. It doesn’t block the traffic — your browser still gets every response it expects — it simply stops storing the repetitive requests in Ghost’s database and stops showing them in the traffic list. For patterns it does suppress, it keeps occasional “sample” flows so you can still see what the pattern looks like, along with a count of how many similar requests were hidden.

How It Works

What this diagram shows — the complete decision tree for every HTTP response:

When a new flow arrives (after the response has already been sent to the client), Ghost first checks if noise detection is enabled at all — if not, the flow is kept without any further checks. Next, it checks whether the flow has been tagged by any addon scripts — tagged flows are always kept because an addon applied business logic to them, meaning someone decided this flow matters. If neither of those early exits applies, Ghost generates a fingerprint (a unique identifier based on the HTTP method, hostname, and URL path) and looks up whether this fingerprint has been seen before. It trims old timestamps outside the 30-second window, then decides: if this pattern isn’t currently marked as noisy, it checks whether the count within the window has reached the threshold (20). If it has, the pattern enters the noisy state and emits the first “sample” flow. If the pattern is already noisy, it first checks the hysteresis exit condition (fewer than 10 requests in the window — half the threshold) to see if the pattern has calmed down. If not, it checks whether enough requests have passed since the last sample (10, the sample rate) to emit another sample. Otherwise, the flow is suppressed — it never enters the database, never appears in the UI, and never triggers a WebSocket broadcast. The client always gets its response regardless.

Fingerprinting

Every HTTP request gets reduced to a simple fingerprint — a short string that identifies the “pattern” of the request. Two requests to the same endpoint count as the same pattern, even if they have different query parameters, headers, or request bodies.

Format: METHOD:lowercase_host:path_without_query

The fingerprint is built from three pieces:

1. HTTP method — used as-is (e.g., GET, POST). Since HTTP methods are uppercase by convention, the fingerprint preserves that.
2. Hostname — lowercased to prevent case mismatches (e.g., Example.Com becomes example.com).
3. URL path — everything before the ? query string. If the URL is /api/heartbeat?ts=12345, only /api/heartbeat is used. The path is NOT lowercased (paths are case-sensitive on most servers).

Examples:

Method	Host	Path	Fingerprint
GET	Example.Com	/api/v1/users	GET:example.com:/api/v1/users
POST	api.test.com	/chat?token=abc	POST:api.test.com:/chat
GET	HOST.COM	/path?a=1&b=2	GET:host.com:/path
DELETE	x.com	/	DELETE:x.com:/

Why this design: Query parameters are stripped because many repetitive endpoints use unique parameters per request (timestamps, nonces, pagination tokens). If the fingerprint included query parameters, every request to /api/poll?ts=1234 and /api/poll?ts=1235 would look like different patterns, defeating the noise detection. Similarly, request bodies and headers are ignored because they change between requests even when the endpoint is the same.

The fingerprint is built using Go’s strings.Builder with a pre-allocated buffer sized to len(method) + 1 + len(host) + 1 + len(path) for efficiency — this avoids memory allocations during string concatenation, which matters since fingerprinting runs on every single response.

Verdicts

Every flow that passes through noise evaluation receives one of three verdicts, which determines what happens to it inside Ghost (the client always receives its response regardless):

Verdict	What It Means	Stored in Database?	Shown in Traffic List?	Broadcast via WebSocket?
NoiseKeep	Normal traffic — this request is not part of a repetitive pattern, or noise detection is disabled, or the flow was tagged by an addon	Yes	Yes	Yes
NoiseSample	This request IS part of a repetitive pattern, but it’s being kept as a representative sample so you can still see what the pattern looks like	Yes, with noise metadata attached	Yes, with an amber ~N badge showing how many similar requests were suppressed	Yes
NoiseSuppress	This request is part of a repetitive pattern and is being silently dropped — Ghost won’t remember it existed	No	No	No

What happens with each verdict in the proxy pipeline:

When a flow gets NoiseKeep or NoiseSample, the proxy continues normally — the response pipeline runs (which includes the storage interceptor that saves the flow to SQLite, the security interceptor that scans for vulnerabilities, and the addon interceptor), and then emitFlow() broadcasts the flow to all connected WebSocket clients so it appears in the traffic list.

When a flow gets NoiseSuppress, the proxy skips the entire response pipeline — ProcessResponse() is never called (so no interceptors run), and emitFlow() is never called (so no WebSocket broadcast). Since the storage interceptor never runs, the flow is never saved to SQLite. The flow simply disappears from Ghost’s perspective, as if it never happened.

For NoiseSample flows, two metadata fields are attached to the flow before it enters the pipeline:

• _noise_group — the fingerprint string (e.g., GET:api.example.com:/heartbeat), so the frontend knows which pattern this sample represents
• _noise_suppressed — a string number (e.g., "9") indicating how many flows were suppressed since the last sample. This is only attached when the suppressed count is greater than zero.

Sliding Window Algorithm

The noise detector uses a sliding window approach to count how many times each pattern appears within a time period. Instead of counting “requests per second” or using fixed time buckets, it tracks the exact timestamp of each request and counts how many fall within the most recent N seconds.

The Data Structure

For each unique fingerprint, the detector maintains a pattern state — an internal record that tracks:

Field	Type	Purpose
timestamps	List of times	Every request timestamp within the current window. Used to count how many requests happened recently. Pre-allocated to hold threshold + 1 entries (21 by default).
noisy	Boolean	Whether this pattern is currently classified as noisy. Once true, it stays true until the hysteresis exit condition is met.
sinceLastKept	Counter	How many requests have been seen since the last sample was emitted. When this reaches the sample rate (10), a new sample is emitted and the counter resets.
suppressedCount	Counter	How many requests were suppressed since the last sample. This number is reported to the frontend when a sample is emitted, so the UI can show “~9 similar requests suppressed.”
lastSeen	Timestamp	When the most recent request with this fingerprint arrived. Used by the cleanup goroutine to garbage-collect stale patterns.

Window Trimming

Every time a new request is evaluated, the detector trims old timestamps from the pattern’s list. Any timestamp older than now - WindowDuration (default: 30 seconds ago) is removed. This means the list always reflects only the most recent activity.

To prevent memory leaks (Go slices retain their backing array even when you slice them down), the detector allocates a new slice when trimming, copying only the remaining timestamps. The new slice has a capacity of max(threshold + 1, remaining_count) to minimize future allocations.

Growth Cap

When a pattern is already noisy, timestamps are still recorded (to track the sliding window for the hysteresis exit check), but the list is capped at threshold * 2 entries (40 by default). If the cap is reached, the last entry is overwritten instead of appending a new one. This prevents memory growth for extremely high-frequency patterns (imagine a heartbeat endpoint firing 100 times per second — without the cap, the timestamp list would grow to 3,000 entries in the 30-second window).

Hysteresis

Hysteresis is a mechanism that prevents rapid oscillation between states. Imagine a pattern that sends exactly 20 requests per 30 seconds — right at the default threshold. Without hysteresis, the detector would classify it as noisy, then as soon as one request exits the window it would drop to 19 and classify it as normal, then the next request would push it back to 20 and classify it as noisy again. This flip-flopping every few seconds would be confusing and produce erratic behavior.

Ghost solves this with asymmetric thresholds:

• Entry threshold — the pattern must reach 20 requests within the window to enter the noisy state (the configured Threshold value)
• Exit threshold — the pattern must drop below 10 requests within the window to exit the noisy state (Threshold / 2, using integer division)

This creates a “dead zone” between 10 and 20 where the detector maintains whatever state it’s already in. A pattern that’s been classified as noisy stays noisy until it calms down significantly (drops to less than half the threshold), and a pattern that’s been classified as normal stays normal until it becomes genuinely repetitive (reaches the full threshold).

On exit from noisy state: When a pattern exits the noisy state (drops below Threshold / 2), the detector checks if any flows were suppressed since the last sample. If so, it emits one final NoiseSample with the suppressed count, so the frontend gets an accurate total of how many flows were hidden. If no flows were suppressed, it simply returns NoiseKeep.

Tagged Flow Exception

Flows that have been tagged by addon scripts are never suppressed, regardless of how frequently their pattern appears. This check happens early in the evaluation — before fingerprinting or any window calculations.

Why: Addons represent intentional business logic. If someone wrote an addon that tags certain API calls (e.g., ghost.onResponse(flow => { if (flow.request.path.includes('/checkout')) flow.tag('checkout') })), those tagged flows are important to the user. Suppressing them because they happen to be frequent would silently hide flows that the user explicitly asked to track.

The check is simple: the Evaluate method receives a hasTags boolean parameter. If true, it returns NoiseKeep immediately.

Configuration

Configurable Settings

Setting	TOML Field	Default	UI Exposed?	Description
Enabled	noise_enabled	true	Yes (toggle)	Master switch. When disabled, all flows get NoiseKeep immediately. Uses a pointer type (*bool) in config so nil (unset) defaults to true while explicit false is respected.
Threshold	noise_threshold	20	Yes (numeric input labeled “Repeat count”)	Number of requests within the time window before a pattern is classified as noisy. The exit threshold is automatically computed as threshold / 2 (integer division).
Window duration	noise_window_secs	30	Yes (numeric input labeled “Time window (seconds)“)	The sliding window size in seconds. Only requests within the most recent N seconds are counted.
Sample rate	noise_sample_rate	10	No (TOML only)	After a pattern enters the noisy state, keep 1 out of every N requests as a representative sample. The other N-1 requests are suppressed. Not exposed in the settings UI — uses the default unless explicitly set in ~/.ghost/config.toml.

Internal Settings (Not Configurable)

Setting	Default	Description
Cleanup interval	5 minutes	How often the background cleanup goroutine runs to garbage-collect stale patterns. Set once at startup and not re-read on reconfiguration.
Stale after	5 minutes	How long a pattern must go unseen before it’s eligible for cleanup. This value IS re-read on each cleanup run (it reads from the current config under the lock), but there’s no way to change it via the config file or API.

TOML Example

[proxy]
noise_enabled = true
noise_threshold = 20
noise_window_secs = 30
noise_sample_rate = 10

Hot-Reload

Noise settings can be changed at runtime through the settings API — the PUT /api/v1/settings and POST /api/v1/settings/import endpoints both reconstruct a NoiseConfig from the updated configuration and call ReconfigureNoise() on the proxy server. The reconfiguration swaps the config under the detector’s mutex but preserves all existing pattern states — patterns that were already classified as noisy remain noisy with their current counters.

One caveat: the cleanup goroutine’s tick interval is read once at startup and is not updated on reconfiguration. If you could change the cleanup interval (you can’t — it’s internal), you’d need to restart Ghost for it to take effect. The StaleAfter duration, on the other hand, is read fresh on each cleanup run.

Pipeline Position

The noise detector runs after the response has been streamed to the client but before the response pipeline (interceptors + storage + WebSocket broadcast). This placement is critical:

Client ← Response ← Upstream Server
                ↓
        Noise Evaluation
        ↓              ↓
    [Suppress]      [Keep/Sample]
        ↓              ↓
     (nothing)    Response Pipeline
                  (Addons → Storage → Security)
                       ↓
                  WebSocket Broadcast

The evaluation happens in evaluateNoise() on the proxy server, which is called at multiple points in the proxy code:

• After a successful upstream response in the plain HTTP handler (handleHTTP)
• After a successful upstream response in the HTTPS CONNECT handler (handleConnect)
• After upstream errors (connection failures, timeouts) in both handlers

When the verdict is NoiseSuppress, the method returns false, and the calling code simply returns — skipping ProcessResponse() (the interceptor pipeline) and emitFlow() (the WebSocket broadcast). Since the storage interceptor is part of the pipeline, suppressed flows are never written to SQLite.

Cleanup Goroutine

A background goroutine runs on a timer to garbage-collect patterns that haven’t been seen recently. Without this, every unique URL pattern Ghost ever sees would accumulate in memory forever — browsing 10,000 different pages would create 10,000 pattern states that would never be cleaned up.

How it works:

1. Started automatically when the NoiseDetector is created via NewNoiseDetector()
2. Uses time.NewTicker with the CleanupInterval (5 minutes by default)
3. On each tick, locks the detector’s mutex and iterates over all tracked patterns
4. Any pattern whose lastSeen timestamp is older than now - StaleAfter (5 minutes by default) is deleted from the map
5. Listens on a stopCh channel — when Stop() is called, the goroutine exits

Stop safety: The Stop() method uses a select with a default case to close stopCh without panicking if it’s already closed (calling close() on an already-closed channel panics in Go). This makes Stop() safe to call multiple times.

Thread Safety

The noise detector uses a single sync.Mutex (not a sync.RWMutex) to protect all access to the patterns map and the config struct. This means:

• Only one goroutine can evaluate a flow at a time
• The cleanup goroutine waits for any in-progress evaluation to finish before running
• Reconfigure() waits for any in-progress evaluation to finish before swapping the config

The Evaluate method optimizes this slightly: it copies the config under the lock, releases the lock to compute the fingerprint (string building doesn’t need synchronization), then re-acquires the lock for the rest of the evaluation. This reduces contention when fingerprint computation overlaps with other operations.

Stats API

The detector has a Stats() method that returns a snapshot of all currently tracked patterns:

Field	Type	Description
fingerprint	string	The METHOD:host:path fingerprint
noisy	boolean	Whether this pattern is currently classified as noisy
requests_in_window	number	How many requests for this pattern are in the current sliding window
suppressed_total	number	Total number of requests suppressed since the last sample was emitted
last_seen	timestamp	When the most recent request with this fingerprint was observed

This method is available on the proxy server as NoiseStats(), but there is no REST API endpoint wired to expose it. The method exists for debugging purposes and potential future use.

Frontend Integration

Settings UI

The noise detection settings appear in the proxy settings panel with:

• A toggle switch labeled “Smart Noise Detection” — maps to noise_enabled
• Two numeric inputs shown only when the toggle is on:
  • “Repeat count” — maps to noise_threshold
  • “Time window (seconds)” — maps to noise_window_secs

The sample rate is not exposed in the UI — it always uses the default (10) unless explicitly set in the TOML config file.

Traffic List Badge

When a sampled noise flow appears in the traffic list, it shows an amber badge with ~N where N is the number of suppressed requests since the last sample. The badge appears next to the flow entry when flow.metadata._noise_suppressed is present. Hovering over the badge shows a tooltip: “Repetitive pattern — N similar requests suppressed.”

Inspector Noise Domains (Separate System)

Ghost has a completely separate noise classification system in the mobile inspector, unrelated to the traffic noise detector described above. This system classifies network traffic by domain to identify SDK noise (analytics, crash reporting, ad tracking) in the mobile inspector’s timeline and bug report features.

It uses a static list of ~50 known SDK domains (e.g., Firebase, AppsFlyer, Bugsnag, Sentry, Branch) plus 4 suffix patterns, and classifies flows as "primary" or "noise" with an optional SDK category label. This is used for UI filtering in the inspector, not for traffic suppression — all flows are still stored and visible in the main traffic list.