Proxy Core

Ghost’s proxy is the heart of the application — a custom Man-in-the-Middle (MITM) HTTP/HTTPS proxy built on Go’s net/http. Every piece of network traffic that Ghost captures flows through this proxy. It handles plain HTTP requests, HTTPS via CONNECT tunneling with TLS interception, WebSocket upgrades, script injection, network throttling, and feeds everything into the interceptor pipeline for breakpoints, map rules, addons, storage, and security analysis.

“Man-in-the-Middle” sounds alarming, but it’s exactly what debugging proxies do — Ghost sits between your browser/app and the server, reading and optionally modifying traffic in both directions. This is how tools like Charles Proxy, Fiddler, and mitmproxy work too. The difference is that Ghost adds AI-powered analysis on top.

Architecture

What this diagram shows — how the proxy routes every request:

The proxy’s ServeHTTP method is the entry point for all traffic. It makes one decision based on the HTTP method:

• CONNECT requests (all HTTPS traffic) go to handleConnect, which decides whether to perform TLS interception (MITM) or pass the traffic through as a raw TCP tunnel (for SSL-bypassed hosts). If intercepting, it enters a keep-alive loop that reads HTTP requests from the decrypted TLS connection, checking each for WebSocket upgrades.
• All other methods (GET, POST, PUT, etc.) are plain HTTP requests that go directly to handleHTTP.

Both paths feed into the same interceptor pipeline and use the same adaptive transport for upstream connections.

Request Lifecycle

Plain HTTP (handleHTTP)

When a plain HTTP request arrives (not tunneled through CONNECT), it follows this 16-step process:

1. Control page intercept — Ghost checks for special URLs before any proxy logic:

   • /ghost-ca.crt on any host → serves the CA certificate as a PEM download
   • /ghost-profile.mobileconfig on any host → generates an iOS Configuration Profile with the CA cert embedded (DER-encoded, base64)
   • Host: ghost.proxy → serves the landing page, or reverse-proxies /api/v1/script/fetch and /api/v1/script/analyze to the API server (for the __ghost helper in injected scripts), or returns connectivity verification at /verify

2. Host extraction — takes the hostname from the URL or Host header, strips the port, lowercases

3. Host filtering — shouldCapture(host) checks the include/exclude lists. Excluded hosts are forwarded directly without creating a flow (no capture, no interceptors, just pass-through)

4. Flow creation — generates a ULID, attaches the current session ID, records StartedAt timestamp, sets source to "proxy"

5. Request capture — reads the request body with io.LimitReader (capped at 10 MB — larger bodies are truncated for storage but forwarded in full), clones all headers, reconstructs the full URL if it was relative

6. Client metadata — records the client’s IP address (from RemoteAddr) and identifies the source application from the User-Agent header

7. Device resolution — a 3-layer system identifies which device sent the request:

   • Layer 1 (User-Agent classification): parses the UA string to classify as mac, ios_simulator, ipad_simulator, android_emulator, ios_device, android_device, or unknown
   • Layer 2 (Simulator registry): on macOS, a background goroutine polls xcrun simctl list devices booted every 15 seconds to track running iOS simulators
   • Layer 3 (PID resolution): when multiple simulators are running, uses lsof to find which process owns the TCP connection, then walks the parent process chain (up to 8 levels) to match against simulator process IDs. Results are cached with a 60-second TTL.

8. Request interceptors — the pipeline runs all interceptors’ OnRequest methods in order. An interceptor can return:

   • ActionDrop — silently drop the request (no response sent)
   • ActionRespond — short-circuit with a synthetic response (used by Map Local rules)
   • nil — continue to the next interceptor

9. Build upstream request — creates a new HTTP request from the captured body (since step 5 already drained the original body), clones headers, removes hop-by-hop headers, sets GetBody for retry support

10. Timing instrumentation — attaches a DialTimings struct to the request context. The dialer populates DNS, TCP, and TLS timings as each phase completes.

11. Transport RoundTrip — sends the request upstream via the adaptive transport. Records the time to first byte (TTFB).

12. Error handling — if the upstream connection fails, Ghost records the error in flow.Error, evaluates noise detection (the failed request may still be emitted), runs response interceptors, emits the flow, and sends a 502 Bad Gateway to the client.

13. Script injection check — if injection rules exist and the response has an HTML content type, Ghost buffers and decompresses the response body, injects scripts, strips CSP headers, and updates Content-Length. The original (pre-injection) body is stored in the flow.

14. Streaming response — for non-injected responses, Ghost uses io.TeeReader to simultaneously stream the response to the client AND capture up to 10 MB for storage. A limitWriter silently discards bytes after the cap, so the client always receives the full response even if the stored copy is truncated.

15. Build flow response — creates the Response struct from captured data, decompresses the stored body (gzip/deflate/brotli) for display in the inspector

16. Timing, noise, interceptors, emit:

    • Assembles the timing breakdown: TTFB = rawTTFB - DNS - TCP (no TLS phase for plain HTTP). Negative TTFB (from pooled connections where DNS/TCP are zero) is clamped.
    • Evaluates noise detection — if the flow matches a noisy pattern, it may be suppressed from storage and emission
    • Runs response interceptors (storage, security analysis, addon tagging)
    • Emits the flow via WebSocket broadcast

HTTPS CONNECT Tunnel (handleConnect)

When a browser wants to access an HTTPS site through a proxy, it sends a CONNECT request asking the proxy to establish a tunnel. Ghost intercepts this tunnel to read the encrypted traffic:

1. Normalize host — appends :443 if no port specified

2. Host filtering — uncaptured hosts get tunnelDirect() — a raw TCP relay that forwards bytes bidirectionally without any interception. Ghost acts as a transparent tunnel.

3. SSL bypass check — hosts in the SSL bypass list also get tunnelDirect(). This preserves the original TLS connection end-to-end — Ghost can’t read the traffic, but the app’s certificate pinning works normally. Bypass patterns support exact matches and wildcard suffixes (e.g., *.example.com stored as .example.com, matched via strings.HasSuffix).

4. Hijack connection — Ghost takes ownership of the raw TCP connection from Go’s HTTP server, sends HTTP/1.1 200 Connection Established to the client

5. TLS handshake (server-side MITM) — Ghost presents a dynamically forged certificate for the requested hostname (from the CertIssuer). The client’s TLS library accepts this certificate because it chains to Ghost’s CA, which you installed as trusted. If the handshake fails (the client rejects the cert), Ghost returns silently — no flow is created.

6. Request loop — Ghost reads HTTP/1.1 requests from the decrypted TLS connection in a keep-alive loop:

   • Re-checks host filters on every request (supports hot-reload — if you change filter settings mid-connection, the new settings take effect immediately)
   • WebSocket detection — if Upgrade: websocket headers are present, Ghost hands off to handleWebSocketUpgrade which takes ownership of the connection for bidirectional frame relay
   • Normal HTTPS requests — processed by handleMITMRequest, which follows the same pipeline as handleHTTP but with additional TLS timing data

7. Loop termination — when the client closes the connection (io.EOF) or a read error occurs, the loop exits

MITM Request Processing (handleMITMRequest)

This is essentially the same as handleHTTP but for HTTPS requests read from a decrypted CONNECT tunnel. Key differences:

• The URL is reconstructed with https:// scheme
• Timing includes the TLS handshake phase: TTFB = rawTTFB - DNS - TCP - TLS
• Response streaming uses chunked transfer encoding directly to the raw net.Conn (not Go’s ResponseWriter)
• On upstream failure, the error response is written directly to the connection

Adaptive Transport

What this diagram shows — how Ghost auto-detects HTTP/2 support:

The adaptiveTransport manages two underlying transports (one for HTTP/1.1, one for HTTP/2) and maintains a per-host protocol cache (host:port → true for h2, false for h1).

For a known host: Uses the cached transport directly.

For an unknown host (first request):

1. Tries HTTP/2 first
2. If successful: caches the host as h2, returns the response
3. If h2 fails: resets the request body via req.GetBody() (the body was already consumed by the failed h2 attempt), tries HTTP/1.1
4. If h1 succeeds: caches as h1, returns the response
5. If both fail: returns a combined error

Connection Pool Settings

Setting	Value	Purpose
Max idle connections	200	Total connection pool size across all hosts
Max idle per host	20	Per-host connection reuse limit
Idle timeout	90s	How long an idle connection stays in the pool
Response header timeout	60s	Max wait time for upstream response headers
TCP dial timeout	10s	Max time to establish a TCP connection
TCP keep-alive	30s	Interval for TCP keepalive probes
Expect-Continue timeout	1s	Timeout for 100 Continue response
H2 max read frame size	16 MB	Large frame support (for CDNs like Google Video)
H2 read idle timeout	30s	H2 connection idle detection
H2 ping timeout	15s	H2 keepalive ping timeout

uTLS Chrome Fingerprint

All upstream TLS connections use uTLS to mimic Chrome’s TLS ClientHello. This prevents servers from detecting that traffic comes from a proxy (some servers block non-browser TLS fingerprints).

The three-phase dial with timing:

1. DNS lookup — net.DefaultResolver.LookupHost(ctx, host) resolves the hostname to IP addresses. Records DNS duration.

2. TCP connect — net.Dialer connects to the first resolved IP with a 10-second timeout and 30-second keepalive. Records TCP duration.

3. TLS handshake — builds Chrome’s ClientHelloSpec from utls.HelloChrome_Auto, overrides the ALPN extension with the requested protocols (either ["h2", "http/1.1"] for the h2 transport or ["http/1.1"] for the h1 transport), creates a utls.UClient with HelloCustom mode. InsecureSkipVerify: true because the proxy is MITM — upstream certificate validation is intentionally skipped. Records TLS duration.

Upstream Proxy Support

When an upstream proxy is configured (for corporate environments where Ghost sits behind another proxy), the adaptive transport with uTLS is replaced by a standard http.Transport with Proxy: http.ProxyURL(proxyURL). uTLS can’t work through CONNECT tunnels (the upstream proxy terminates TLS). ForceAttemptHTTP2: true is set because Go’s stdlib can see the real *tls.Conn (not a uTLS wrapper).

Interceptor Pipeline

The pipeline runs a chain of interceptors on every flow, in this order:

Breakpoints → Map Rules → Addons → Storage → Security

Each interceptor implements three methods:

• Name() — identifier string
• OnRequest(ctx, *Flow) *Action — runs before the request goes upstream
• OnResponse(ctx, *Flow) *Action — runs after the response comes back

Action types:

Action	Value	Meaning
ActionContinue	0	Default — continue to the next interceptor
ActionDrop	1	Silently drop the flow (no response sent to client)
ActionRespond	2	Short-circuit with a synthetic response (OnRequest only — used by Map Local)

The pipeline is immutable after construction (defensive copy). Updates happen atomically via SetPipeline, so interceptor changes take effect on the next request without race conditions.

Certificate Authority

Ghost generates its own Certificate Authority (CA) on first run:

Property	Value
Algorithm	ECDSA P-256 (fast, modern, small keys)
Validity	10 years
Organization	Ghost Proxy
Common Name	Ghost Root CA
Path constraints	MaxPathLen: 0 — no intermediate CAs allowed
Serial number	Random 128-bit (per RFC 5280)
NotBefore	Backdated 1 hour (handles clock skew between devices)
Storage	~/.ghost/ghost-ca.crt (0644 permissions) and ~/.ghost/ghost-ca.key (0600 — owner-only read)

Leaf Certificate Generation

For each hostname Ghost intercepts, it dynamically generates a leaf certificate:

Property	Value
Algorithm	ECDSA P-256 (one key per leaf)
Validity	24 hours (short-lived for security)
NotBefore	Backdated 1 hour
SAN (Subject Alternative Name)	The requested hostname + wildcard for the parent domain (e.g., api.example.com gets api.example.com and *.example.com). IP addresses are supported.
OCSP stapling	Ghost generates a “Good” OCSP response signed by its CA and staples it to the leaf certificate. This prevents Windows SChannel errors about “revocation status unknown.”
TLS minimum	TLS 1.2

Performance: Leaf certificates are cached in an LRU cache with thundering-herd protection — if multiple concurrent requests need a cert for the same host, only one goroutine generates it while the others wait (single-flight pattern).

Body Handling

Capture and Storage

Request and response bodies are captured with a 10 MB limit (maxBodyCapture). Bodies larger than this are truncated for storage but always forwarded to the client/server in full.

For streaming responses, Ghost uses io.TeeReader — a Go construct that reads from one source and writes to two destinations simultaneously. The response streams to the client at full speed while a limitWriter captures the first 10 MB for the flow database.

Decompression

Stored response bodies are decompressed for display in the inspector. Ghost supports three compression formats:

Format	Library	Header Value
gzip	compress/gzip	Content-Encoding: gzip
deflate	compress/flate	Content-Encoding: deflate
brotli	andybalholm/brotli	Content-Encoding: br

On successful decompression, the stored body is replaced with the decoded version, ContentLength is updated, and the Content-Encoding header is removed. On failure, the original compressed bytes are kept unchanged.

For script injection, decompression uses a smaller 5 MB limit with a truncation guard — if the decompressed data exceeds 5 MB, Ghost checks by reading one extra byte. If data remains (truncated), it returns the original compressed body and skips injection rather than injecting into broken HTML.

Header Cleaning

Hop-by-Hop Headers

These headers are connection-level (between client and proxy, or proxy and server) and must not be forwarded:

Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization, TE, Trailers, Transfer-Encoding, Upgrade

Additional Headers Removed

Header	Why It’s Removed
Alt-Svc	Prevents QUIC/HTTP3 upgrades that would bypass the proxy entirely
Content-Encoding	Removed from stored responses after decompression (the stored body is now uncompressed)
Content-Length	Removed from upstream requests (Go recalculates it from the actual body), updated after script injection
CSP headers (4 variants)	Stripped during script injection to allow injected scripts to execute

Timing Instrumentation

Ghost captures per-request timing broken down into phases:

Phase	What It Measures	How It’s Captured
DNS Lookup	Time to resolve the hostname to an IP address	net.DefaultResolver.LookupHost timing in dialUpstreamTLS
TCP Connect	Time to establish the TCP connection	net.Dialer.DialContext timing in dialUpstreamTLS
TLS Handshake	Time to complete the TLS negotiation (HTTPS only)	utls.UClient.HandshakeContext timing in dialUpstreamTLS
TTFB	Time to first byte — server processing time	Calculated as rawTTFB - DNS - TCP - TLS. Clamped to rawTTFB if negative (pooled connections have zero DNS/TCP/TLS).
Transfer	Time to download the complete response body	time.Since(ttfbMoment) after body is fully read
Total	End-to-end request duration	time.Since(flow.StartedAt)

For plain HTTP requests, there’s no TLS phase. For pooled connections (reusing an existing TCP connection), DNS and TCP are both zero — the connection is already established.

Noise Detection

After every response is sent to the client, Ghost evaluates whether the flow is “noise” — repetitive background traffic that clutters the flow list without adding value:

1. Builds a fingerprint: METHOD:host:pathWithoutQuery
2. The NoiseDetector tracks request frequency per fingerprint using a sliding window (30 seconds, threshold of 20)
3. Hysteresis: enters “noisy” state at 20 requests/window, exits when count drops below 10 (half the threshold). This prevents rapid on/off toggling.
4. Noisy flows are either sampled (1 in 10 kept) or suppressed entirely
5. Flows with addon-applied tags are never suppressed — if something explicitly tagged a flow, it’s considered important

The client always receives the full response regardless of noise classification. Noise detection only affects whether the flow is stored and emitted via WebSocket.

Connection Lifecycle

Reset

ResetConnections() closes all idle connections on both transports and clears the per-host protocol cache. Called during:

• Proxy stop
• Network recovery (VPN reconnection, interface changes)
• Manual connection reset

Graceful Shutdown

When the proxy stops:

1. Atomically sets running to false
2. Stops the device resolver and noise detector
3. Calls httpServer.Shutdown() with a 10-second timeout — this stops accepting new connections and waits for in-flight requests to complete
4. If the timeout expires (requests still in flight after 10 seconds), forces httpServer.Close()
5. Resets connections for clean transport state

Active connections are tracked via Go’s ConnState callback with a sync.WaitGroup — new connections increment, closed/hijacked connections decrement.

Server Settings

The HTTP server uses ReadTimeout: 0 and WriteTimeout: 0 — deliberately no timeouts on reads/writes because proxy connections are long-lived (keep-alive, WebSocket, large uploads/downloads). Only IdleTimeout: 90s is set to clean up stale connections.

Constants Summary

Constant	Value	Purpose
maxBodyCapture	10 MB	Body truncation limit for flow storage
idleTimeout	90s	Keep-alive idle timeout
upstreamTimeout	60s	Upstream response header timeout
shutdownTimeout	10s	Graceful shutdown drain time
maxWSPayload	1 MB	WebSocket frame payload capture cap
maxInjectionDecompressSize	5 MB	Injection decompression bomb guard
maxLocalFileSize	50 MB	Map Local file serving cap
maxBodyScanSize	1 MB	Security interceptor body scan cap
leafCertValidity	24 hours	Dynamic leaf certificate lifetime
leafCertBackdate	1 hour	Certificate NotBefore clock skew tolerance
caValidityYears	10	CA certificate lifetime

Error Handling

Scenario	What Happens
DNS failure	Error propagated from dialer, flow gets Error field set, client receives 502
TCP connect failure	Same as DNS failure — 502 to client
TLS handshake failure (upstream)	Connection closed, flow gets error, 502 to client
TLS handshake failure (client-side MITM)	Debug logged, connection closed silently — no flow created (client rejected the proxy cert)
Upstream response timeout	After 60 seconds with no response headers, error recorded, 502 to client
Body read failure	Warning logged, partial body stored — flow is still emitted
Stream to client failure	Error recorded in flow.Error, flow is still emitted (the flow was at least partially successful)
Decompression failure	Original compressed bytes kept — display shows the raw compressed data
Injection decompression exceeds 5 MB	Injection skipped entirely, original response forwarded as-is
Pipeline is nil	Plain HTTP: forwards directly without capture. MITM: returns silently.
Hijack failure	500 Internal Server Error sent to client