Profile Apps Behind Localhost Tunnels with Flamegraphs

Why profile an application under remote traffic?

Logs answer questions about events: which route ran, whether an exception occurred, which status code was returned, or how long an operation took. Metrics summarize behavior over time: request rates, latency percentiles, error counts, CPU utilization, resident memory, heap size, and garbage-collection activity. Both are essential, but neither necessarily identifies the function responsible for resource consumption.

A profiler samples or records what an application runtime is doing. A sampled CPU profile can reveal where execution time accumulates. An allocation profile can reveal which call paths create objects or account for allocated bytes. A retained-heap snapshot can show which objects remain reachable and what holds them in memory. These are related tools, but they do not answer the same question.

A flamegraph presents captured call stacks as a hierarchy. Frames stacked vertically represent call relationships, while horizontal width represents a larger share of the measured resource. In a sampled CPU flamegraph, width commonly reflects how often a frame appeared in CPU samples. In an allocation flamegraph, it may represent allocated bytes or allocation samples. The unit must be recorded with every result.

Remote traffic matters because a loopback test can omit behavior that appears when an application receives requests through a public entry point. Remote clients may use different connection reuse patterns, concurrency, payload sizes, headers, timing, protocol behavior, or cancellation patterns. Slower clients may also keep requests or responses active for longer. These differences can expose application bottlenecks that a fast localhost request does not reproduce.

With Localtonet, the client on the application host establishes an outbound connection to one of our relay servers. An HTTP tunnel provides a public HTTPS address, while a raw TCP tunnel provides a public host and port. This lets you direct controlled remote traffic to a local service without inbound router port forwarding, firewall changes for an unsolicited inbound connection, VPN setup, or a public IP address.

The useful question is therefore not whether loopback and tunneled latency are identical. They are different request paths, so some difference is expected. The useful question is whether the remote workload changes application-side behavior. If the same handler becomes hotter, allocation per request rises, lock waits increase, or garbage collection consumes more time, the profile provides evidence that the application is responding differently.

The practical implementation in this guide uses Node.js and Clinic Flame for an end-to-end CPU workflow. The experimental method remains useful for other runtimes, but profiler installation, process attachment, symbols, capture formats, and memory semantics must be checked against that runtime's primary documentation.

Build a measurement model before collecting data

Start by separating an observed request into measurable components rather than treating one duration as a single unexplained number. At minimum, record client-observed latency and application-observed processing time. If the application calls databases, storage services, local models, or upstream APIs, capture their timings separately where the application already supports that instrumentation.

Keep timestamps synchronized across the load generator, application logs, metrics system, and profile capture. A profile that cannot be aligned with the traffic window may include startup, idle maintenance, health checks, or unrelated work. Record the start and end of each run, request rate, concurrency, completed responses, errors, transferred bytes, application version, and profile type.

Also distinguish inclusive and self cost. Inclusive cost attributes a frame with work performed by its descendants. Self cost attributes work directly to that frame. A broad request handler may have high inclusive cost because it invokes expensive child functions, while a serialization routine below it may account for significant self cost.

Prepare a safe and repeatable profiling test

The complete CPU recipe below assumes a Node.js application that can be started from a known entry file, such as server.js. Substitute your real entry file and local endpoint. Before profiling, confirm the supported Node.js version for your application and review the current Node.js command-line documentation.

Prepare the following:

• A Node.js application that already starts successfully and listens on a documented local IP address and port.
• A current Node.js installation supported by the application.
• Permission to install Clinic as a development dependency and run a profiler on the host.
• A fixed request corpus, payload set, concurrency model, duration, and warm-up procedure.
• A remote load generator under your control for the public-path run.
• Application logs and metrics that can be aligned with each profile window.
• A current Localtonet client and a device-specific authentication token obtained through our platform.
• A relay server or region selected from the values currently available in the Localtonet product or dashboard.
• Enough disk space for diagnostic artifacts and a private location in which to store them.
• A rollback plan so profiling and public exposure can be stopped if the service becomes unstable.

Use synthetic or non-sensitive data whenever possible. Heap snapshots can contain application values, while CPU and allocation profiles can disclose symbols and source paths. Store artifacts outside publicly served directories, apply your normal access controls, and define how long the files should be retained.

Choose a workload that represents the behavior under investigation. A one-request smoke test can verify connectivity but rarely creates a useful profile. An uncontrolled stress test can overwhelm the host and obscure the original issue. Begin below saturation, increase load deliberately, and identify the point at which latency, errors, queue depth, allocation, or memory changes.

Define the experiment before starting it

Use at least three conditions: idle, direct local traffic, and tunneled traffic. For the primary comparison, an A/B/A sequence is stronger than a single pair:

1. A1: direct traffic to the local application address.
2. B: equivalent remote traffic through the Localtonet endpoint.
3. A2: repeat the direct condition after the tunneled test.

Keep the application build, configuration, host, dataset, warm-up, request corpus, duration, and profiler settings constant. Match request count only when requests are approximately equivalent. For streaming responses, uploads, downloads, persistent connections, or variable payloads, also match bytes transferred, connection count, active duration, and operation type.

Generate a Node.js CPU flamegraph with Clinic Flame

Clinic Flame provides a practical Node.js path because it launches the selected application command, collects CPU profiling data, and generates an interactive flamegraph report when the process stops. Launching the process through Clinic also reduces the risk of attaching to the wrong operating-system process.

This workflow covers profiler installation, application selection, startup, warm-up, traffic generation, capture shutdown, report generation, verification, and artifact handling. Run it in a development or controlled test environment first. Profiling adds overhead, and the amount must be measured for your application.

Install Clinic from the project directory:

npm install --save-dev clinic

Verify that the application is not already running on the test port, then launch the intended entry file:

npx clinic flame -- node server.js

Replace server.js with the actual entry file. Do not copy this command if the project requires a different startup command, environment, working directory, preload module, or runtime flag. Record those inputs because changing them between runs can change profile results.

After the application reports that it is ready, perform the defined warm-up and measured traffic. Use the application's existing integration test or load-generation tool where possible. The tool and request corpus should remain identical between direct and tunneled conditions. Only the destination should change.

When the measured window finishes, return to the Clinic terminal and press Ctrl+C. Wait for report processing to complete. Clinic normally produces diagnostic data in the project workspace and opens or identifies the generated report. Exact filenames can vary by version, so use Clinic's reported output rather than building automation around an assumed filename.

Validate that the capture contains the intended workload

Do not accept a report merely because it opened. Confirm all of the following:

• The application log shows the expected route or operation during the profile window.
• The completed request count and error count match the load generator's record.
• The flamegraph contains recognizable application frames rather than only idle, startup, or framework initialization work.
• The capture duration overlaps the measured workload rather than a long inactive period.
• Repeated captures contain similar major frames, even if sample percentages vary.
• The Node.js and Clinic versions, application build, configuration, and workload are recorded.

A useful CPU capture needs enough work to sample recurring code paths. If the graph is mostly idle, increase representative operations or shorten inactive time around the workload. If initialization dominates, warm up before the measured requests and keep the post-test idle period short.

Use Node's built-in CPU profile when you need a standard artifact

Node.js also provides the --cpu-prof command-line option documented in the official Node.js CLI reference. It starts the V8 CPU profiler and writes a CPU profile when the process exits. This can be useful when a standard .cpuprofile artifact fits an existing workflow.

node --cpu-prof server.js

Warm up the application, run the measured workload, and stop the process cleanly so Node can write the profile. Load the resulting profile into a compatible local viewer. Keep it private because it may contain function names and source paths. Clinic Flame remains the primary illustrated workflow in this tutorial because it directly produces the flamegraph-oriented report used for comparison.

Add a Localtonet HTTP or TCP test path

Choose the tunnel family that matches the application protocol. Use an HTTP tunnel for a local web application or HTTP API. HTTP and File Server tunnels support Random Sub Domain, Custom Sub Domain, or Custom Domain process types, and the HTTP options serve content at a public HTTPS address. Check current documentation before applying exact custom-domain DNS settings.

Use a raw TCP tunnel when the application protocol should pass as a TCP stream rather than through HTTP handling. The TCP tunnel points to the local IP address and port reachable from the client device and provides a public host and port.

Neither option should be described as a VPN. HTTP and TCP tunneling publish a selected service. VPN Manager is our separate private mesh VPN feature.

The following is our maintained common tunnel lifecycle. It explains the sequence shared by Localtonet workflows, but it is not a substitute for every protocol-specific dashboard field. Use the current dashboard and the relevant protocol documentation for the exact options available in your account and client version.

For confirmed HTTP-specific fields and setup details, use our Localtonet HTTP tunnel documentation. For TCP, use the current Localtonet documentation navigation and dashboard rather than inferring fields from the HTTP workflow.

Run the direct and tunneled CPU captures

For the direct condition, start the application through Clinic Flame and send the fixed workload to its local address. Stop the process, save the report, and label it direct-a1.

For the tunneled condition, start a fresh equivalent Clinic run, start the Localtonet tunnel, and send the same workload from a remote system under your control to the assigned public endpoint. Stop Clinic after the request window, save the report as tunneled-b, and stop the tunnel when it is no longer needed.

Finally, repeat the direct condition and save it as direct-a2. If A2 resembles B more than A1, elapsed time or accumulated state may be affecting the result. Possible causes include cache growth, thermal throttling, a dependency slowdown, background work, garbage-collection state, or a queue that did not fully drain.

Separate allocation profiling from retained-heap analysis

A generic “memory profile” is not enough to diagnose every memory problem. Use the profile type that matches the question:

Node.js allocation profiling

Node's --heap-prof option starts the V8 heap profiler and writes a heap profile when the process exits. It is useful for investigating allocation behavior, not for proving that every sampled object remains retained.

node --heap-prof server.js

Use the same sequence as the CPU test: start the process, warm it up, run one defined workload, stop it cleanly, and preserve the generated .heapprofile artifact. Open the file in a compatible local memory-profile viewer. Record the profile's displayed unit before interpreting width or percentages.

Compare allocated bytes or samples per completed operation where the tool supports that calculation. Common paths worth investigating include body buffering, response construction, serialization, decompression, parsing, logging, regular expressions, image processing, duplicate copies, and per-request caches. A hot allocation path may be legitimate, so verify call counts and object lifetime before changing it.

Node.js retained-heap snapshots

A retained-heap investigation needs snapshots taken at comparable lifecycle points. Node supports writing a heap snapshot in response to a configured signal. On operating systems that support SIGUSR2, a controlled test can launch the application with the heap-snapshot signal enabled:

node --heapsnapshot-signal=SIGUSR2 server.js &
APP_PID=$!

The shell variable records the process identifier of the Node.js process just started. This explicit selection helps avoid signaling an unrelated Node process. After the application reaches the chosen observation point, request a snapshot:

kill -USR2 "$APP_PID"

After the snapshot has been written, stop the application cleanly:

kill -INT "$APP_PID"
wait "$APP_PID"

Compare snapshots only when they represent similar conditions, such as after the same number of workload cycles and the same idle period. Inspect object types, dominators, and retaining paths rather than relying only on total heap size. The Chrome DevTools heap snapshot documentation explains the snapshot views used to investigate retained objects.

Process resident memory, V8 heap size, allocated bytes, and retained objects are not interchangeable. A runtime may reserve memory without returning it immediately to the operating system. Native modules, buffers, code pages, and memory-mapped files may sit outside the JavaScript heap. A cache may intentionally remain warm. Capture the same signals at the same points in each run: before traffic, near peak load, immediately after traffic, and after a comparable idle period.

Contention and waiting require different evidence

A CPU flamegraph may not explain an application that is slow while CPU utilization remains moderate. The process may be waiting on database connections, file operations, upstream responses, internal concurrency limits, worker threads, promises, or event-loop work.

Correlate the profile with event-loop delay, active requests, queue depth, dependency timings, and connection-pool use. If throughput plateaus while waiting requests grow, determine which finite resource is exhausted. Do not infer network latency merely because no wide CPU frame appears.

How to interpret CPU and memory flamegraphs

A flamegraph is a map of captured call stacks. Each rectangle is a frame. Frames stacked vertically show call relationships. Horizontal width represents the proportion of the measured resource attributed to stacks containing that frame. Horizontal position generally does not represent chronological order, so a flamegraph should not be read from left to right as a request timeline.

Brendan Gregg's flamegraph documentation provides the foundational explanation of stack aggregation and frame width. Viewer behavior can differ, so also use the documentation for the profiler that created the artifact.

Start with the widest recurring frames

Wide frames identify major resource consumers within that capture. Zoom into them and inspect their descendants and callers. A wide request handler may simply be the parent of all request work. Continue until you locate specific functions with meaningful self cost, repeated descendants, unexpected recursion, or calls occurring more often than the design requires.

Use a table, top-function, or bottom-up view when labels are difficult to read. Sort by self and inclusive samples when available. A function with large inclusive cost coordinates expensive work. A function with large self cost performs expensive work directly.

This approach is also visible in Cloudflare's Workers profiling case study, which recommends capturing repeated profiles, inspecting the widest functions, using table views, and checking source maps. That article concerns Cloudflare Workers and Durable Objects. Its Workers-specific commands, dashboard workflow, and runtime behavior are not instructions for profiling a conventional localhost Node.js process.

Look for repeated and avoidable work

Profiles often reveal correct work performed too many times. Examples include serializing the same data twice, recomputing metrics, parsing configuration per request, traversing an object tree recursively from multiple levels, compiling patterns repeatedly, or rebuilding immutable response fragments. Verify call counts with counters, traces, or narrowly scoped logging before changing behavior.

A wide frame is not automatically waste. Compression, cryptography, parsing, model inference, image transforms, and copying response bytes may be necessary. Ask whether the work can be reduced, reused, batched, bounded, moved out of the request path, or performed with a more suitable algorithm without changing required behavior.

Interpret allocation width and retained size correctly

In a memory visualization, width may represent allocated bytes, allocation samples, object count, shallow size, or retained size. These meanings are not equivalent. A path that creates many short-lived objects can dominate an allocation profile but contribute little to a later heap snapshot. A smaller number of long-lived objects can dominate retained memory while barely appearing in a short allocation capture.

Identify the unit before making a claim. Then inspect object type, allocation stack, retaining path, and lifetime when supported. Ask whether an object belongs to a request, cache, queue, connection, global collection, event subscription, callback, or native resource. A demonstrated ownership path is stronger evidence than one wide bar.

Account for symbols, generated code, and source maps

Optimized runtimes can inline functions, merge frames, or show runtime internals around application frames. Native extensions may appear as unresolved symbols when suitable symbol information is unavailable. Transpiled, bundled, or minified JavaScript may require source maps for readable source-level names, depending on the profiler and viewer.

If symbols are missing, do not guess which application function a frame represents. Preserve the exact application build, generated output, and mapping artifacts required by the profiling tool. Confirm that source maps correspond to the captured build, then repeat or reprocess the profile using the tool's documented procedure.

Compare localhost and tunneled results without blaming the wrong layer

Compare equivalent runs in a fixed order. Start with completed requests, errors, payload volume, and achieved throughput. Next compare client-observed latency and application-observed duration. Then compare CPU utilization, profile composition, allocation, garbage collection, live memory, event-loop behavior, queues, and dependency timing.

Normalize results when run durations or operation counts differ. A function can accumulate more samples simply because one run lasted longer or completed more work. Useful normalizations include CPU samples or time per completed operation, allocated bytes per request, allocations per message, and memory retained after an equal number of workload cycles. Use only units supported by the profiler and metrics you actually collected.

Interpret the A/B/A sequence

Compare A1 with B, then compare A1 with A2. If both direct runs resemble one another and B differs consistently, the request path or remote workload characteristics may be associated with the change. If A2 resembles B, time-dependent state is a stronger explanation. Alternate and repeat important conditions to establish normal variation.

Verify every optimization

After changing code, repeat the same matrix. Confirm that the target frame shrank in absolute or normalized terms, not merely as a percentage because another function became larger. Check throughput, tail latency, errors, allocation, retained memory, and correctness.

Keep concise experiment records containing the hypothesis, application version, environment, workload, profile settings, artifacts, metrics, outcome, and decision. This prevents repeated investigation of the same frame and makes regressions easier to recognize.

Troubleshooting profiling and tunnel test anomalies

Clinic does not generate a useful report

Confirm that the application was launched through the npx clinic flame -- command and that it handled requests before being stopped. Allow Clinic to finish processing after the interrupt. Check that the current user can write to the project directory and that sufficient disk space is available. If startup fails, first run the underlying Node command without Clinic and resolve the application error.

The flamegraph is empty or mostly idle

Verify that requests reached the profiled application instance. Align the active profile interval with the load window and increase the number of representative operations rather than adding idle capture time. If the application delegates work to workers or child processes, determine which process performs the expensive operation and whether the selected profiler captures it.

Startup dominates the CPU flamegraph

Keep startup and warm-up procedures identical across conditions, then shorten the measured session after warm-up. If startup itself is not the subject of the test, do not compare one cold run with one warm run. Generate enough steady-state traffic that recurring request work is visible.

Function names are missing or difficult to recognize

Confirm that the captured application build matches its source maps and generated files. Bundled or minified output can obscure names. Native work may appear without useful symbols. Do not attribute an unresolved frame by guesswork. Correct the symbol or source-map setup and repeat the capture.

The tunnel exists but the public endpoint is unavailable

Creating a tunnel does not start it. Confirm that the selected Localtonet client or device is connected and that the tunnel is running. Verify that the configured local IP address and port are reachable from that device. Test the application locally before investigating the public path.

HTTP works locally but behaves differently remotely

Compare the method, path, host handling, headers, authentication, content type, body, redirects, and response compression. Ensure the remote load generator is not changing the request or following redirects differently. Inspect application logs for the exact route and status. Do not disable authentication merely to simplify the benchmark.

A TCP service fails only from the remote generator

Confirm that the client is connecting to the assigned public host and port and speaking the exact application protocol. Compare connection lifetime, handshake, message framing, and byte sequence with the local test. A successful TCP connection does not prove that the application-level exchange is valid.

CPU is high but no application frame is obvious

Look for V8, garbage-collector, kernel, native-library, or unresolved-symbol frames. Compare process CPU with machine-wide CPU so another process is not mistaken for the application. Determine whether worker threads or child processes perform the work. Resolve symbol limitations before assigning the cost to application code.

Memory remains high after traffic stops

Identify which measurement remains high. Resident memory can stay elevated because Node or native libraries retain reserved pages even after live heap falls. Compare JavaScript heap, external memory, buffers, native allocations, caches, active connections, queued work, and process resident memory separately. Repeat equal load and idle cycles to determine whether memory stabilizes or grows without a bound.

Heap snapshots fail or destabilize the application

The host may not have enough free memory to create the snapshot safely, or the process may be too busy to tolerate the pause. Reproduce the issue with a smaller controlled dataset, capture on a test host with sufficient headroom, and avoid requesting several snapshots simultaneously. Do not treat a failed production snapshot as permission to repeat the operation without assessing the risk.

Profiling changes the result too much

Compare a profiled and unprofiled local baseline. Shorten the capture, collect one profile type at a time, and reduce unrelated logging or tracing. Avoid collecting CPU, allocation, snapshots, verbose request logs, and full distributed traces simultaneously unless the host can tolerate their combined overhead.

Remote latency varies significantly between runs

Repeat tests from the same remote system and keep the same Localtonet relay selection. Record whether connections are new or reused. Keep the application host free from unrelated CPU, disk, and memory pressure. Report a latency distribution, including a median and relevant tail percentiles, rather than relying on one request or a single average.

Frequently asked questions