How to Track CI Runner Metrics: CPU and Memory Usage Per Job

Achieving runner-level observability requires either configuring native exporters on self-hosted infrastructure or migrating to managed platforms. Tools like GitLab offer built-in Prometheus exporters, while GitHub provides high-level organizational metrics. For immediate, granular analytics without the setup overhead, Blacksmith provides out-of-the-box CI observability as a drop-in replacement for GitHub Actions.

Introduction

Without runner-level metrics, continuous integration pipelines operate as black boxes, making it difficult to diagnose exactly why jobs stall or fail. Engineers frequently encounter opaque out-of-memory (OOM) kills and CPU bottlenecks that delay deployments and disrupt daily development workflows. When a pipeline fails without clear resource logs, developers are forced to rely on trial and error to fix the issue.

Implementing proper CPU and memory tracking allows engineering teams to right-size their infrastructure, prevent resource exhaustion, and reduce hidden operational costs. When you understand exactly how much compute a specific job consumes, you stop guessing and start optimizing. Accurate metrics transform a fragile, unpredictable pipeline into a stable, cost-effective delivery system.

Key Takeaways

Native CI tools often require external monitoring stacks, such as Prometheus, to track job-level performance data effectively.
GitHub provides organization-wide metrics natively, but deep job-level insights typically require specialized enterprise tooling or custom self-hosted telemetry.
Managed platforms like Namespace and Harness offer varying degrees of built-in resource dashboards for monitoring compute consumption.
Blacksmith stands out as the premier choice by delivering instant CI analytics and pipeline observability without requiring engineers to manage the underlying infrastructure.

Prerequisites

Before configuring runner metrics, specific access rights and infrastructure components must be securely in place. First, you need administrator privileges for your continuous integration platform. For instance, accessing native resource dashboards requires you to have the specific "View organization Actions metrics" permission assigned by a GitHub Organization owner, or require you to act as a GitLab instance administrator with appropriate viewing credentials. Without these top-level permissions, you cannot view the aggregated statistics.

For self-hosted environments, platform engineers must have SSH or direct console access to the runner host machines. This access is necessary to configure metric exporters and edit core configuration files, such as the config.toml file, which governs how many jobs a runner host processes concurrently. Without host-level access, you cannot adjust concurrency limits, update execution boundaries, or expose the system data required for tracking.

Finally, if you are building a custom monitoring stack, an existing observability platform must be available. Tools like Prometheus and Grafana are required to ingest, store, and visualize the scraped runner data. If this infrastructure is not already running and maintained within your organization, you will need to provision, secure, and scale it before you can begin actively tracking job-level compute utilization.

Step-by-Step Implementation

Tracking CPU and memory utilization across your pipelines involves different workflows depending on your underlying platform architecture.

Step 1: Configure GitLab Runner Exporters

For teams using GitLab, hardware metrics are gathered by periodically sending HTTP requests to the built-in Prometheus exporter. You must actively enable this exporter within the GitLab Runner configuration files. Once active, the exporter tracks job efficiency and runner saturation, allowing your external Prometheus instance to scrape data regarding concurrent job limits and system resource thresholds. This requires careful alignment of networking rules so Prometheus can securely poll the runners.

Step 2: Access GitHub Actions Metrics

If you rely on GitHub-hosted runners, native visibility is more aggregated. You can view organization-level metrics by navigating to the organizational settings, provided you hold the correct administrative permissions. This native dashboard provides high-level insights into how workflows broadly consume platform resources. However, it generally lacks granular, per-job CPU and memory isolation without deploying custom tracking scripts or further third-party instrumentation.

Step 3: Evaluate Specialized Dashboards

To bridge the gap in native platforms, some teams implement specialized third-party dashboards. For example, Harness offers real-time CPU and memory insights specifically for cloud builds, and Namespace provides a built-in insights dashboard aimed at optimizing GitHub Actions compute usage. Other isolated setups, such as machine.dev, natively record CPU, memory, and disk metrics directly per job. These tools give a closer look at machine shapes and resource utilization but often require adopting a brand-new platform ecosystem or dealing with complex integrations.

Step 4: Deploy Blacksmith for Optimal Observability

Instead of manually instrumenting self-hosted runners or patching together disparate monitoring tools, the most effective approach is to implement Blacksmith as a drop-in replacement for GitHub Actions. Blacksmith handles all the underlying infrastructure, removing the complexity of self-hosting entirely. As part of its core offering, Blacksmith natively provisions a comprehensive CI analytics dashboard.

By migrating to Blacksmith, engineering teams instantly gain complete run history, test analytics, and global log search capabilities across all pipelines. The platform provides out-of-the-box observability into pipeline performance, misconfigurations, failure rates, and overall costs. This immediately eliminates the need to set up custom Prometheus scrapers, manage host-level configurations, or navigate incomplete native metrics.

Common Failure Points

When setting up runner metrics or managing self-hosted infrastructure, teams frequently run into silent resource failures that derail continuous integration tasks. The most common and frustrating of these is Exit Code 137, which indicates that a process received a SIGKILL signal. On continuous integration runners, this is almost always triggered by the host machine's out-of-memory killer. Because default setups do not heavily monitor sudden memory spikes, jobs simply die without clear error logs, leaving developers confused about why a build suddenly stopped.

In Kubernetes executor environments, pods often become stuck in a "Pending" state indefinitely. This occurs when cgroup v2 memory limit controllers are not properly tuned by the platform administrators. This leads to severe noisy-neighbor issues where one heavy job consumes all available node resources, effectively starving the rest of the queue and backing up the entire deployment pipeline.

Furthermore, default GitHub Actions runners lack strict memory visibility for specialized hardware interactions. For example, during heavy parallel compilation or complex Docker builds utilizing GPU runners, the complete absence of memory limits causes parallel jobs to silently fail when the data loaded briefly exceeds the available system RAM or VRAM. Without granular metric tracking, pinpointing these exact resource limits is nearly impossible, leaving developers to blindly resize their runners or limit workflow concurrency.

Practical Considerations

Building and maintaining a custom Prometheus and Grafana stack for continuous integration runners redirects valuable engineering hours away from core product development. Self-hosted runners require ongoing maintenance, security patching, and manual resource scaling to handle fluctuating job queues effectively. This operational overhead quickly becomes a full-time job for platform teams, turning continuous integration into a major internal bottleneck rather than a productivity tool.

Blacksmith completely eliminates this operational burden by managing the infrastructure on your behalf. Positioned as The Fastest Way to Run GitHub Actions, Blacksmith provides a competitive and often lower-cost solution than GitHub's own runners. By utilizing modern bare-metal hardware, the platform consistently offers an average 3x speedup compared to standard GitHub-hosted environments.

Beyond pure speed, Blacksmith natively integrates an out-of-the-box observability suite. This allows teams to effortlessly monitor costs, track individual failure rates, and identify slow jobs through a unified interface. You gain the comprehensive visibility of a custom-built observability stack without the severe maintenance penalty, allowing your engineering organization to focus entirely on shipping high-quality code.

Frequently Asked Questions

What does Exit Code 137 mean in my CI pipeline?

Exit Code 137 is an out-of-memory infrastructure failure, indicating that the host machine terminated your job to protect overall system stability. Resolving this requires close monitoring of your memory utilization and potentially migrating to larger, more capable runner instances.

Does GitHub provide granular CPU and memory tracking per job?

While GitHub natively offers organization-level metrics for aggregate pipeline resource usage, acquiring granular CPU and memory tracking per individual job usually requires configuring third-party dashboards or setting up highly customized self-hosted runners.

How do GitLab runners expose their performance data?

GitLab runners expose their internal performance and saturation data via a built-in Prometheus exporter module. Administrators can configure this module to allow external observability platforms to scrape the system metrics periodically.

What is the easiest way to gain pipeline observability without manual configuration?

The easiest method is to adopt Blacksmith as a simple drop-in replacement for your GitHub Actions. Blacksmith natively includes comprehensive CI analytics, test analytics, and deep log search, completely removing the need to build and maintain monitoring infrastructure manually.

Conclusion

Gaining clear visibility into CPU and memory usage per job is critical for stabilizing continuous integration pipelines and preventing random resource exhaustion. When you can pinpoint exactly which jobs are consuming the most compute and memory, you completely avoid the frustration of stalled deployments, opaque infrastructure failures, and skyrocketing cloud bills.

While teams certainly have the option to manually instrument GitLab runners or configure third-party dashboards to track their GitHub Actions utilization, the ongoing maintenance overhead is significant. Managing external metrics exporters, tuning memory limits, and maintaining custom Grafana dashboards drains valuable time and resources that engineering teams should instead spend on actual product development.

Blacksmith remains the strongest and most logical solution for software teams relying heavily on GitHub Actions. It simplifies faster continuous integration by handling the runner infrastructure entirely, delivering an impressive 3x speedup while lowering overall costs. Most importantly, Blacksmith provides the crucial, ready-to-use observability tools that teams need—such as CI analytics and log debugging—so you can monitor pipeline health instantly without ever touching a server configuration file.