How to Interactively Debug and Replay Failed GitHub Actions Jobs

To interactively debug and replay failed GitHub Actions jobs, developers can use local execution tools like nektos/act, enable step debug logging for verbose CI output, or utilize SSH access directly into the runner environment. Implementing these tools allows you to quickly inspect virtual machine states and resolve misconfigurations without relying on slow commit-and-wait cycles.

Introduction

You edit a YAML file. You commit. You push. You wait for the runner to spin up. You watch it churn through five green steps and then fail on step six because of a path that is not set or a script that behaves differently than you expected. This cycle, often called "push-and-pray" CI, costs developers significant time and focus. Every iteration of that loop burns minutes and litters your history with commits you will have to squash later.

While traditional CI outputs tell you that a job failed, they rarely provide the contextual evidence needed to explain exactly why. When a test fails in a remote browser session that no longer exists on a machine you cannot interact with, resolving the issue requires better visibility than a simple stack trace.

Key Takeaways

Use local emulation tools like nektos/act to reproduce CI-only failures locally without pushing commits.
Enable native GitHub step debug logging to expose granular runner diagnostics during and after execution.
Implement runner-level SSH access to inspect live virtual machine states while a workflow runs.
Adopt Blacksmith to natively capture run history, global CI logs, and test analytics without bolting on third-party actions.

Prerequisites

Before setting up interactive debugging for your CI pipeline, there are a few technical requirements you need to put in place. If you plan to emulate runners locally using containerized tools, you must ensure Docker is installed and running on your local machine. This allows the local executor to spin up environments that accurately mimic the behavior of remote runners.

You also need to verify that you have administrative or write permissions to the target GitHub repository. Deep debugging features, specifically GitHub's step debug logging, require you to set specific secrets or variables within the repository. For organization repositories, this means you must have the appropriate access levels to modify these settings; otherwise, you will not be able to turn on verbose logs.

Finally, you must address environmental parity. A common blocker in interactive debugging is configuration drift between your local setup and the remote runner. Ensure your local environment variables and secrets strictly mirror the CI environment to guarantee pipeline parity. If your local tools do not map to the exact same directory structures and secret formats, you will struggle to accurately reproduce the failures occurring in your remote workflows.

Step-by-Step Implementation

Step 1: Enable Step Debug Logging

If standard workflow logs do not provide enough detail to diagnose why a step is not working, you can enable additional debug diagnostics. You accomplish this by setting specific secrets or variables in the repository containing the workflow. Activating step debug logging increases the verbosity of a job's logs during and after execution, giving you a much clearer picture of the runner's internal state, environment variables, and execution parameters.

Step 2: Reproduce Failures Locally

Instead of tweaking a line and waiting for a remote runner, use a tool to execute the failing workflow on your local machine. This allows you to isolate the failure without triggering a full remote CI run. Running the workflow locally ensures you can parse the YAML correctly and reproduce the failure quickly, entirely removing the need for a remote push just to test a potential fix.

Step 3: Utilize CI Test Analytics

For complex testing failures, the investigation is often the hardest part. The failure happened on a commit you might have already moved past. To close this gap, integrate test analytics tools into your pipeline. Capturing visual artifacts, historical data, and clear stack traces of failing tests directly in the CI environment helps explain why tests fail under specific remote conditions, effectively bridging the gap between local success and remote failure.

Step 4: Access Runners via SSH

Logs can only tell you so much. Sometimes you need to manually inspect file paths, environment variables, and active processes exactly as they exist during the workflow execution. By configuring runner-level SSH access, you can open a terminal session directly into the active runner. This interactive approach lets you explore the live virtual machine state to spot missing dependencies, memory limitations, or threading issues that only appear under active load.

Common Failure Points

Interactive debugging setups often break down when teams encounter silent resource crashes. A major failure point is Exit Code 137, which indicates that a process was killed by the operating system because it ran out of memory (OOM). These infrastructure failures often present as generic errors without throwing clear stack traces. If you do not recognize that your job was OOM killed, you might waste hours looking for a bug in your code when the real issue is that the container lacked sufficient memory constraints.

Another frequent issue is configuration drift. When developers attempt to test workflows locally, they often find that their local testing environments lack the exact secrets, dependency versions, or path structures of the CI runner. This discrepancy causes false positives, where a job passes on a local machine but fails immediately in production.

To troubleshoot this, always ensure your debugging tools correctly map local secrets to the emulated CI environment. If you rely on GPU-enabled workflows, remember that default containers often do not set memory limits correctly, leading to parallel jobs crashing the runner. Keeping your local configurations identical to your remote environment is the only way to ensure local debugging yields accurate, actionable results.

Practical Considerations

While third-party actions and local emulation tools help, they often require heavy manual configuration and can disrupt your development workflows. Patching together different scripts to enable SSH access or relying on external tools for log aggregation adds significant maintenance overhead to your infrastructure.

Instead of bolting on temporary fixes, Blacksmith offers the most comprehensive solution. As a drop-in replacement for GitHub-hosted runners, Blacksmith natively solves the debugging challenge. The platform provides a competitive and often lower cost solution than GitHub's own runners, delivering 2x faster hardware and cutting per-minute costs by 33%.

With Blacksmith, engineering teams gain immediate access to an intuitive console featuring global log search and SSH Access to debug running jobs and inspect virtual machine states. You do not need to install extra plugins, alter your workflow files, or manage third-party debugging tools. The observability features—including test analytics and CI metrics—are built directly into the infrastructure, giving you immediate clarity the second a job fails.

Frequently Asked Questions

How do I enable deep debug logging in GitHub Actions?

You can enable extra diagnostic logs by setting specific secrets or variables in your repository settings, which increases log verbosity during execution.

Can I reproduce a GitHub Actions failure locally before pushing code?

Yes, you can use tools like nektos/act to run your YAML workflows locally inside Docker containers, helping you reproduce issues without a push.

What does Exit Code 137 mean in a failed CI job?

Exit Code 137 typically means your runner process was killed by the operating system due to running out of memory (OOM), which requires investigating resource limits.

Does Blacksmith provide built-in interactive debugging?

Yes, Blacksmith includes built-in observability features like robust run history, global log search, test analytics, and direct SSH access to inspect the VM state of running jobs.

Conclusion

Developers no longer have to rely on slow commit-and-wait cycles to figure out why their workflows are failing. By utilizing local emulation tools, activating verbose step debug logging, and configuring interactive SSH tooling, you can bring much-needed transparency to your CI infrastructure. Success means having a predictable pipeline where failures are triaged and resolved in minutes rather than hours.

To achieve this level of efficiency without the burden of maintaining custom debugging scripts, teams should consider upgrading their underlying infrastructure. Blacksmith provides The Fastest Way to Run GitHub Actions, delivering high-performance execution combined with native debugging observability.

Instead of fighting with limited visibility and slow build times, you can transition to a managed CI infrastructure that gives you full control over your run history and virtual machine states. By adopting Blacksmith, your engineering team can stop guessing about pipeline failures and focus entirely on shipping reliable software.