StackConvert
How I Debug Production Issues in Laravel - hero image illustrating production debugging in a Laravel application: logs, request correlation IDs, queue job tracing, and incident investigation

How I Debug Production Issues in Laravel

Rahmat Ullah profile photoRahmat Ullah
•8 min read•Engineering Insights, Laravel, Debugging

Production bugs are different. They do not happen while you are staring at the code. They happen at 2am, under load, with incomplete logs and a customer waiting for an answer. You do not have a debugger attached. You cannot pause the system. The bug already happened. The system moved on. All you have left is evidence. The question is not "why is this code wrong" but "what was true at the moment it broke."

Production debugging is mostly investigation. The code is one suspect. The data is another. The infrastructure is a third. The hard part is uncertainty - you almost never know which suspect did it when you start.

This is where many systems stop failing because of code and start failing because nobody can explain what happened.

This is the eighth post in my Laravel series. I have written about the mistakes I stopped making, scalable application structure, performance work in production, API design, authentication and authorization, the abstractions I keep and the ones I cut, and third-party integrations. This one is the layer underneath all of them: what to do when something in production breaks and the team is staring at you for an answer.

Table of Contents

The Real Problem

Most debugging failures come from assumptions made too early.

A user reports a slow page. You assume database. You look at queries. The queries are fine. You spend an hour. The actual cause was a Redis cluster timeout on a session lookup. You never looked because you assumed.

"Works locally" means almost nothing in distributed systems. Local has one server, one database, one user, no cache pressure, no queue backlog, and no third-party flakiness. Production has all of those at once. The bug you cannot reproduce locally is not a mystery. It is the system telling you that the environment matters more than you thought.

Debugging is not guessing. It is narrowing possibilities until one survives the evidence.

Where Production Debugging Goes Wrong

The same patterns appear every time I review an incident post-mortem:

  • Random dump() and dd() statements committed under pressure, then forgotten in the codebase
  • Restarting services before understanding the issue, which clears the symptom and destroys the evidence
  • Ignoring logs because nobody trusts them - usually because nobody invested in making them trustworthy
  • Debugging symptoms instead of causes - the 500 error is a symptom; the question is what is throwing it
  • No correlation IDs, so a request through API → queue → external API → webhook → DB write is six disconnected log lines nobody can stitch back together
  • Unstructured Laravel logging that dumps strings into a file with no severity, no consistent shape, no context
  • Zero visibility into queues, jobs, or outbound HTTP calls - which is where about half of production bugs actually live

If your only signal that something failed is a customer email, you are debugging blind.

How I Approach Production Debugging Now

Start with evidence

Logs first. Metrics second. Assumptions last. If I cannot find a log line that proves my theory, the theory is not yet a theory. I write it down as a hypothesis and keep digging.

Reconstruct the timeline

What happened first? Not what the customer noticed - what the system did. A 502 at 14:03 usually has a cause at 14:01 or 14:02. I open the logs five minutes before the reported time and read forward. Most root causes appear before the alarm goes off.

Narrow the blast radius

One user or all users? One server or all servers? One queue or the entire system? This single question often eliminates half the possibilities in thirty seconds. If only one user is affected, it is data. If only one server, it is configuration. If everything is broken, it is a deploy or a shared dependency.

Trace requests across systems

Every Laravel request that crosses a boundary (queue, HTTP client, webhook, external service) gets a correlation ID. I push it into the log context and propagate it on outbound headers. Without that, debugging anything async is archaeology.

Reproduce safely

Never guess in production. Reproduce in staging with a captured payload, or replay a queue job with the same job ID into a sandbox worker. The cost of "let me just try this in prod" is too high. I learned that the slow way.

Logging

The single highest-leverage thing you can do for future debugging is to fix your logs before you need them.

Useless:

Log::error('Payment failed');

Useful:

Log::error('payment.charge.failed', [
    'request_id'        => request()->header('X-Request-ID'),
    'user_id'           => $user->id,
    'order_id'          => $order->id,
    'provider'          => 'stripe',
    'provider_response' => $exception->getResponse()?->getBody()?->getContents(),
    'attempt'           => $attempt,
]);

The structured version takes ten seconds longer to write and saves an hour the next time something breaks. It is also searchable: "find every payment.charge.failed for user 4821 in the last week" is one query in any log aggregator. The string version is grep.

I never log raw passwords, full card numbers, API secrets, or full PII payloads. If I need to track a sensitive field, I log a hash of it or the last four characters. Logs are part of the system. Treat them like one.

Queues and Async Problems

About 60% of the production bugs I have debugged in the last few years live in queues, not in HTTP requests. Laravel queue debugging is its own discipline, and the patterns are predictable:

Async systems fail differently because cause and effect are separated by time.

  • Failed jobs retrying into a loop because the failure is not transient - the third attempt makes the same broken call as the first
  • Duplicate processing because the worker crashed after the side effect but before marking the job complete
  • Race conditions between two workers picking up jobs that touch the same row in the same millisecond
  • Delayed failures where the job ran, returned successfully, but the resulting state was wrong - and you find out two hours later from a customer

The fixes are unglamorous. Idempotency keys on jobs that perform side effects. Exponential backoff with a max retry. Dead-letter queues you actually monitor. Database row locks where state matters. And logging on job start, success, failure, and retry - with the job ID in the context every time.

A Real Production Example

A team I worked with shipped a Stripe webhook handler that processed payment_intent.succeeded and marked the order as paid. It worked in staging. It worked in production for two weeks. Then customers started reporting being charged twice for a single order.

The wrong assumption: "Stripe only sends the webhook once." Stripe sends each event once on the happy path, but it retries on any 5xx, any timeout, and any non-2xx response. Our handler ran an expensive flow that occasionally took longer than the 10-second webhook timeout. Stripe would time out, retry, and the handler would run again - on an order that was now already paid.

What we tried first: adding if ($order->paid) return at the top. It sort of worked, but the race window between two concurrent executions was wide enough that both passed the check.

What actually fixed it:

  1. A unique index on (stripe_event_id) in a webhook_events table
  2. Insert the event ID first, inside a transaction, before doing any work
  3. On a duplicate-key violation, return 200 immediately - we have already processed this event
  4. Move the expensive part into a queued job so the webhook returns under one second

The fix took two hours. Finding the root cause took two days, because the original handler logged "Payment processed" with no event ID, no idempotency check, and no signal that the same event had been seen before.

The lesson was not "Stripe sends duplicates." The lesson was: if your handler is not idempotent and you do not log the event ID, you have no way to even detect that this is happening.

What I Avoid Now

  • Blind restarts before I have captured the state of the system
  • "Works on my machine" as an explanation for anything
  • Logging too little because "we'll add logs later" - later is always after the incident
  • Logging sensitive data because it is convenient in the moment
  • Debugging from memory: "I think the cache was the problem last time" - read the actual logs, not your recollection of them
  • Fixing symptoms without writing down the root cause and what would prove the same bug never recurs

Key Principles I Follow

  • Evidence before assumptions
  • Reproduce before fixing
  • Logs are part of the system, not an afterthought
  • Distributed systems fail in weird ways - timing, state, and visibility, in that order
  • Correlate everything that crosses a boundary
  • If you cannot answer "how would I know this happened again", the fix is not done

Closing Thoughts

Good debugging is not about being the smartest engineer in the room. It is about reducing uncertainty faster than the system creates it. The engineers I admire most are not the ones who guess right. They are the ones who set up the system so guessing is rarely needed.

If I could give one piece of advice to a team that wants to debug production Laravel applications faster: stop investing in your debugging skills and start investing in your logs, your tracing, and your correlation IDs. Tooling beats talent in incident response.

What was the hardest production bug you've ever debugged? I am collecting war stories for the next post in this series. The weirder, the better.