When debugging Python with gdb, is a must-have resource. Check out for more info. Breaking at labels especially useful.

Since launching we’ve had a lot of new contributors show up and add some awesome new features, like improved automated runbooks, Linear integration, and a many new detections. Come join us on our mission to deliver reliability intelligence to the world!.

Before I went to bed last night I asked @OpenAI Codex to improve test coverage for preq. Woke up with a PR ready to merge that improved the coverage to the exact amount requested. Very excited to see how much more work I can do in parallel now.

Github links (always looking for a ⭐️):. * *

Enjoyed being on @rawkode to talk about preq and CREs -- recording of the live stream (along with the amazing intro music about preq) is here

Use to get start using them in your cluster.

We've been adding more CREs for common @nats_io problems. We've been long-time users of @synadia nats and eager to contribute back.

I should make the amazing part explicit -- I've never met this person and they live in Ireland. But now, whenever I'm able to make a trip out to Ireland, I've got someone I can now meet for a beer!.

We just published several new rules to detect common Istio mesh problems. Prequel customers can find them with the "istio" tag.

Open source communities are amazing. We're all aligned around the same mission -- sharing and using reliability intelligence to fix software problems faster. Yesterday we added automated runbooks to preq. Today someone improved it by adding a Linear integration.

Live with @rawkode now!.

References. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

Download `preq` at to try out rules like these and other open source CREs (and give it a 🌟). Follow us for more content like this:. * *

PREQUEL-2025-0076. This CRE rule uses a correlation window across NGINX controller logs, Ingress configuration containing Lua code, and process OOMs (including silent worker OOM crashes) to detect the Lua Snippet Leaks problem above. rules:. - cre:. id: PREQUEL-2025-0076.

The following CRE rules were created to detect these problems. PREQUEL-2025-0071. This CRE rule uses a correlation window across NGINX controller logs, Kubernetes events, and process OOMs (including silent worker OOM crashes) to detect the CPU Cores Cause Silent Worker Crashes.

As others in the community have noted, request processing with Lua should avoid using timers [13]. After this anti-pattern was fixed, the problem was mitigated.

After enough requests with cache misses, the worker processes consumed too much memory and hit silent OOM crashes in the controller container.

An Ingress object on the controller is using an ` annotation to use a Lua script to implement an adaptive cache on requests to a specific endpoint [12]. If a request has a cache miss, it returns stale data but starts a timer to refresh the cache

Another worker memory leak was observed by PRRT in a different ingress-nginx controller configuration. The following log message was observed during the problem:

As others had done in the issues above, we added a `worker-processes` limit to our deployment configuration. This fixed the problem in production. PRRT created a new CRE rule PREQUEL-2025-0071 to detect this problem. CREs—Common Reliability Enumerations—are standardized rules.

And we observed the same errors in the ingress-nginx controller logs: