How to troubleshoot runaway (100% CPU) deno process

Deno has been running a couple of websites rock solid for many months. But during last week or so I've had the server (using Deno.serve()) go unresponsive twice. That is, connection times out and I can see 100% CPU usage for the Deno process. No idea if it's a Deno bug, or bug in my code (like a blocking infinite loop or something). Thing is, I have no stacktrace, console.log output, or error logs to work with. I only know Deno process is stuck with something, using a lot of CPU and unable to serve requests (connection times out). Problem is I don't know where it is stuck. I do push new versions out every couple of days or so, as well as update deno to latest version regularly. Both deno and my own code has changed in this timeframe so I can't really say whether this problem is related to some version of Deno, or is it related to my userland code updates. Is there any way to perhaps stop/interrupt a running Deno process from the outside and get some info what it was doing at the moment to try and find the problem? Again not sure if it's Deno internal problem, or a problem with my userland code, or a problem in some third party library. No hints in stdout/stderr. I have no idea how to reproduce the issue other than letting it run on production server and just wait perhaps days or weeks for the issue to manifest. Since this only happens rarely, on production server, obviously I'd like to run any diagnostics or tracing tools quickly to then kill deno and restart the process as soon as possible. 🤔
DAH
Deno AI Helper42d ago
I wasn't able to find the answer to your question in the official Deno documentation. However, based on the information you've provided, there are a few general suggestions I can offer, but please note that these are not based on the Deno documentation, and you should double-check the accuracy of this advice. 1. Profiling: If you suspect a performance issue, you might want to use profiling tools to monitor your Deno process. You can use the --inspect flag when starting your Deno process, which will allow you to connect Chrome DevTools and take a CPU profile to see where the CPU time is being spent. 2. Logging: Enhance logging in your application to include more verbose information about the application's state and behavior. This might help you identify patterns or specific requests that lead to the high CPU usage. 3. Graceful Shutdown: Implement a graceful shutdown mechanism in your server code that can be triggered externally. This could allow you to attempt a controlled shutdown and possibly log the current state of the application before it completely stops responding. 4. Resource Limits: Use operating system tools to limit the resources available to the Deno process. This won't solve the underlying issue but might mitigate the impact on the rest of your system and make the problem manifest more quickly for easier diagnosis. 5. Monitoring: Set up monitoring for your application that can alert you when CPU usage spikes or when the application becomes unresponsive. This can help you respond more quickly to issues. 6. Version Control: Since you update Deno and your code regularly, use version control (if you're not already) to track changes. This can help you identify which changes might be correlated with the onset of the
DAH
Deno AI Helper42d ago
issue. Remember, these suggestions are not from the Deno documentation, and you should verify their applicability to your situation. For a more specific solution, you might want to reach out to the Deno community or look for similar issues reported by other users.
R
raunioroo40d ago
Not so well versed with os signals etc. But I guess what I'm looking for, is a way to send a kill signal kind of thing from the terminal, which would stop Deno whatever it's doing, but also cause it to puke out a Javascript or Rust stacktrace. Which would help diagnose where it's stuck, is it on an infinite JS loop or what. Gonna bump this, as it seems this keeps happening roughly once or twice a day now on a live web server (deno suddenly stops responding and uses 100% CPU). I set up some additional logging and monitoring to detect this and quickly recover, but little in the ways of tools to debug and find the root cause. Log timestamps suggest there wasn't any complex/more involved requests being served at the time of the process freezing, which hints at this might be some Deno/V8 bug as opposed to javascript problems in "userland". Going with that train of thought or assumption, it might be something introduced between 1.40.0, and 1.41.1, and persisting with 1.41.2 since everything has been running rock solid before. For sure it started happening only after Temporal came out (kinda makes me suspect a little some problems with Temporal implementation... but it's a long shot). Is there any way to get some usable information out from a runaway deno process? Like, I wouldn't know what to do with a memory dump, but something like interrupting the process and getting a rust backtrace might help to know what it is/was doing when it is having problems.
L
Leokuma26d ago
More people having the same problem https://github.com/denoland/deno/issues/23033
GitHub
100% CPU usage · Issue #23033 · denoland/deno
I have a server with Deno running and every few days, the server got stuck due 100% CPU usage: Restarting the process fixes the problem. I found several bugs about Deno using 100% CPU so not sure i...
R
raunioroo25d ago
For full disclosure; i'm one of the commenters in this issue, jtoppine (noticed too that someone had similar problems and had created an issue). So I guess that makes total of two people having what appears to be the same problem. I haven't run to these freezes for a while now, everything has been running smoothly since. The cause remains a mystery, and I guess low priority now that it seems like a rare occurence. I was able to implement quick detection & recovery if it ever happens again so it's not as much as a disastrous event anymore. Still interested in potential ways to investigate if these issues happen in the future though! In the github issue it was suggested to run Deno with remote debugger enabled - A good idea, but feels like not practically feasible. (running a headless cloud VPS production server with debug hook enabled, and keeping a SSH tunnel open / remote debugger connected for days in hopes the issue reproduces. That is, if the debugger and remote connection even work when the process is in a 100% CPU usage, unresponsive state) Like, maybe it could work, but I kinda gave up trying to set it up without even trying :)
L
Leokuma25d ago
Yes. It would be nice if we could just connect the debugger via CLI at any time and stop the execution of the program. Node has a builtin CLI-based debugger https://nodejs.org/api/debugger.html#debugger
More Posts
Import SCSS in TS filesHey, I am currently building a template for web development and I want to use ESBuild as a bundler. How to modify HTTP cache folderHello, I'm using Astro framework with Deno and Directus, but after a while of the website running onDeno deployment can't connect to the Mongo Cloud, due to the Mongo Cloud Network Access Rules.Is there a way to add the Deno Deploy IP to Mongo Cloud? The Deno Depoy Regions only region names fIs there a way to pull a module from x in favour of jsr?I have ported my module `@vicary/fresh-graphql` to JSR and I would like to redirect my users go JSR In-Source Testing with deno test?Can I do something like [vitest's In-Source Testing](https://vitest.dev/guide/in-source.html) with `How to use vendoring with node_modulesAs trying to recreate a `DENO_DIR` via `deno cache` creates different results on different machines,Make an image cropper that crops from data uriHow to make an image cropper that crops to the given dimensionsBest practice for releasing a deno package on GitHubIs there a best practice for the release process of a Deno library on GitHub? I use semantic-releasedeno oak : howto force reply immediatly, and process the request later ?Bonjour, My server needs to reply an ACK (200 OK) for each request. If it did not, the client dies. Rust: Unable to bootstrap MainWorkerMy message was too long, moved it to a GitHub discussion: https://github.com/denoland/deno/discussioTailwindCSS: Class-Strings build with variablesSince TailwindCSS does not support classes with variables (like I used before with twind), how wouldImport path autocomplete doesn't workHi 👋 It behaves like it's just a string and not an import path. I have the latest version of Deno aDeno: How to import npm scoped packages with @ in deno?I am trying to import @whiskeysockets/baileys in deno like this: import * as a from "npm:@whiskeysoTo avoide tariling slash in URL on deno deployWe have used deno deploy to build website done using Astro frame work (Astrowind Theme)in productioImporting internal modulesI would like to use https://github.com/denoland/deno/blob/main/ext/console/01_console.js in my code,Strange error after lint in github actionsfor a some reason after: ```yml - name: Run linter run: deno lint ``` I get error that saysHow to encrypt and decrypt files from S3 without loading the hole content into memory?How to encrypt and decrypt files from S3 without loading the hole content into memory? I have this Worksapces / Monorepo supportDeno does not currently have the capability to resolve `workspace:^` references and this breaks any