Upgrade to Deno 2.2 caused an outage
Today, we upgraded from 2.1.10 to 2.2.11 and this caused an outage. Although we have extensive observability, there is no indication of what could have been the cause of it. The only things we know is that:
1) We are certain this is related to the Deno version, as we have reverted this only
2) Whatever it was, this caused high memory pressure
I don't even know where to start from.
6 Replies
To note: we ship an executable produced by
deno compile
.I’d create a GitHub issue
Hard to tell what's going on without knowing more details.
1. Is there anything in the logs?
2. What makes you conclude that it's memory related?
3. What about CPU usage?
4. How confident are you that it relates to the Deno version? Are there any other infrastructure changes that could have affected this
Not super comfortable doing that without having more details.
Is there anything in the logs?We didn't have logs when I posted, because it turns out that the issue breaks OpenTelemetry. We found logs by looking directly at AWS. Here it is:
What makes you conclude that it's memory related?Right after deployment, graph shows memory increasing progressively. At some point connections to the DB start to drop. Then ECS restarts the containers as they become unhealthy. Rince and repeat.
How confident are you that it relates to the Deno version? Are there any other infrastructure changes that could have affected thisA 100% sure. We're doing continuous deployment. We ship one commit at a time, and the commit only upgraded from Deno 2.1.10 to Deno 2.2.11. Once we reverted, everything went back to normal. My hypothesis so far: we're relying on a version of OpenTelemetry than may not be supported by Deno 2.2. Our import map looks like this: I think we'll use OTel's native integration in Deno 2.2 before releasing agian. Actually we have the same issue with Postgres, so it really points to a bug in the node compatibility layer.
Thanks for sharing more details. I've made an issue out of that so that it shows up in our tracker https://github.com/denoland/deno/issues/29094
GitHub
TypeError: expected AsyncWrap · Issue #29094 · denoland/deno
Originally reported in discord: https://discord.com/channels/684898665143206084/1366759238218551317/1366759238218551317 Version: Deno 2.2.11 Is there anything in the logs? We didn't have logs w...
Thank you very much. Going over there then.