Yacine Hmito•2mo ago

Upgrade to Deno 2.2 caused an outage

Today, we upgraded from 2.1.10 to 2.2.11 and this caused an outage. Although we have extensive observability, there is no indication of what could have been the cause of it. The only things we know is that: 1) We are certain this is related to the Deno version, as we have reverted this only 2) Whatever it was, this caused high memory pressure I don't even know where to start from.

6 Replies

Yacine HmitoOP•2mo ago

To note: we ship an executable produced by deno compile.

iuioiua•2mo ago

I’d create a GitHub issue

marvinh.•2mo ago

Hard to tell what's going on without knowing more details. 1. Is there anything in the logs? 2. What makes you conclude that it's memory related? 3. What about CPU usage? 4. How confident are you that it relates to the Deno version? Are there any other infrastructure changes that could have affected this

Yacine HmitoOP•2mo ago

Not super comfortable doing that without having more details.

Is there anything in the logs?

We didn't have logs when I posted, because it turns out that the issue breaks OpenTelemetry. We found logs by looking directly at AWS. Here it is:

April 29, 2025 at 13:58 (UTC+2:00) Unhandled rejection TypeError: expected AsyncWrap
April 29, 2025 at 13:58 (UTC+2:00) at _getNewAsyncId (node:net:62:35)
April 29, 2025 at 13:58 (UTC+2:00) at _initSocketHandle (node:net:424:29)
April 29, 2025 at 13:58 (UTC+2:00) at Socket.connect (node:net:800:7)
April 29, 2025 at 13:58 (UTC+2:00) at Timeout.connect [as _onTimeout] (file:///tmp/deno-compile-diamond/.deno_compile_node_modules/localhost/postgres/3.4.5/src/connection.js:345:12)


April 29, 2025 at 13:58 (UTC+2:00) Error sending spans to exporter: TypeError: expected AsyncWrap 
April 29, 2025 at 13:58 (UTC+2:00) at _getNewAsyncId (node:net:62:35) 
April 29, 2025 at 13:58 (UTC+2:00) at _initSocketHandle (node:net:424:29) 
April 29, 2025 at 13:58 (UTC+2:00) at Socket.connect (node:net:800:7) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.connect [as createConnection] (node:net:1275:17) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.createSocket (node:_http_agent:271:26) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.addRequest (node:_http_agent:222:10) 
April 29, 2025 at 13:58 (UTC+2:00) at new ClientRequest (node:http:263:18) 
April 29, 2025 at 13:58 (UTC+2:00) at new HttpsClientRequest (node:https:73:79) 
April 29, 2025 at 13:58 (UTC+2:00) at request (node:https:123:10) 
April 29, 2025 at 13:58 (UTC+2:00) at HttpExporterTransport.sendWithHttp [as _send] (file:///tmp/deno-compile-diamond/.deno_compile_node_modules/localhost/@opentelemetry/otlp-exporter-base/0.57.2/build/src/transport/http-transport-utils.js:45:17)

April 29, 2025 at 13:58 (UTC+2:00) Unhandled rejection TypeError: expected AsyncWrap
April 29, 2025 at 13:58 (UTC+2:00) at _getNewAsyncId (node:net:62:35)
April 29, 2025 at 13:58 (UTC+2:00) at _initSocketHandle (node:net:424:29)
April 29, 2025 at 13:58 (UTC+2:00) at Socket.connect (node:net:800:7)
April 29, 2025 at 13:58 (UTC+2:00) at Timeout.connect [as _onTimeout] (file:///tmp/deno-compile-diamond/.deno_compile_node_modules/localhost/postgres/3.4.5/src/connection.js:345:12)


April 29, 2025 at 13:58 (UTC+2:00) Error sending spans to exporter: TypeError: expected AsyncWrap 
April 29, 2025 at 13:58 (UTC+2:00) at _getNewAsyncId (node:net:62:35) 
April 29, 2025 at 13:58 (UTC+2:00) at _initSocketHandle (node:net:424:29) 
April 29, 2025 at 13:58 (UTC+2:00) at Socket.connect (node:net:800:7) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.connect [as createConnection] (node:net:1275:17) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.createSocket (node:_http_agent:271:26) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.addRequest (node:_http_agent:222:10) 
April 29, 2025 at 13:58 (UTC+2:00) at new ClientRequest (node:http:263:18) 
April 29, 2025 at 13:58 (UTC+2:00) at new HttpsClientRequest (node:https:73:79) 
April 29, 2025 at 13:58 (UTC+2:00) at request (node:https:123:10) 
April 29, 2025 at 13:58 (UTC+2:00) at HttpExporterTransport.sendWithHttp [as _send] (file:///tmp/deno-compile-diamond/.deno_compile_node_modules/localhost/@opentelemetry/otlp-exporter-base/0.57.2/build/src/transport/http-transport-utils.js:45:17)

What makes you conclude that it's memory related?

Right after deployment, graph shows memory increasing progressively. At some point connections to the DB start to drop. Then ECS restarts the containers as they become unhealthy. Rince and repeat.

How confident are you that it relates to the Deno version? Are there any other infrastructure changes that could have affected this

A 100% sure. We're doing continuous deployment. We ship one commit at a time, and the commit only upgraded from Deno 2.1.10 to Deno 2.2.11. Once we reverted, everything went back to normal. My hypothesis so far: we're relying on a version of OpenTelemetry than may not be supported by Deno 2.2. Our import map looks like this:

    "@opentelemetry/api": "npm:@opentelemetry/api@1.9.0",
    "@opentelemetry/core": "npm:@opentelemetry/core@1.30.1",
    "@opentelemetry/exporter-trace-otlp-proto": "npm:@opentelemetry/exporter-trace-otlp-proto@0.57.2",
    "@opentelemetry/resources": "npm:@opentelemetry/resources@1.30.1",
    "@opentelemetry/semantic-conventions": "npm:@opentelemetry/semantic-conventions@1.30.0",
    "@opentelemetry/sdk-trace-base": "npm:@opentelemetry/sdk-trace-base@1.30.1",
    "@opentelemetry/sdk-trace-node": "npm:@opentelemetry/sdk-trace-node@1.30.1",
    "@opentelemetry/sdk-node": "npm:@opentelemetry/sdk-node@0.57.2",

    "@opentelemetry/api": "npm:@opentelemetry/api@1.9.0",
    "@opentelemetry/core": "npm:@opentelemetry/core@1.30.1",
    "@opentelemetry/exporter-trace-otlp-proto": "npm:@opentelemetry/exporter-trace-otlp-proto@0.57.2",
    "@opentelemetry/resources": "npm:@opentelemetry/resources@1.30.1",
    "@opentelemetry/semantic-conventions": "npm:@opentelemetry/semantic-conventions@1.30.0",
    "@opentelemetry/sdk-trace-base": "npm:@opentelemetry/sdk-trace-base@1.30.1",
    "@opentelemetry/sdk-trace-node": "npm:@opentelemetry/sdk-trace-node@1.30.1",
    "@opentelemetry/sdk-node": "npm:@opentelemetry/sdk-node@0.57.2",

I think we'll use OTel's native integration in Deno 2.2 before releasing agian. Actually we have the same issue with Postgres, so it really points to a bug in the node compatibility layer.

marvinh.•2mo ago

Thanks for sharing more details. I've made an issue out of that so that it shows up in our tracker https://github.com/denoland/deno/issues/29094

GitHub

TypeError: expected AsyncWrap · Issue #29094 · denoland/deno

Originally reported in discord: https://discord.com/channels/684898665143206084/1366759238218551317/1366759238218551317 Version: Deno 2.2.11 Is there anything in the logs? We didn't have logs w...

Yacine HmitoOP•2mo ago

Thank you very much. Going over there then.

Upgrade to Deno 2.2 caused an outage

Did you find this page helpful?