Yacine Hmito
Yacine Hmito
PostsComments
DDeno
Created by Yacine Hmito on 4/29/2025 in #help
Upgrade to Deno 2.2 caused an outage
Thank you very much. Going over there then.
14 replies
DDeno
Created by Yacine Hmito on 4/29/2025 in #help
Upgrade to Deno 2.2 caused an outage
Actually we have the same issue with Postgres, so it really points to a bug in the node compatibility layer.
14 replies
DDeno
Created by Yacine Hmito on 4/29/2025 in #help
Upgrade to Deno 2.2 caused an outage
I think we'll use OTel's native integration in Deno 2.2 before releasing agian.
14 replies
DDeno
Created by Yacine Hmito on 4/29/2025 in #help
Upgrade to Deno 2.2 caused an outage
Our import map looks like this: 
    "@opentelemetry/api": "npm:@opentelemetry/api@1.9.0",
    "@opentelemetry/core": "npm:@opentelemetry/core@1.30.1",
    "@opentelemetry/exporter-trace-otlp-proto": "npm:@opentelemetry/exporter-trace-otlp-proto@0.57.2",
    "@opentelemetry/resources": "npm:@opentelemetry/resources@1.30.1",
    "@opentelemetry/semantic-conventions": "npm:@opentelemetry/semantic-conventions@1.30.0",
    "@opentelemetry/sdk-trace-base": "npm:@opentelemetry/sdk-trace-base@1.30.1",
    "@opentelemetry/sdk-trace-node": "npm:@opentelemetry/sdk-trace-node@1.30.1",
    "@opentelemetry/sdk-node": "npm:@opentelemetry/sdk-node@0.57.2",
    "@opentelemetry/api": "npm:@opentelemetry/api@1.9.0",
    "@opentelemetry/core": "npm:@opentelemetry/core@1.30.1",
    "@opentelemetry/exporter-trace-otlp-proto": "npm:@opentelemetry/exporter-trace-otlp-proto@0.57.2",
    "@opentelemetry/resources": "npm:@opentelemetry/resources@1.30.1",
    "@opentelemetry/semantic-conventions": "npm:@opentelemetry/semantic-conventions@1.30.0",
    "@opentelemetry/sdk-trace-base": "npm:@opentelemetry/sdk-trace-base@1.30.1",
    "@opentelemetry/sdk-trace-node": "npm:@opentelemetry/sdk-trace-node@1.30.1",
    "@opentelemetry/sdk-node": "npm:@opentelemetry/sdk-node@0.57.2",
14 replies
DDeno
Created by Yacine Hmito on 4/29/2025 in #help
Upgrade to Deno 2.2 caused an outage
My hypothesis so far: we're relying on a version of OpenTelemetry than may not be supported by Deno 2.2.
14 replies
DDeno
Created by Yacine Hmito on 4/29/2025 in #help
Upgrade to Deno 2.2 caused an outage
How confident are you that it relates to the Deno version? Are there any other infrastructure changes that could have affected this
A 100% sure. We're doing continuous deployment. We ship one commit at a time, and the commit only upgraded from Deno 2.1.10 to Deno 2.2.11. Once we reverted, everything went back to normal.
14 replies
DDeno
Created by Yacine Hmito on 4/29/2025 in #help
Upgrade to Deno 2.2 caused an outage
What makes you conclude that it's memory related?
Right after deployment, graph shows memory increasing progressively. At some point connections to the DB start to drop. Then ECS restarts the containers as they become unhealthy. Rince and repeat.
14 replies
DDeno
Created by Yacine Hmito on 4/29/2025 in #help
Upgrade to Deno 2.2 caused an outage
Is there anything in the logs?
We didn't have logs when I posted, because it turns out that the issue breaks OpenTelemetry. We found logs by looking directly at AWS. Here it is: 
April 29, 2025 at 13:58 (UTC+2:00) Unhandled rejection TypeError: expected AsyncWrap
April 29, 2025 at 13:58 (UTC+2:00) at _getNewAsyncId (node:net:62:35)
April 29, 2025 at 13:58 (UTC+2:00) at _initSocketHandle (node:net:424:29)
April 29, 2025 at 13:58 (UTC+2:00) at Socket.connect (node:net:800:7)
April 29, 2025 at 13:58 (UTC+2:00) at Timeout.connect [as _onTimeout] (file:///tmp/deno-compile-diamond/.deno_compile_node_modules/localhost/postgres/3.4.5/src/connection.js:345:12)


April 29, 2025 at 13:58 (UTC+2:00) Error sending spans to exporter: TypeError: expected AsyncWrap 
April 29, 2025 at 13:58 (UTC+2:00) at _getNewAsyncId (node:net:62:35) 
April 29, 2025 at 13:58 (UTC+2:00) at _initSocketHandle (node:net:424:29) 
April 29, 2025 at 13:58 (UTC+2:00) at Socket.connect (node:net:800:7) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.connect [as createConnection] (node:net:1275:17) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.createSocket (node:_http_agent:271:26) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.addRequest (node:_http_agent:222:10) 
April 29, 2025 at 13:58 (UTC+2:00) at new ClientRequest (node:http:263:18) 
April 29, 2025 at 13:58 (UTC+2:00) at new HttpsClientRequest (node:https:73:79) 
April 29, 2025 at 13:58 (UTC+2:00) at request (node:https:123:10) 
April 29, 2025 at 13:58 (UTC+2:00) at HttpExporterTransport.sendWithHttp [as _send] (file:///tmp/deno-compile-diamond/.deno_compile_node_modules/localhost/@opentelemetry/otlp-exporter-base/0.57.2/build/src/transport/http-transport-utils.js:45:17) 
April 29, 2025 at 13:58 (UTC+2:00) Unhandled rejection TypeError: expected AsyncWrap
April 29, 2025 at 13:58 (UTC+2:00) at _getNewAsyncId (node:net:62:35)
April 29, 2025 at 13:58 (UTC+2:00) at _initSocketHandle (node:net:424:29)
April 29, 2025 at 13:58 (UTC+2:00) at Socket.connect (node:net:800:7)
April 29, 2025 at 13:58 (UTC+2:00) at Timeout.connect [as _onTimeout] (file:///tmp/deno-compile-diamond/.deno_compile_node_modules/localhost/postgres/3.4.5/src/connection.js:345:12)


April 29, 2025 at 13:58 (UTC+2:00) Error sending spans to exporter: TypeError: expected AsyncWrap 
April 29, 2025 at 13:58 (UTC+2:00) at _getNewAsyncId (node:net:62:35) 
April 29, 2025 at 13:58 (UTC+2:00) at _initSocketHandle (node:net:424:29) 
April 29, 2025 at 13:58 (UTC+2:00) at Socket.connect (node:net:800:7) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.connect [as createConnection] (node:net:1275:17) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.createSocket (node:_http_agent:271:26) 
April 29, 2025 at 13:58 (UTC+2:00) at Agent.addRequest (node:_http_agent:222:10) 
April 29, 2025 at 13:58 (UTC+2:00) at new ClientRequest (node:http:263:18) 
April 29, 2025 at 13:58 (UTC+2:00) at new HttpsClientRequest (node:https:73:79) 
April 29, 2025 at 13:58 (UTC+2:00) at request (node:https:123:10) 
April 29, 2025 at 13:58 (UTC+2:00) at HttpExporterTransport.sendWithHttp [as _send] (file:///tmp/deno-compile-diamond/.deno_compile_node_modules/localhost/@opentelemetry/otlp-exporter-base/0.57.2/build/src/transport/http-transport-utils.js:45:17)
14 replies
DDeno
Created by Yacine Hmito on 4/29/2025 in #help
Upgrade to Deno 2.2 caused an outage
Not super comfortable doing that without having more details.
14 replies
DDeno
Created by Yacine Hmito on 4/29/2025 in #help
Upgrade to Deno 2.2 caused an outage
To note: we ship an executable produced by deno compile.
14 replies