Phatso Posts

Phatso

•Created by Phatso on 8/12/2024 in #help

Importing local files (streaming large JSON files)

is there an easy way to load a massive json file into memory? i found: https://deno.land/x/json_stream@v0.1.0-pre.12 but it's fetch on a local file example doesn't seem to work for me:

error: Uncaught (in promise) TypeError: Invalid URL:

error: Uncaught (in promise) TypeError: Invalid URL:

am i doing something wrong or is there a new way to load local files

6 replies

DDeno

•Created by Phatso on 5/10/2024 in #help

Websockets API with async processing (is there a better way?)

My Deno app has a websockets layer that receives "requests" from the websocket clients. The clients ask it to update data in the database, retrieve data, etc. similar to a REST api There will be 5-10 clients who will be sending a very large volume of requests. In my initial setup, I created this websockets layer that would offload the actual work for each request to a pool of Workers, who would then send the response back through the Worker pool. I'm hitting some kind of bottleneck. Despite my best efforts, I don't know how to identify it. So I have a few questions: 1. Do you know of an obvious bottleneck with this setup that I'm not realizing? 2. Is there a better way to go about this? I would really like to keep this app simple, and it doesn't feel simple right now A couple of bits of context: 1. I simply need to use Websockets. In the ecosystem I'm developing for, HTTP calls are not a good option 2. My main goal with the Worker Pool was to make sure my Websockets layer wasn't being blocked or delayed while each request is processed, but I'm starting to second guess this design Some code, in case it helps paint a better picture: The websockets layer:

import { WebSocketClient } from "websocket/mod.ts"

import type { WebSocketServer } from "websocket/mod.ts"
import type { WorkerPool } from "./worker_pool.ts"

export const handleSocket = (wss: WebSocketServer, workerPool: WorkerPool) => {
    wss.on("connection", function (ws: WebSocketClient) {
        const returnMessage = (message: string) => {
            ws.send(message)
        }

        ws.on("message", function (message: string) {
            const command = JSON.parse(message)
            const { type } = command

            if (type === "request") {
                workerPool.handleMessage(command, returnMessage)
            }
        })
    })
}

import { WebSocketClient } from "websocket/mod.ts"

import type { WebSocketServer } from "websocket/mod.ts"
import type { WorkerPool } from "./worker_pool.ts"

export const handleSocket = (wss: WebSocketServer, workerPool: WorkerPool) => {
    wss.on("connection", function (ws: WebSocketClient) {
        const returnMessage = (message: string) => {
            ws.send(message)
        }

        ws.on("message", function (message: string) {
            const command = JSON.parse(message)
            const { type } = command

            if (type === "request") {
                workerPool.handleMessage(command, returnMessage)
            }
        })
    })
}

The Worker Pool is attached as a file to this message. I'm feeling worried about my decisions here, I need to come up with a good solution soon. Any help at all is greatly appreciated 🙏

12 replies

DDeno

•Created by Phatso on 12/6/2022 in #help

Import from variable path?

Howdy - I'm trying to make my Cloudflare Workers app self-hostable in a Docker Container. To do this, I'm going to run it with Deno when self-hosting. Ideally I would like to change none of my actual application code and defer any environment-specific setup to different files. i.e.:

src/
  - index.js
  - selfhost.js
  - cloudflare.js

src/
  - index.js
  - selfhost.js
  - cloudflare.js

And then:

// index.js
import { app, serve } from (isSelfHost ? "./selfhost.js" : "./cloudflare.js")

// index.js
import { app, serve } from (isSelfHost ? "./selfhost.js" : "./cloudflare.js")

I know this doesn't work, so my question is how do I achieve something like this? Ultimately the problem is I can't be requiring Deno packages in cloudflare workers or vice versa.

3 replies