Phatso•4mo ago

Importing local files (streaming large JSON files)

is there an easy way to load a massive json file into memory? i found: https://deno.land/x/json_stream@v0.1.0-pre.12 but it's fetch on a local file example doesn't seem to work for me:

error: Uncaught (in promise) TypeError: Invalid URL:

error: Uncaught (in promise) TypeError: Invalid URL:

am i doing something wrong or is there a new way to load local files

4 Replies

ioB•4mo ago

Could you share your code

Doctor 🤖•4mo ago

You can use JsonParseStream if it's a file with a single json object on each line.

JsonParseStream - @std/json - JSR

@std/json on JSR: (Streaming) parsing and serializing of JSON files

Doctor 🤖•4mo ago

But if it's just one big array or object, then you'd just load it in like any other json file.

const json = JSON.parse(await Deno.readTextFile('path.json'))

const json = JSON.parse(await Deno.readTextFile('path.json'))

NDH•4mo ago

I've used this in my in-mem DB code.

// the hundredK.json file contains 100,000 user objects
// [
//    [0,{"id":0,"value":{"id":0,"first":"Tough","last":"Republic","age":55}}],
//    [1,{"id":1,"value":{"id":1,"first":"Impassioned","last":"Stock","age":10}}],
//    ...
// ]
import * as largeJson from './hundredK.json' assert {type: 'json'};
type User = { id: number, first: string, last: string, age: number }
//@ts-ignore
const UserMap: Map<number, User> = new Map(largeJson.default)
console.log(UserMap.get(1))
console.log(UserMap.get(99990))

// To save the contents of the map to a file
Deno.writeTextFileSync("hundredKJson.json", JSON.stringify([...UserMap.entries()]))

// the hundredK.json file contains 100,000 user objects
// [
//    [0,{"id":0,"value":{"id":0,"first":"Tough","last":"Republic","age":55}}],
//    [1,{"id":1,"value":{"id":1,"first":"Impassioned","last":"Stock","age":10}}],
//    ...
// ]
import * as largeJson from './hundredK.json' assert {type: 'json'};
type User = { id: number, first: string, last: string, age: number }
//@ts-ignore
const UserMap: Map<number, User> = new Map(largeJson.default)
console.log(UserMap.get(1))
console.log(UserMap.get(99990))

// To save the contents of the map to a file
Deno.writeTextFileSync("hundredKJson.json", JSON.stringify([...UserMap.entries()]))