patwasalinguist
patwasalinguist15mo ago

Deno KV: Should I expect 'value too large' errors inserting a JSON file with kv.set?

I was trying out kv and quickly ran into something I found surprising;
let kv = await Deno.openKv('omg.db')

let hieroglyphsJSON = await Deno.readTextFile('./hieroglyphs.json')
let hieroglyphs = JSON.parse(hieroglyphsJSON)

kv.set(["hieroglyphs"], hieroglyphs)
let kv = await Deno.openKv('omg.db')

let hieroglyphsJSON = await Deno.readTextFile('./hieroglyphs.json')
let hieroglyphs = JSON.parse(hieroglyphsJSON)

kv.set(["hieroglyphs"], hieroglyphs)
hieroglyphs.json is not huge (360K), and yet I get this error:
$ deno run --unstable --allow-write --allow-read kv.js
error: Uncaught (in promise) TypeError: value too large (max 65536 bytes)
kv.set(["hieroglyphs"], hieroglyphs)
^
at Kv.set (ext:deno_kv/01_db.ts:87:41)
at file:///Users/me/kv.js:8:4
at eventLoopTick (ext:core/01_core.js:183:11)
$ deno run --unstable --allow-write --allow-read kv.js
error: Uncaught (in promise) TypeError: value too large (max 65536 bytes)
kv.set(["hieroglyphs"], hieroglyphs)
^
at Kv.set (ext:deno_kv/01_db.ts:87:41)
at file:///Users/me/kv.js:8:4
at eventLoopTick (ext:core/01_core.js:183:11)
I feel like I’m missing something obvious; am I trying to do something kv wasn’t designed for? TIA
5 Replies
tuhana :3
tuhana :315mo ago
Error is obvious, you can put maximum 65536 bytes of data and apparently your data is bigger than that.
patwasalinguist
patwasalinguist15mo ago
forgive my ignorance. that's the maximum size that can be inserted as a value? that seems very small to me.
tuhana :3
tuhana :315mo ago
KV is still an unstable feature, so it's subject to change, maybe you can create a discussion about it on GitHub or at #general
NDH
NDH15mo ago
There are plans for Blob storage. Also you could break that up into 64k chunks and use a key sequence to fetch (get) the chunks.
// foreach chunk - index++
kv.set(["hieroglyphs", "chunks", index], chunk)
// then
const iter = kv.list<string>({ prefix: ["hieroglyphs"] });
// foreach chunk - index++
kv.set(["hieroglyphs", "chunks", index], chunk)
// then
const iter = kv.list<string>({ prefix: ["hieroglyphs"] });
patwasalinguist
patwasalinguist15mo ago
I see, thank you both