Deno KV: Should I expect 'value too large' errors inserting a JSON file with kv.set?

I was trying out kv and quickly ran into something I found surprising;

let kv = await Deno.openKv('omg.db')

let hieroglyphsJSON = await Deno.readTextFile('./hieroglyphs.json')
let hieroglyphs = JSON.parse(hieroglyphsJSON)

kv.set(["hieroglyphs"], hieroglyphs)

let kv = await Deno.openKv('omg.db')

let hieroglyphsJSON = await Deno.readTextFile('./hieroglyphs.json')
let hieroglyphs = JSON.parse(hieroglyphsJSON)

kv.set(["hieroglyphs"], hieroglyphs)

hieroglyphs.json is not huge (360K), and yet I get this error:

$ deno run --unstable --allow-write --allow-read kv.js
error: Uncaught (in promise) TypeError: value too large (max 65536 bytes)
kv.set(["hieroglyphs"], hieroglyphs)
   ^
    at Kv.set (ext:deno_kv/01_db.ts:87:41)
    at file:///Users/me/kv.js:8:4
    at eventLoopTick (ext:core/01_core.js:183:11)

$ deno run --unstable --allow-write --allow-read kv.js
error: Uncaught (in promise) TypeError: value too large (max 65536 bytes)
kv.set(["hieroglyphs"], hieroglyphs)
   ^
    at Kv.set (ext:deno_kv/01_db.ts:87:41)
    at file:///Users/me/kv.js:8:4
    at eventLoopTick (ext:core/01_core.js:183:11)

I feel like I’m missing something obvious; am I trying to do something kv wasn’t designed for? TIA

5 Replies

tuhana :3•16mo ago

Error is obvious, you can put maximum 65536 bytes of data and apparently your data is bigger than that.

patwasalinguistOP•16mo ago

forgive my ignorance. that's the maximum size that can be inserted as a value? that seems very small to me.

tuhana :3•16mo ago

KV is still an unstable feature, so it's subject to change, maybe you can create a discussion about it on GitHub or at #general

Fifth-Normal-Form•16mo ago

There are plans for Blob storage. Also you could break that up into 64k chunks and use a key sequence to fetch (get) the chunks.

// foreach chunk - index++ 
kv.set(["hieroglyphs", "chunks", index], chunk)
// then
const iter = kv.list<string>({ prefix: ["hieroglyphs"] });

// foreach chunk - index++ 
kv.set(["hieroglyphs", "chunks", index], chunk)
// then
const iter = kv.list<string>({ prefix: ["hieroglyphs"] });

patwasalinguistOP•16mo ago

I see, thank you both