Deno KV: Should I expect 'value too large' errors inserting a JSON file with kv.set?

I was trying out kv and quickly ran into something I found surprising;

let kv = await Deno.openKv('omg.db')

let hieroglyphsJSON = await Deno.readTextFile('./hieroglyphs.json')
let hieroglyphs = JSON.parse(hieroglyphsJSON)

kv.set(["hieroglyphs"], hieroglyphs)

let kv = await Deno.openKv('omg.db')

let hieroglyphsJSON = await Deno.readTextFile('./hieroglyphs.json')
let hieroglyphs = JSON.parse(hieroglyphsJSON)

kv.set(["hieroglyphs"], hieroglyphs)

hieroglyphs.json is not huge (360K), and yet I get this error:

$ deno run --unstable --allow-write --allow-read kv.js
error: Uncaught (in promise) TypeError: value too large (max 65536 bytes)
kv.set(["hieroglyphs"], hieroglyphs)
   ^
    at Kv.set (ext:deno_kv/01_db.ts:87:41)
    at file:///Users/me/kv.js:8:4
    at eventLoopTick (ext:core/01_core.js:183:11)

$ deno run --unstable --allow-write --allow-read kv.js
error: Uncaught (in promise) TypeError: value too large (max 65536 bytes)
kv.set(["hieroglyphs"], hieroglyphs)
   ^
    at Kv.set (ext:deno_kv/01_db.ts:87:41)
    at file:///Users/me/kv.js:8:4
    at eventLoopTick (ext:core/01_core.js:183:11)

I feel like I’m missing something obvious; am I trying to do something kv wasn’t designed for? TIA

5 Replies

tuhana :3•14mo ago

Error is obvious, you can put maximum 65536 bytes of data and apparently your data is bigger than that.

patwasalinguist•14mo ago

forgive my ignorance. that's the maximum size that can be inserted as a value? that seems very small to me.

tuhana :3•14mo ago

KV is still an unstable feature, so it's subject to change, maybe you can create a discussion about it on GitHub or at #general

NDH•14mo ago

There are plans for Blob storage. Also you could break that up into 64k chunks and use a key sequence to fetch (get) the chunks.

// foreach chunk - index++ 
kv.set(["hieroglyphs", "chunks", index], chunk)
// then
const iter = kv.list<string>({ prefix: ["hieroglyphs"] });

// foreach chunk - index++ 
kv.set(["hieroglyphs", "chunks", index], chunk)
// then
const iter = kv.list<string>({ prefix: ["hieroglyphs"] });

patwasalinguist•14mo ago

I see, thank you both