KV - Not possible to maintain in real world?
I have a question regarding Deno KV: I just checked the docs and it seems straight forward so far BUT since it is not possible to query for values or let the DB itself create secondary indexes I wonder how the real world workflow could look like?
- If I want to query a single value of a huge object, I need to retrieve the full object if I didnt save them as secondary indexes before.
- If I want to query for a specific value of a huge object efficiently, I need to create secondary indexes. But what if I need a new query for a new feature that I develop? Now I need to create new secondary indexes for all existing values before I can do this query? At what point in my application should I add them? On startup? Or should I write a migration script each time I want to add a new query?
- What if I have 1 million users and I have implemented a findByMail() function by creating secondary indexes for all million users, but one day I dont need the findByMail() function anymore. Now I have 2 million database entries from which 1 million are unused. At what point in my application do I delete them? On startup? I need a migration script each time I want to remove a new query?
- How do I perform operations like "Get all users with age greater 21"? I might add another secondary index for the age but then the .get() function does not have the option to perform this filter on the database side.
For example if I think about DynamoDB, which is also a kv storage, I know that when I define a secondary index the database will handle it for me and I dont have to create them myself and keep them updated. Also I can query for specific fields and filter the results. Is this minimalism of Deno KV by design or is this just too early to ask for those things? If it is by design then I feel like Deno KV needs another layer to automate secondary index handling and allow for querying and filtering. If these things are coming, is there any ETA?
17 Replies
"huge object". Note that KV has a limit of 64kb for values.
Therefore every individual value/object retrieval is fast.
sounds like you are looking for SQL/NoSQL database like features in a key value store. instead of working around limitations of kv store, you might look into using a database which sounds like more of the correct tool for your job?
I will use DynamoDB for sure but it seemed to me that Deno KV wants to be the go-to solution (together with the whole other eco system). But right now it is not maintainable at all.
outside of deno deploy, deno kv is built on top of SQLite. You could use that. Or MongoDB, or some SQL db like Postgres, MySQL, whatever. If working from deno deploy, it does support also DynamoDB
In one Youtube video Luca was telling how a high level API for KV could look like and something like this is missing for sure. But from what I see (handling indexes, missing querys and filters) KV needs a full second layer not just a "Mongoose".
And I wonder if there are plans to add those features in the future
From what I see I cannot even set the order asc/desc when using .get()
kv.list({prefix: ["my_key"]}, {reverse:true});Ah true, I checked the wrong fn
But what about "Give me all X in the time beween date A and date B". It is not possible to let this operation be done by the database. I need to filter on my app which means I first need to retrieve 1 million values before I can do so.
kv.list({start: ["key_prefix", startDateTime], end: ["key_prefix", endDateTime]})
where the date is one of the key partsOk you got me on this one too
Still everything I want to query I need to create a secondary index for and I can never forget to update on of those secondary indexes
Yes, KV does require manual handling of secondary indexes, but this can be done via a single function which does the writes in an atomic transaction updating all secondary indexes at once. Incidentally, transactions are stronger than DynamoDB's offering.
Maybe someone else can chime in on that (the future developments etc), I don't have enough experience with key value stores to comment on that. Other than, that all the features/problems you are describing sound like basic features for a "normal database". Personally I use traditional (including NoSQL) databases for all primary storage, then deno kv for data that doesn't need to necessarily be long lived and need to be maintained to the same extent (caches, logs, sessions and the like). BUT I might be underestimating the possible use cases for deno KV. Certainly in Deno Deploy there is some incentive to do as much as possible with Deno KV, as that's AFAIK meant as the " native persistency layer" on Deploy, and probably is more efficient than connecting to external database services. But for local development / self hosted / cloud VPS, I see little need to shoehorn KV to problems more suited for traditional dbs
For me it sounds like that with Deno runtime, Fresh, KV, Deploy, etc. they want to give the world an all in one solution to develop all kinds of applications. I guess they provide all features to make KV be usable easy and reliable for everyone and any usecase but I didnt read anything about exact the future.
For now as Fresh doesnt support SCSS and KV doesnt abstract enough I will need to go back to Node, Nuxt and DynamoDB or MongoDB
You can use MongoDB and I believe DynamoDB just fine in Deno, though. Not familiar with Fresh or SCSS so idk
Deno KV's minimalism is by design. It's intended to offer the building blocks that can be used to create a DB or DB engine the way you want. From what I can tell, they are still working on creating a bigger variety of "building blocks". Ergonomics doesn't seem to be their focus by now, and I'm not sure if it will ever be. They probably expect the community to cover that area while they focus on the core. A couple of libs have already popped up from userland. You can search for "kv" in #showcase
I've been working on an SQLite Query Engine for Deno KV
ala Cockroach DB
definitely possible
Will you need kv-codecs? I have them.
Maybe at some point, not there yet!