D
Deno

help

Purging Kv keys

Hhexagon56k6/26/2023
I have a logger which write logs to a kv store with dual keys, something like
kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)
kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)
So far all is well.. But a problem pups up when i want to purge all logs before a specific start time, i could loop through all processes, deleting using this selector:
{
prefix: ["logs_by_process", currentProcessId],
end: ["logs_by_process", currentProcessId, startTime],
}
{
prefix: ["logs_by_process", currentProcessId],
end: ["logs_by_process", currentProcessId, startTime],
}
But as end-users can add/delete processes over time, i cannot be sure i known which processes exist in history, so i would need to use a wildcard, like:
{
prefix: ["logs_by_process", "*"],
end: ["logs_by_process", "*", startTime],
}
{
prefix: ["logs_by_process", "*"],
end: ["logs_by_process", "*", startTime],
}
Any idéas on how to make this happen efficiently, without looping over all possible keys? Actual, non-functioning code: https://github.com/Hexagon/pup/blob/afd8d3f3b292c8e317d21dea9c7d0509f1e40d59/lib/core/logger.ts#L236
KWKevin Whinnery6/27/2023
I think you might need to introduce a third key to enable this behavior:
['logs_by_timestamp', '<timestamp>', '<process-id>']
['logs_by_timestamp', '<timestamp>', '<process-id>']
There isn't a way to filter keys with wildcards in Deno KV unfortunately (it's based on Foundation DB, which doesn't have a SCAN like Redis which would enable this)
Hhexagon56k6/27/2023
But if i introduce a third key, wouldn't the value(log object) be left behind if i index it using three keys and delete two of the keys? Hmm, i guess i could
kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)
/* Below is a new key, which essentially is a time based (instead of process-based) lookup-table for the keys starting with logs_by_process, ... */
kv.set(["logs_by_process_time_lookup", Date.now()],["logs_by_process", "current-process-id", Date.now()])
kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)
/* Below is a new key, which essentially is a time based (instead of process-based) lookup-table for the keys starting with logs_by_process, ... */
kv.set(["logs_by_process_time_lookup", Date.now()],["logs_by_process", "current-process-id", Date.now()])
BBBustin Base7/1/2023
I don't think you're going to want to use kv db for logs. That will quickly grow way beyond what you should put into a single key. Consider using Azure Log Analytics or something like that. Its a pretty simple single secret configuration and pretty cheap if you don't go crazy with the logs. Here is a single file you could snipe, or if you want I could make it into a stand alone module: https://github.com/justinmchase/grove/blob/main/src/logging/azure.logger.ts
GitHub
grove/src/logging/azure.logger.ts at main · justinmchase/grove
Contribute to justinmchase/grove development by creating an account on GitHub.

Looking for more? Join the community!

Recommended Posts
What is the right way to bundle native modules?If I'm not able to import a native module with the built-in "import" keyword, what is the ideal way HandleScope in async opIs there a way to get v8::HandleScope inside async op if I add it as ``` scope: &mut v8::HandleScopDeno LSP randomly breaks until a restart (unable to send result to client)This wasnt an issue until recently. In vscode randomly the lsp stops working. Trying to use the autoIs anyone working on LLMs for Deno so I don't have to learn it in disgusting Python?I'm coming to terms with the reality of having to learn this technology but not on Python.. I mean..Permission denied (os error 13)Please help me, I'm desparate. It doesn't print anything else.Is there a rich text (wysiwyg) editor for deno fresh ?I'm working on a blog, and i need a rich text editor for articles.deno_core usageAre there any repos using deno_core in their code + how would I enable the unstable features and URLError reading `allocUnsafe` using npm:icsHello there, I try to use npm:ics in my project and in some runs it works, in others it doesn't (whiUsing discord.js in Deno?Apparently Deno supports discord.js now. Awesome. But now I can't make it work. According to debug Type alias with inherited type parameter constraintsIs there any way to do this? I want SVLTN to just be a short-name alias for SomeVeryLongTypeName, anFresh: global stateIs it possible to have a global state that can be accessed from anywhere? Currently I'm passing the Run Deno on Linux without /proc mountedI have a few automatic scripts using deno to run stuff, and wanted to migrate them to another Machin