Hexagon
Hexagon•2y ago

Purging Kv keys

I have a logger which write logs to a kv store with dual keys, something like 
kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)
kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)
So far all is well.. But a problem pups up when i want to purge all logs before a specific start time, i could loop through all processes, deleting using this selector: 
{
  prefix: ["logs_by_process", currentProcessId],
  end: ["logs_by_process", currentProcessId, startTime],
}
{
  prefix: ["logs_by_process", currentProcessId],
  end: ["logs_by_process", currentProcessId, startTime],
}
But as end-users can add/delete processes over time, i cannot be sure i known which processes exist in history, so i would need to use a wildcard, like: 
{
  prefix: ["logs_by_process", "*"],
  end: ["logs_by_process", "*", startTime],
}
{
  prefix: ["logs_by_process", "*"],
  end: ["logs_by_process", "*", startTime],
}
Any idéas on how to make this happen efficiently, without looping over all possible keys? Actual, non-functioning code: https://github.com/Hexagon/pup/blob/afd8d3f3b292c8e317d21dea9c7d0509f1e40d59/lib/core/logger.ts#L236
4 Replies
Hexagon
HexagonOP•2y ago
Kevin Whinnery
Kevin Whinnery•2y ago
I think you might need to introduce a third key to enable this behavior: 
['logs_by_timestamp', '<timestamp>', '<process-id>']
['logs_by_timestamp', '<timestamp>', '<process-id>']
There isn't a way to filter keys with wildcards in Deno KV unfortunately (it's based on Foundation DB, which doesn't have a SCAN like Redis which would enable this)
Hexagon
HexagonOP•2y ago
But if i introduce a third key, wouldn't the value(log object) be left behind if i index it using three keys and delete two of the keys? Hmm, i guess i could 
kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)
/* Below is a new key, which essentially is a time based (instead of process-based) lookup-table for the keys starting with logs_by_process, ... */
kv.set(["logs_by_process_time_lookup", Date.now()],["logs_by_process", "current-process-id", Date.now()])
kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)
/* Below is a new key, which essentially is a time based (instead of process-based) lookup-table for the keys starting with logs_by_process, ... */
kv.set(["logs_by_process_time_lookup", Date.now()],["logs_by_process", "current-process-id", Date.now()])
Bustin Base
Bustin Base•2y ago
I don't think you're going to want to use kv db for logs. That will quickly grow way beyond what you should put into a single key. Consider using Azure Log Analytics or something like that. Its a pretty simple single secret configuration and pretty cheap if you don't go crazy with the logs. Here is a single file you could snipe, or if you want I could make it into a stand alone module: https://github.com/justinmchase/grove/blob/main/src/logging/azure.logger.ts
GitHub
grove/src/logging/azure.logger.ts at main · justinmchase/grove
Contribute to justinmchase/grove development by creating an account on GitHub.

Did you find this page helpful?