Purging Kv keys

I have a logger which write logs to a kv store with dual keys, something like

kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)

kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)

So far all is well.. But a problem pups up when i want to purge all logs before a specific start time, i could loop through all processes, deleting using this selector:

{
  prefix: ["logs_by_process", currentProcessId],
  end: ["logs_by_process", currentProcessId, startTime],
}

{
  prefix: ["logs_by_process", currentProcessId],
  end: ["logs_by_process", currentProcessId, startTime],
}

But as end-users can add/delete processes over time, i cannot be sure i known which processes exist in history, so i would need to use a wildcard, like:

{
  prefix: ["logs_by_process", "*"],
  end: ["logs_by_process", "*", startTime],
}

{
  prefix: ["logs_by_process", "*"],
  end: ["logs_by_process", "*", startTime],
}

Any idéas on how to make this happen efficiently, without looping over all possible keys? Actual, non-functioning code: https://github.com/Hexagon/pup/blob/afd8d3f3b292c8e317d21dea9c7d0509f1e40d59/lib/core/logger.ts#L236

4 Replies

HexagonOP•2y ago

https://tenor.com/view/this-is-fine-fire-burning-life-living-gif-6373010

Tenor

Kevin Whinnery•2y ago

I think you might need to introduce a third key to enable this behavior:

['logs_by_timestamp', '<timestamp>', '<process-id>']

['logs_by_timestamp', '<timestamp>', '<process-id>']

There isn't a way to filter keys with wildcards in Deno KV unfortunately (it's based on Foundation DB, which doesn't have a SCAN like Redis which would enable this)

HexagonOP•2y ago

But if i introduce a third key, wouldn't the value(log object) be left behind if i index it using three keys and delete two of the keys? Hmm, i guess i could

kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)
/* Below is a new key, which essentially is a time based (instead of process-based) lookup-table for the keys starting with logs_by_process, ... */
kv.set(["logs_by_process_time_lookup", Date.now()],["logs_by_process", "current-process-id", Date.now()])

kv.set(["logs_by_time", Date.now()], logObject)
kv.set(["logs_by_process", "current-process-id", Date.now()],logObject)
/* Below is a new key, which essentially is a time based (instead of process-based) lookup-table for the keys starting with logs_by_process, ... */
kv.set(["logs_by_process_time_lookup", Date.now()],["logs_by_process", "current-process-id", Date.now()])

Bustin Base•2y ago

I don't think you're going to want to use kv db for logs. That will quickly grow way beyond what you should put into a single key. Consider using Azure Log Analytics or something like that. Its a pretty simple single secret configuration and pretty cheap if you don't go crazy with the logs. Here is a single file you could snipe, or if you want I could make it into a stand alone module: https://github.com/justinmchase/grove/blob/main/src/logging/azure.logger.ts

GitHub

grove/src/logging/azure.logger.ts at main · justinmchase/grove

Contribute to justinmchase/grove development by creating an account on GitHub.