Indexing Items by Last Updated Time in Deno KV

Hello, I need some help with indexing in Deno KV. I have a list of threads, and each thread contains posts. When a post is added to a thread, I want to update the "last updated" timestamp for that thread. Additionally, I need to sort the threads by their "last updated" timestamp. I created a secondary index with the key space ['last_updated', lastUpdatedTime, threadId]. However, this approach results in duplicates because I can't remove the older "last updated" timestamps. My goal is to get a list of 200 unique threads, sorted by the "last updated" timestamp. How can I achieve this without getting duplicate entries for the same thread? Is there a way I can do this with secondary indexes?
10 Replies
Deno AI Helper
I apologize, but I wasn't able to find the answer to your question in the official Deno documentation. However, based on the information you've provided, it seems like you're trying to maintain a unique index for each thread based on the "last updated" timestamp. One potential issue with your current approach is that you're creating a new index entry every time a thread is updated without removing the old entry, which leads to duplicates. To avoid this, you would typically need to ensure that you update the existing index entry for a thread rather than creating a new one each time. Since I don't have access to the specifics of how Deno KV handles secondary indexes, I can't provide a Deno-specific solution. However, a common approach in such scenarios is to: 1. Retrieve the current index entry for the thread if it exists. 2. Update the index entry with the new "last updated" timestamp. 3. Save the updated index entry back to the KV store. This advice is not based on the Deno documentation, so please double-check the accuracy of this advice and consider reaching out to the Deno community or reviewing the Deno KV documentation for more specific guidance on handling secondary indexes and unique constraints within Deno KV.
Olli
Olli4w ago
I see the issue. One way to go about it might be to delete the existing index entries and then setting the new one on every update, so that there is always only 1 copy of any given thread:
type Thread = {
lastUpdated: string
posts: unknown[]
}

const kv = await Deno.openKv()

async function createThread(id: string, thread: Thread) {
await kv.atomic()
.check({
key: ["threads", id],
versionstamp: null
})
.set(["threads", id], thread)
.set(["threads", "last_updated", thread.lastUpdated, id], thread)
.commit()
}

async function updateThread(id: string, thread: Thread) {
// Get the current thread value
const currentThread = await kv.get<Thread>(["threads", id])
if (!currentThread.value) {
// ...do something
return
}

// Set the new thread entry
const atomic = kv.atomic()
atomic.set(["threads", id], thread)

// Delete any existing index entries
const iter = kv.list({ prefix: ["threads", "last_updated", currentThread.value.lastUpdated] })
for await (const { key } of iter) {
atomic.delete(key)
}

// Set the new index entry
await atomic
.set(["threads", "last_updated", thread.lastUpdated, id], thread)
.commit()
}
type Thread = {
lastUpdated: string
posts: unknown[]
}

const kv = await Deno.openKv()

async function createThread(id: string, thread: Thread) {
await kv.atomic()
.check({
key: ["threads", id],
versionstamp: null
})
.set(["threads", id], thread)
.set(["threads", "last_updated", thread.lastUpdated, id], thread)
.commit()
}

async function updateThread(id: string, thread: Thread) {
// Get the current thread value
const currentThread = await kv.get<Thread>(["threads", id])
if (!currentThread.value) {
// ...do something
return
}

// Set the new thread entry
const atomic = kv.atomic()
atomic.set(["threads", id], thread)

// Delete any existing index entries
const iter = kv.list({ prefix: ["threads", "last_updated", currentThread.value.lastUpdated] })
for await (const { key } of iter) {
atomic.delete(key)
}

// Set the new index entry
await atomic
.set(["threads", "last_updated", thread.lastUpdated, id], thread)
.commit()
}
This might be a bit excessive, but I dont really see any way to use unique indexing when there is a timestamp and id involved.
Cheat-Code Sam
This actually makes a lot of sense, I never would have thought of that. Thank you!
Olli
Olli4w ago
Just a heads up, the id and lastUpdated parts of the secondary index key should maybe be flipped?
Cheat-Code Sam
Why so? Wouldn't I query by
kv.list({ prefix: ["threads", "last_updated"] }, { limit: 20 })
kv.list({ prefix: ["threads", "last_updated"] }, { limit: 20 })
Which would return a list of threads sorted by last updated, with a limit of 20.
Olli
Olli4w ago
yeah, actually that part makes sense. But when deleting them, I think there will be an issue since it might delete other thread entries with the same lastUpdated value? I dont think my proposal is a valid solution after all
Cheat-Code Sam
const thread = await kv.get<Thread>(["thread", id])
await kv
.atomic()
.delete(["thread", id])
.delete(["thread", "last_updated", thread.value.lastUpdated, id])
.commit()
const thread = await kv.get<Thread>(["thread", id])
await kv
.atomic()
.delete(["thread", id])
.delete(["thread", "last_updated", thread.value.lastUpdated, id])
.commit()
I think this would work
Olli
Olli4w ago
yeah
Cheat-Code Sam
And when a thread is updated I could do something like this
const newThreadthread = "modified thread object"
await kv
.atomic()
.set(["thread", id], thread)
.set(["thread", "last_updated", lastUpdated, id], thread)
.delete(["thread", "last_updated", oldLastUpdated, id])
.commit()
const newThreadthread = "modified thread object"
await kv
.atomic()
.set(["thread", id], thread)
.set(["thread", "last_updated", lastUpdated, id], thread)
.delete(["thread", "last_updated", oldLastUpdated, id])
.commit()
It's a little exhaustive, but that's just the nature of working with the kv
Olli
Olli4w ago
Since you knwo it will be only 1 entry I think that should be fine