Cheat-Code Sam•8mo ago

Indexing Items by Last Updated Time in Deno KV

Hello, I need some help with indexing in Deno KV. I have a list of threads, and each thread contains posts. When a post is added to a thread, I want to update the "last updated" timestamp for that thread. Additionally, I need to sort the threads by their "last updated" timestamp. I created a secondary index with the key space ['last_updated', lastUpdatedTime, threadId]. However, this approach results in duplicates because I can't remove the older "last updated" timestamps. My goal is to get a list of 200 unique threads, sorted by the "last updated" timestamp. How can I achieve this without getting duplicate entries for the same thread? Is there a way I can do this with secondary indexes?

10 Replies

Deno AI Helper•8mo ago

I apologize, but I wasn't able to find the answer to your question in the official Deno documentation. However, based on the information you've provided, it seems like you're trying to maintain a unique index for each thread based on the "last updated" timestamp. One potential issue with your current approach is that you're creating a new index entry every time a thread is updated without removing the old entry, which leads to duplicates. To avoid this, you would typically need to ensure that you update the existing index entry for a thread rather than creating a new one each time. Since I don't have access to the specifics of how Deno KV handles secondary indexes, I can't provide a Deno-specific solution. However, a common approach in such scenarios is to: 1. Retrieve the current index entry for the thread if it exists. 2. Update the index entry with the new "last updated" timestamp. 3. Save the updated index entry back to the KV store. This advice is not based on the Deno documentation, so please double-check the accuracy of this advice and consider reaching out to the Deno community or reviewing the Deno KV documentation for more specific guidance on handling secondary indexes and unique constraints within Deno KV.

Olli•7mo ago

I see the issue. One way to go about it might be to delete the existing index entries and then setting the new one on every update, so that there is always only 1 copy of any given thread:

type Thread = {
  lastUpdated: string
  posts: unknown[]
}

const kv = await Deno.openKv()

async function createThread(id: string, thread: Thread) {
  await kv.atomic()
    .check({
      key: ["threads", id],
      versionstamp: null
    })
    .set(["threads", id], thread)
    .set(["threads", "last_updated", thread.lastUpdated, id], thread)
    .commit()
}

async function updateThread(id: string, thread: Thread) {
  // Get the current thread value
  const currentThread = await kv.get<Thread>(["threads", id])
  if (!currentThread.value) {
    // ...do something
    return
  }

  // Set the new thread entry
  const atomic = kv.atomic()
  atomic.set(["threads", id], thread)

  // Delete any existing index entries
  const iter = kv.list({ prefix: ["threads", "last_updated", currentThread.value.lastUpdated] })
  for await (const { key } of iter) {
    atomic.delete(key)
  }

  // Set the new index entry
  await atomic
    .set(["threads", "last_updated", thread.lastUpdated, id], thread)
    .commit()
}

type Thread = {
  lastUpdated: string
  posts: unknown[]
}

const kv = await Deno.openKv()

async function createThread(id: string, thread: Thread) {
  await kv.atomic()
    .check({
      key: ["threads", id],
      versionstamp: null
    })
    .set(["threads", id], thread)
    .set(["threads", "last_updated", thread.lastUpdated, id], thread)
    .commit()
}

async function updateThread(id: string, thread: Thread) {
  // Get the current thread value
  const currentThread = await kv.get<Thread>(["threads", id])
  if (!currentThread.value) {
    // ...do something
    return
  }

  // Set the new thread entry
  const atomic = kv.atomic()
  atomic.set(["threads", id], thread)

  // Delete any existing index entries
  const iter = kv.list({ prefix: ["threads", "last_updated", currentThread.value.lastUpdated] })
  for await (const { key } of iter) {
    atomic.delete(key)
  }

  // Set the new index entry
  await atomic
    .set(["threads", "last_updated", thread.lastUpdated, id], thread)
    .commit()
}

This might be a bit excessive, but I dont really see any way to use unique indexing when there is a timestamp and id involved.

Cheat-Code SamOP•7mo ago

This actually makes a lot of sense, I never would have thought of that. Thank you!

Olli•7mo ago

Just a heads up, the id and lastUpdated parts of the secondary index key should maybe be flipped?

Cheat-Code SamOP•7mo ago

Why so? Wouldn't I query by

kv.list({ prefix: ["threads", "last_updated"] }, { limit: 20 })

kv.list({ prefix: ["threads", "last_updated"] }, { limit: 20 })

Which would return a list of threads sorted by last updated, with a limit of 20.

Olli•7mo ago

yeah, actually that part makes sense. But when deleting them, I think there will be an issue since it might delete other thread entries with the same lastUpdated value? I dont think my proposal is a valid solution after all

Cheat-Code SamOP•7mo ago

const thread = await kv.get<Thread>(["thread", id])
await kv
  .atomic()
  .delete(["thread", id])
  .delete(["thread", "last_updated", thread.value.lastUpdated, id])
  .commit()

const thread = await kv.get<Thread>(["thread", id])
await kv
  .atomic()
  .delete(["thread", id])
  .delete(["thread", "last_updated", thread.value.lastUpdated, id])
  .commit()

I think this would work

Olli•7mo ago

yeah

Cheat-Code SamOP•7mo ago

And when a thread is updated I could do something like this

const newThreadthread = "modified thread object"
await kv
  .atomic()
  .set(["thread", id], thread)
  .set(["thread", "last_updated", lastUpdated, id], thread)
  .delete(["thread", "last_updated", oldLastUpdated, id])
  .commit()

const newThreadthread = "modified thread object"
await kv
  .atomic()
  .set(["thread", id], thread)
  .set(["thread", "last_updated", lastUpdated, id], thread)
  .delete(["thread", "last_updated", oldLastUpdated, id])
  .commit()

It's a little exhaustive, but that's just the nature of working with the kv

Olli•7mo ago

Since you knwo it will be only 1 entry I think that should be fine