A person•9mo ago

Caching results from API

Hi, I'm currently working on a project where I get data from an API (https://www.robotevents.com/api/v2), process the data, and return it to the browser. I was wondering about how I could cache the results from the API because if the website has let's say 5000 users then the API won't be able to handle all of those, so I want to cache the data somewhere and update it every 24 hours or so. Do I use something like Deno KV for this and then when the server get's a request it checks if data is already in that database and returns it. Also, how do I call the API every 24 hours? Thank you for your help.

8 Replies

Deno AI Helper•9mo ago

Hello! Caching results from an API is a great way to improve performance and reduce the load on the API server. In Deno, you can use the Cache API to cache Request/Response objects, which can be particularly useful when you're generating responses dynamically. The documentation provides an example of how to use the Cache API:

import { serve } from "https://deno.land/std@0.158.0/http/server.ts";

const CACHE = await caches.open("v1");

serve(async (req: Request) => {
  const res = await CACHE.match(req);
  if (res) {
    res.headers.set("x-cache-hit", "true");
    return res;
  }

  const { searchParams } = new URL(req.url);
  const name = searchParams.get("name");
  const response = new Response(`Hello ${name}!`);

  await CACHE.put(req, response.clone());
  return response;
});

import { serve } from "https://deno.land/std@0.158.0/http/server.ts";

const CACHE = await caches.open("v1");

serve(async (req: Request) => {
  const res = await CACHE.match(req);
  if (res) {
    res.headers.set("x-cache-hit", "true");
    return res;
  }

  const { searchParams } = new URL(req.url);
  const name = searchParams.get("name");
  const response = new Response(`Hello ${name}!`);

  await CACHE.put(req, response.clone());
  return response;
});

In this example, the cache is persisted to the file system, so restarting Deno will not flush the cache. You can use similar logic to cache the responses from the API you're calling. When a request comes in, you can check if the data is already in the cache and return it if available. If not, you can fetch the data from the API, cache it, and then return it to the user. As for updating the cache every 24 hours, you can use a simple interval to trigger the cache update. Here's a general idea of how you might set up such an interval:

// Function to update the cache
async function updateCache() {
  // Logic to call the API and update the cache
}

// Call `updateCache` immediately to populate the cache
updateCache();

// Set an interval to call `updateCache` every 24 hours
setInterval(updateCache, 24 * 60 * 60 * 1000);

// Function to update the cache
async function updateCache() {
  // Logic to call the API and update the cache
}

// Call `updateCache` immediately to populate the cache
updateCache();

// Set an interval to call `updateCache` every 24 hours
setInterval(updateCache, 24 * 60 * 60 * 1000);

Please note that the above code is a general guide and not directly from the documentation. You'll need to adapt it to fit your specific use

Deno AI Helper•9mo ago

case, including handling the API calls and processing the data as needed. Regarding Deno KV, it's a globally replicated database that you can use to add state to your serverless functions or applications. It's very easy to use, as shown in the documentation:

const kv = await Deno.openKv();

const kv = await Deno.openKv();

You could use Deno KV to store the processed data from the API if you need a more persistent and globally replicated solution. However, for simple caching purposes, the Cache API might be sufficient. Remember to handle the cache invalidation logic according to your needs, such as setting a TTL (time to live) for the cache entries or manually clearing them after 24 hours. 🦕

Helpful Links: How to Build a CRUD API with Oak and Deno KV Deno Docs How to Use Redis with Deno | Deno Docs Deno 1.26 Release Notes Building Serverless Functions with Deno A list of every web API in Deno

A personOP•9mo ago

Can I do the same with Deno Web Workers?

Esente•9mo ago

There is no Deno Web Workers. And you can't use the Cache API on Deno Deploy, yet You can also use Deno.cron to fetch the API every 24h

raunioroo•9mo ago

Do you just mean there is no Deno-branded version of Web Workers? Because Web Workers in Deno definitely is a thing. (not sure if cache api is usable inside web workers, but would be slightly surprised if it wasn't.)

Esente•9mo ago

Ah, sorry I was wrong. I was basing off my previous encounter with errors not having Worker available, but that was probably on Deploy.

A personOP•9mo ago

I probably is cause you can just put it inside the function that calls the worker

cknight•9mo ago

I'd use KV to store the cached entry (assuming data is always less than 64kb). When a request gets the data from KV, if it isn't there, then call the API and store the value. You can use Deno cron to call the API every 24 hours:

Deno.cron("Call API every day at 1am", "0 1 * * *", async () => {
  const result = await callApi();
  await kv.set(["myAPICache"], result);
});

Deno.cron("Call API every day at 1am", "0 1 * * *", async () => {
  const result = await callApi();
  await kv.set(["myAPICache"], result);
});