krofdrakula•3y ago

Setting up complex values for `Deno.bench` function runs

i'm currently exploring the bench tools in Deno and have come across a case where i can't seem to be able to implement. in the simplest case, let's say we wanted to compare two functions that mutate a given array, but that array is nontrivially large:

const a = <T>(arr: T[], ...items:T[]): T[] => {};
const b = <T>(arr: T[], ...items:T[]): T[] => {};

const getSample = () => {
  const sample:number[] = [];
  for (let i = 0; i < 1e6; i++) sample.push(i);
  return sample;
};

Deno.bench('a', () => {
  const data = getSample();
  a(data, -1);
});

Deno.bench('b', () => {
  const data = getSample();
  b(data, -1);
});

const a = <T>(arr: T[], ...items:T[]): T[] => {};
const b = <T>(arr: T[], ...items:T[]): T[] => {};

const getSample = () => {
  const sample:number[] = [];
  for (let i = 0; i < 1e6; i++) sample.push(i);
  return sample;
};

Deno.bench('a', () => {
  const data = getSample();
  a(data, -1);
});

Deno.bench('b', () => {
  const data = getSample();
  b(data, -1);
});

the problem with these benches is that they build the data within the benchmark function, which i do not want to measure as part of the benchmark. i couldn't find an analog prerun function in the docs, is there a way to pass data created outside the measured function to it? something like:

Deno.bench({
  name: 'a',
  prerun: () => getSample(),
  fn(data) { // `data` === `ReturnType<typeof prerun>`
    a(data, -1);
  }
});

Deno.bench({
  name: 'a',
  prerun: () => getSample(),
  fn(data) { // `data` === `ReturnType<typeof prerun>`
    a(data, -1);
  }
});

5 Replies

krofdrakulaOP•3y ago

one idea that pops into my head is to write an IIFE that creates a closure and returns a function for bench() to consume:

Deno.bench('a', (() => {
  const data = getSample();
  return () => a(data, -1);
})());

Deno.bench('a', (() => {
  const data = getSample();
  return () => a(data, -1);
})());

but that looks a bit brittle and could easily be broken by omitting the parens at the end by mistake. actually... thinking about it now, the problem with that example is that the closure is only created once while the returned function runs many times, essentially having the same problem as an array outside the bench() scope so it doesn't really solve the problem. there also don't seem to be any before() or after() hooks that one could hook into to set or reset state after each execution

AapoAlas•3y ago

Unfortunately this is indeed a missing feature as far as I know.

krofdrakulaOP•3y ago

i was hoping i was being myopic and not seeing an option, but it looks like that is the case then 😐

AapoAlas•3y ago

I am deeply to sorry to have to dispel your hopes for impaired vision 🙂

krofdrakulaOP•3y ago

i posted an issue on GH to describe it, let's see what the devs have to say. https://github.com/denoland/deno/issues/17589

GitHub

Deno.bench does not allow specifying setup/teardown functions for...

Benchmarking functions that mutate their operands is currently impossible to measure without also including the setup portion of the run in the measurement. This will cause the benchmarks to give e...

Setting up complex values for `Deno.bench` function runs

Did you find this page helpful?