recommended way to get file checksum

Hi, I am creating an application that needs to get checksums for files. The code currently looks like this:

import * as node_crypto from 'node:crypto'

  async function get_checksum() {
    const hash = node_crypto.createHash('sha256')
    using file = await Deno.open(this.#filepath)
    for await (const chunk of file.readable) {
      hash.update(chunk)
    }
    return hash.digest('hex')
  }

import * as node_crypto from 'node:crypto'

  async function get_checksum() {
    const hash = node_crypto.createHash('sha256')
    using file = await Deno.open(this.#filepath)
    for await (const chunk of file.readable) {
      hash.update(chunk)
    }
    return hash.digest('hex')
  }

This is fairly though. A ~1GB file will take around 3 minutes to generate a checksum with the code above, whereas the built in linux utility sha256sum can do the same in 2.5 seconds. Parallelizing this isnt straightforward to me though, since the hash update is blocking

1 Reply

Doctor 🤖•3mo ago

import { encodeHex } from '@std/encoding'

const hash = encodeHex(await crypto.subtle.digest('SHA-256', await Deno.readFile(path)))

import { encodeHex } from '@std/encoding'

const hash = encodeHex(await crypto.subtle.digest('SHA-256', await Deno.readFile(path)))

The Web Crypto API does not seem to support streaming files in, and if you aren't consistent with the chunk sizes by rehashing for a readable manner, then you will get different results each time.