how to hash very large files

We have deprecated the old std/hash code in favor of the crypto module, but I do not know what the standard approach to hashing something piecemeal looks like. The advice around the crypto digest generally looks like this:

async function crypto_hash(filepath: string) {
  const file_buffer = await Deno.readFile(filepath)
  const hash_buffer = await crypto.subtle.digest('SHA-256', file_buffer)
  const hash_array = Array.from(new Uint8Array(hash_buffer))
  const hash_hex = hash_array.map((b) => b.toString(16).padStart(2, '0')).join('')
  return hash_hex
}

async function crypto_hash(filepath: string) {
  const file_buffer = await Deno.readFile(filepath)
  const hash_buffer = await crypto.subtle.digest('SHA-256', file_buffer)
  const hash_array = Array.from(new Uint8Array(hash_buffer))
  const hash_hex = hash_array.map((b) => b.toString(16).padStart(2, '0')).join('')
  return hash_hex
}

this works fine on smaller files, but I need to occasionally hash a big file, like 2GB files. This is a lot of data to hold in memory, and if I could leverage readable streams that would be ideal. If I need to do that, should I simply feed the

hash_buffer

hash_buffer

back into itself, concatenating new data as I go?

Deno•4y ago•

14 replies

andykais

how to hash very large files

async function crypto_hash(filepath: string) {
  const file_buffer = await Deno.readFile(filepath)
  const hash_buffer = await crypto.subtle.digest('SHA-256', file_buffer)
  const hash_array = Array.from(new Uint8Array(hash_buffer))
  const hash_hex = hash_array.map((b) => b.toString(16).padStart(2, '0')).join('')
  return hash_hex
}

async function crypto_hash(filepath: string) {
  const file_buffer = await Deno.readFile(filepath)
  const hash_buffer = await crypto.subtle.digest('SHA-256', file_buffer)
  const hash_array = Array.from(new Uint8Array(hash_buffer))
  const hash_hex = hash_array.map((b) => b.toString(16).padStart(2, '0')).join('')
  return hash_hex
}

hash_buffer

hash_buffer

back into itself, concatenating new data as I go?

how to hash very large files

Similar Threads

how to hash very large files

Similar Threads

Similar Threads

Similar Threads