D
Deno

help

how to hash very large files

Aandykais1/17/2023
We have deprecated the old std/hash code in favor of the crypto module, but I do not know what the standard approach to hashing something piecemeal looks like. The advice around the crypto digest generally looks like this:
async function crypto_hash(filepath: string) {
const file_buffer = await Deno.readFile(filepath)
const hash_buffer = await crypto.subtle.digest('SHA-256', file_buffer)
const hash_array = Array.from(new Uint8Array(hash_buffer))
const hash_hex = hash_array.map((b) => b.toString(16).padStart(2, '0')).join('')
return hash_hex
}
async function crypto_hash(filepath: string) {
const file_buffer = await Deno.readFile(filepath)
const hash_buffer = await crypto.subtle.digest('SHA-256', file_buffer)
const hash_array = Array.from(new Uint8Array(hash_buffer))
const hash_hex = hash_array.map((b) => b.toString(16).padStart(2, '0')).join('')
return hash_hex
}
this works fine on smaller files, but I need to occasionally hash a big file, like 2GB files. This is a lot of data to hold in memory, and if I could leverage readable streams that would be ideal. If I need to do that, should I simply feed the hash_buffer back into itself, concatenating new data as I go?
IioB1/17/2023
See the conversation here https://discord.com/channels/684898665143206084/1064614652081885394/1064614652081885394 I had the same issue, the conclusion is I'm not quite sure how to do this with current web specs
Aandykais1/17/2023
hmm well its a question for everyone using the web, not just deno so there will definitely be an answer eventually Ill keep researching. Imo a readable stream interface in std/crypto would add a lot of value though
IioB1/17/2023
I believe webcrypto allows streaming of some sort but I forget
Aandykais1/17/2023
I found this, which seems to say it cant https://github.com/w3c/webcrypto/issues/73 I think heres the crux of it. I can make a very simple function that hashes my file chunkwise, appending files as I go. I will have consistency within my app, because I will always hash it the same way. However, that will not be the same standard sha256 that I might get out of the linux command line, or any number of tools. I need to reimplement the standard sha256 hash algorithm for chunks, which is going to take a bit of doing
IioB1/17/2023
that seems to be the case, yeah unfortunate when you just want to verify a checksum though
IioB1/17/2023
looks like https://github.com/wintercg/proposal-webcrypto-streams could be the solution to this problem long-term
GitHub
GitHub - wintercg/proposal-webcrypto-streams
Contribute to wintercg/proposal-webcrypto-streams development by creating an account on GitHub.
IioB1/17/2023
unfortunate that std/hash was removed before all usecases were fully considered
Aandykais1/17/2023
well, I suppose there is nothing broken in the older hashing code, its just wasm vs native so a tad slower https://deno.land/std@0.160.0/hash/mod.ts Ill probably just use this for now fwiw, web devs outside of deno have been feeling this pain as well https://lists.w3.org/Archives/Public/public-webcrypto/2016Nov/0000.html

Looking for more? Join the community!

Recommended Posts
xstate - no types importedimport * as xstate from "https://deno.land/x/xstate@xstate@4.35.2/src/index.ts"; file is cached, yetexample of kysely running on deno deploy?Deno Deploy awesomely can connect to postgres directly from the edge https://deno.com/blog/deploy-poUpdating digest with web cryptoFor some context, I'm updating some code that was importing from `std/hash` (specifically `std/hash/What would be the best third party module implementing `deflate()`?Currently using `npm:pako` but I'd like to avoid npm specifiers (modules in general).Pass string to struct FFIQuick question about Struct-Value feature, what would be the best way to pass `*const u8` (or stringdeno blog without html sanitizationI want to be able to javascript Canvas API things in my blog - surely this is possible?Killing subprocess not workingAnyone know what I'm doing wrong when trying to kill a process? I have a process start another procPossible to check of a file is being run as worker?```ts if (import.meta.main) { main() } ``` is there a way to disable calling `main()` if the file Deno linter is telling me that writeAll from streams conversion is deprecatedIs import { writeAll } from "https://deno.land/std@0.171.0/streams/conversion.ts" deprecated?How to get Deno's default compilerOptions value?I want to add more compilerOptions.lib options like this: ``` "compilerOptions": { "jsx": "readeno.run not working as expectedso when i run `mysql --defaults-extra-file=./tmpcnf_from_f_execute_query.cnf -e "DROP DATABASE test_Why doesn't latest Deno standard library for encoding CSV files support StringifyOptions?I need to remove headers see: https://deno.land/std@0.167.0/encoding/csv.ts?s=stringifyIs there any way to trace where permissions prompts are coming from?I'm running some code through Deno and its prompting for some permissions. How can I find out where modifying the Deno objectheyo. is there any way to add our own properties to the `Deno` object? I read something which said s