fro.profesional•9mo ago

How to encrypt and decrypt files from S3 without loading the hole content into memory?

How to encrypt and decrypt files from S3 without loading the hole content into memory? I have this basic example (example in comments), I wonder is some of this can be used.

3 Replies

fro.profesionalOP•9mo ago

// Encrypts data using AES-GCM
async function encryptData(data: string, key: CryptoKey) {
  const encoder = new TextEncoder();
  const iv = crypto.getRandomValues(new Uint8Array(12)); // Initialization Vector
  const encrypted = await crypto.subtle.encrypt(
    {
      name: "AES-GCM",
      iv: iv,
    },
    key,
    encoder.encode(data),
  );

  return { encrypted, iv };
}

// Decrypts data using AES-GCM
async function decryptData(
  encrypted: ArrayBuffer,
  iv: Uint8Array,
  key: CryptoKey,
) {
  const decrypted = await crypto.subtle.decrypt(
    {
      name: "AES-GCM",
      iv: iv,
    },
    key,
    encrypted,
  );

  const decoder = new TextDecoder();
  return decoder.decode(decrypted);
}

async function generateKeyString(): Promise<string> {
  try {
    const key = await crypto.subtle.generateKey(
      {
        name: "AES-GCM",
        length: 256,
      },
      true,
      ["encrypt", "decrypt"],
    );

    const exportedKey = await crypto.subtle.exportKey("jwk", key);
    return JSON.stringify(exportedKey);
  } catch (error) {
    console.error("Error generating key:", error);
    return "";
  }
}

async function generateKeyFromString(
  keyString: string,
): Promise<CryptoKey> {
  const keyData = JSON.parse(keyString);
  const key = await crypto.subtle.importKey(
    "jwk",
    keyData,
    { name: "AES-GCM", length: 256 },
    true,
    ["encrypt", "decrypt"],
  );
  return key;
}

// Example usage
async function example() {
  const keyString = await generateKeyString();
  const cryptoKey = await generateKeyFromString(keyString)!;

  const stringToEncrypt = "Hello, world!";

  const {
    // Save the encrypted data and the IV to DB
    encrypted,
    iv,
  } = await encryptData(stringToEncrypt, cryptoKey);

  const decryptedMessage = await decryptData(encrypted, iv, cryptoKey);

  console.log("Original:", stringToEncrypt);
  console.log("Encrypted:", encrypted);
  console.log("Decrypted:", decryptedMessage);
}

example();

// Encrypts data using AES-GCM
async function encryptData(data: string, key: CryptoKey) {
  const encoder = new TextEncoder();
  const iv = crypto.getRandomValues(new Uint8Array(12)); // Initialization Vector
  const encrypted = await crypto.subtle.encrypt(
    {
      name: "AES-GCM",
      iv: iv,
    },
    key,
    encoder.encode(data),
  );

  return { encrypted, iv };
}

// Decrypts data using AES-GCM
async function decryptData(
  encrypted: ArrayBuffer,
  iv: Uint8Array,
  key: CryptoKey,
) {
  const decrypted = await crypto.subtle.decrypt(
    {
      name: "AES-GCM",
      iv: iv,
    },
    key,
    encrypted,
  );

  const decoder = new TextDecoder();
  return decoder.decode(decrypted);
}

async function generateKeyString(): Promise<string> {
  try {
    const key = await crypto.subtle.generateKey(
      {
        name: "AES-GCM",
        length: 256,
      },
      true,
      ["encrypt", "decrypt"],
    );

    const exportedKey = await crypto.subtle.exportKey("jwk", key);
    return JSON.stringify(exportedKey);
  } catch (error) {
    console.error("Error generating key:", error);
    return "";
  }
}

async function generateKeyFromString(
  keyString: string,
): Promise<CryptoKey> {
  const keyData = JSON.parse(keyString);
  const key = await crypto.subtle.importKey(
    "jwk",
    keyData,
    { name: "AES-GCM", length: 256 },
    true,
    ["encrypt", "decrypt"],
  );
  return key;
}

// Example usage
async function example() {
  const keyString = await generateKeyString();
  const cryptoKey = await generateKeyFromString(keyString)!;

  const stringToEncrypt = "Hello, world!";

  const {
    // Save the encrypted data and the IV to DB
    encrypted,
    iv,
  } = await encryptData(stringToEncrypt, cryptoKey);

  const decryptedMessage = await decryptData(encrypted, iv, cryptoKey);

  console.log("Original:", stringToEncrypt);
  console.log("Encrypted:", encrypted);
  console.log("Decrypted:", decryptedMessage);
}

example();

Deno AI Helper•9mo ago

I wasn't able to find the answer to your question in the official Deno documentation. However, based on your message, I can provide some general advice. When dealing with large files and trying to avoid loading the entire content into memory, you would typically use streams. Streams allow you to process data incrementally as it is being read or written, which is ideal for handling large files. In Deno, you can use the reader and writer interfaces along with the copy function from the std/streams module to work with streams. If you have an example in the comments, please ensure that it uses streaming methods to read from and write to S3. Please note that this advice is not based on the Deno documentation, and I would advise you to double-check the accuracy of this advice. If you're using a third-party library for S3 interactions, make sure to consult the library's documentation for streaming support.

Helpful Links: Hashing - Deno by Example Backups | Deno Docs Deno Docs Hex and Base64 Encoding - Deno by Example

pyrote•9mo ago

The Web Crypto API that Deno uses has no support for streams so I don't think its possible with the basic crypto.subtle. You could break the data into chunks and encrypt each chunk separately but then you are basically creating your own file format that only your code understands. Chunking can also potentially weaken the encryption if you don't handle IV generation and auth tags properly. I would look for a encryption library that can handle streams. Node's crypto module has support for ReadStreams so maybe Deno's Node compatibility is far enough along that it supports streams.