AapoAlas Comments

Re: The WeakMap. It should actually be created outside of the function scope, so that it itself cannot be GC'd. You can then start code-golfing to something like (pseudo-code incoming because I'm on mobile):

const BUF_TO_STRING_BUFS_MAP = new WeakMap<Uint8Array, Uint8Array[]>();
function toCStringArray(strings: string[]) {
  const buf = new BigUint64Array(strings.length);
  const chunks = strings.map((str, I) => {
    const chunk = encode(str + "\0");
    buf[i] = Deno.UnsafePointer.valueOf(chunk);
    return chunk;
  });
  BUF_TO_STRINGS_BUF_MAP.set(buf, chunks);
  return new Uint8Array(buf.buffer);
}

const BUF_TO_STRING_BUFS_MAP = new WeakMap<Uint8Array, Uint8Array[]>();
function toCStringArray(strings: string[]) {
  const buf = new BigUint64Array(strings.length);
  const chunks = strings.map((str, I) => {
    const chunk = encode(str + "\0");
    buf[i] = Deno.UnsafePointer.valueOf(chunk);
    return chunk;
  });
  BUF_TO_STRINGS_BUF_MAP.set(buf, chunks);
  return new Uint8Array(buf.buffer);
}

21 replies

DDeno

•Created by carragom on 4/9/2024 in #help

How to go from a JS string[] to a C array of pointers

And there is indeed an advantage to using Uint8Array as your go-to-choice whenever you pass data into FFI as "buffer": Due to reasons, Deno has to choose one buffer type to optimise for, and Uint8Array is the most versatile one we can choose so that one it is. As a result, with a trivial benchmark you should see perhaps even a 100x difference in performance between the exact same FFI call but just changing from a BigUint64Array into a Uint8Array.

21 replies

DDeno

•Created by carragom on 4/9/2024 in #help

How to go from a JS string[] to a C array of pointers

Return is reusing the underlying buffer.

21 replies

DDeno

•Created by carragom on 4/9/2024 in #help

How to go from a JS string[] to a C array of pointers

Your WeakMap should be a map from your buf into chunks. Right now you're saying "until ptr exists, chunk must exist." Then you immediately let the ptr object go out of scope and potentially get garbage collected. The chunk is then free to be GC'd as well.

21 replies

DDeno

•Created by carragom on 4/9/2024 in #help

How to go from a JS string[] to a C array of pointers

Deno internally does fun stuff with a WeakMap in Deno.UnsafePointer.of(arrayBuffer). This returns an opaque pointer object and we put it in as a key into a WeakMap with arrayBuffer as the value. The result is that as long as the pointer object lives, the buffer memory also lives.

21 replies

DDeno

•Created by carragom on 4/9/2024 in #help

How to go from a JS string[] to a C array of pointers

Alternatively, you can just encode the strings one by one and create pointers to those resulting buffers. Then, allocate an array of pointers and write the pointers into that. Finally, use a WeakMap to bind the string-buffers to stay alive as long as the pointers-buffer is alive. This gets rid of the extra copy but retains the N+1 allocations.

21 replies

DDeno

•Created by carragom on 4/9/2024 in #help

How to go from a JS string[] to a C array of pointers

This does suffer from doing two memory copies for the strings instead of one. Also you're doing N+1 memory allocations where N is the number of strings. You could calculate the encoded length from the string lengths times some constant. Any constant below 3 (IIRC) does mean that it's theoretically possible for the estimated encoded length to be too small, but that is fairly unlikely. This would mean no extra memory copy and only one allocation. If the buffer ends up being too small, you can detect that based on the encoder's return value (bytes read, bytes written) and create a new, larger buffer to replace the first one. This would be an extra memory copy but it's very, very unlikely to happen.

21 replies

DDeno

•Created by carragom on 4/9/2024 in #help

How to go from a JS string[] to a C array of pointers

And here's a usage example: https://github.com/aapoalas/libclang_deno/blob/5d6d8ed33627c6f12949efc15c3f7ccf07f26b3a/lib/mod.ts#L364-L380

21 replies

DDeno

•Created by carragom on 4/9/2024 in #help

How to go from a JS string[] to a C array of pointers

The basic idea is this: Create a buffer that is large enough to hold params.length pointers plus all of the param strings encoded. Then encode the strings into the buffer and write in pointers that point to the start of each string (so essentially self-references; the buffer contains pointers that point within the buffer).

21 replies

DDeno

•Created by carragom on 4/9/2024 in #help

How to go from a JS string[] to a C array of pointers

Here's an example of how I pass string args: https://github.com/aapoalas/libclang_deno/blob/main/lib/utils.ts#L9-L46

21 replies

DDeno

•Created by kitty on 4/8/2024 in #help

Deno.UnsafeCallback Error (bypass string from c++ to deno)

No problem. Let me know if you find any more questions about FFI 🙂 Also if you didn't know yet, check out Denonomicon, and the WTF is FFI videos on Youtube

9 replies

DDeno

•Created by RJ Budzyński on 4/7/2024 in #help

deno notebook can't import skia_canvas

Were you possibly using the general --unstable flag back then?

24 replies

DDeno

•Created by RJ Budzyński on 4/7/2024 in #help

deno notebook can't import skia_canvas

(Specifically I remember someone asking why print to stdout from within FFI libs aren't appearing in notebook output. That's still not done though.)

24 replies

DDeno

•Created by RJ Budzyński on 4/7/2024 in #help

deno notebook can't import skia_canvas

I think I remember people using FFI from notebooks somehow. Probably just a matter of figuring out how the unstable-ffi parameter should be passed in.

24 replies

DDeno

•Created by RJ Budzyński on 4/7/2024 in #help

deno notebook can't import skia_canvas

🤔 Honestly not sure.

24 replies

DDeno

•Created by RJ Budzyński on 4/7/2024 in #help

deno notebook can't import skia_canvas

https://stackoverflow.com/questions/76974555/glibcxx-3-4-32-not-found-error-at-runtime-gcc-13-2-0

24 replies