ventgrey•5mo ago

Help with FFI strings

I've been testing with the deno FFI to see if it's powerful enough to leverage the power of other languages into Deno. This however has been very unfruitful as FFI documentations seems to be "everywhere" when it comes to types, take for example anything that returns a char*. There are very few hints on how is one supposed to handle such returns. I was working with a function similar to this: extern char* searchByDinoSpecies(char* dinosaursString, char* species); You can assume the dinosaurString to be a JSON string or a YAML string, idk. Whatever your compiled C-grandson language lets you handle In my search I found this article: https://medium.com/deno-the-complete-reference/calling-c-functions-from-deno-part-2-pass-buffers-ad168a3b6cc7 However, two problems arise from here: 1. You cannot use the pointer type in parameters: [] when loading my dylib because I'll get a TypeError: Invalid FFI pointer type, expected null, or External Reading the official Deno docs, the documentation states:

buffer type accepts TypedArrays as parameter, but it always returns a pointer object or null when used as result type like the pointer type.

- Source: https://docs.deno.com/runtime/manual/runtime/ffi_api/ Okay, so maybe everything as a buffer should work? Well, both parameters sure work when setting buffer but another problem arises. The output, either pointer or buffer is completely useless. The previous Medium Post shows something like this:

const ret = dylib.symbols.toAndFroBuffer(buffer);
const dataView = new Deno.UnsafePointerView(ret);

const ret = dylib.symbols.toAndFroBuffer(buffer);
const dataView = new Deno.UnsafePointerView(ret);

However, when tried to replicate this, another TypeError arises:

Argument of type 'PointerValue<unknown>' is not assignable to parameter of type 'PointerObject<unknown>'.
  Type 'null' is not assignable to type 'PointerObject<unknown>'

Argument of type 'PointerValue<unknown>' is not assignable to parameter of type 'PointerObject<unknown>'.
  Type 'null' is not assignable to type 'PointerObject<unknown>'

Discord has an awfully lame limit, continuing in the comments (?)

Medium

Calling C functions from Deno: Part 2 — Pass buffers

Learn how to call C functions from Deno and pass arbitrary buffers to and fro

8 Replies

ventgreyOP•5mo ago

[CONTINUE 1] Then I decided to see how others do it and I ended up in the Bun docs. Which has an interesting take when it comes to strings (not providing link to avoid spamming more links) but goes like this: JavaScript strings and C-like strings are different, and that complicates using strings with native libraries. OK, so maybe a "CString" like Bun it's what I need. I found Deno issue #14248 where a Deno release is mentioned, and there, in a codeblock an example of such method is used and it requires yer another UnsafePointerView object, neither pointer nor buffer work for this approach. Which leads me to think, that nothing that returns char* is remotely usable in the FFI and should return a uint8 instead (?) https://docs.deno.com/api/deno/~/Deno.UnsafePointerView Docs also go nowhere when explaining such returned types. Don't get me wrong, this is probably a 99.9% skill issue by me. So, if returning any pointer of buffer from dylib is the same as doing nothing. Why is the PointerValue a returned thing if that is supposed to return a PointerObject? Why is that not documented? How can I help with that documentation if I manage to figure this out? Does a PointerValue instead of a PointerObject mean i messed up somewhere in the compiled language? I thought this was possible in the past because some stackoverflow answers point that pointer is old and work must be done with Uint8Array if you want to somehow handle the function results. When was this changed? was 1.14 the only release with changes to the FFI? Apologies in advance if my tone might be interpreted as un-supportive or if the text length is not adequate.

Deno AI Helper•5mo ago

Handling strings with Deno's FFI can indeed be a bit tricky, but let's see if we can clarify some of the confusion based on the documentation. Firstly, when dealing with functions that return a char* from a C library, you would typically use the pointer type in Deno's FFI to represent the returned value. As per the documentation, the buffer type is used for passing JavaScript TypedArrays as parameters to your FFI functions, and when used as a return type, it will return a pointer object or null. Here's a step-by-step approach to handle a function like searchByDinoSpecies: 1. Define the FFI bindings in Deno using Deno.dlopen with the buffer type for the input parameters and pointer for the return type:

const dylib = Deno.dlopen("path_to_library", {
  "searchByDinoSpecies": {
    parameters: ["buffer", "buffer"],
    result: "pointer",
  },
});

const dylib = Deno.dlopen("path_to_library", {
  "searchByDinoSpecies": {
    parameters: ["buffer", "buffer"],
    result: "pointer",
  },
});

2. When calling searchByDinoSpecies, you need to convert your input strings (JSON or YAML) into TypedArrays (Uint8Array), ensuring they are null-terminated C-style strings:

const encoder = new TextEncoder();
const dinosaurStringArray = encoder.encode(dinosaurString + "\0");
const speciesArray = encoder.encode(species + "\0");

const encoder = new TextEncoder();
const dinosaurStringArray = encoder.encode(dinosaurString + "\0");
const speciesArray = encoder.encode(species + "\0");

3. Call the function with these arrays:

const resultPointer = dylib.symbols.searchByDinoSpecies(dinosaurStringArray, speciesArray);

const resultPointer = dylib.symbols.searchByDinoSpecies(dinosaurStringArray, speciesArray);

4. To handle the returned char* (which is now a pointer object in Deno), you can use Deno.UnsafePointerView to read the string:

const resultView = new Deno.UnsafePointerView(resultPointer);
const resultString = resultView.getCString(); // This will give you the string

const resultView = new Deno.UnsafePointerView(resultPointer);
const resultString = resultView.getCString(); // This will give you the string

Regarding the error you encountered: ```typescript Argument of type 'PointerValue<unknown>' is not assignable

Deno AI Helper•5mo ago

to parameter of type 'PointerObject<unknown>'. Type 'null' is not assignable to type 'PointerObject<unknown>'

``

This seems to suggest that there's a mismatch between the expected types. Ensure that you are using the correct types as per the latest Deno FFI API. If

PointerValue` is being returned, it might be an outdated type or an issue with the type definitions. As of Deno 1.31, pointers are represented as plain objects or null for null pointers. If you're still encountering issues, it might be helpful to check the exact version of Deno you're using and ensure that your code aligns with the FFI API for that version. Remember that the FFI API is marked as unstable, and there may be changes between versions. Lastly, if you're still stuck, consider reaching out to the Deno

Helpful Links: Deno 1.17 Release Notes Deno 1.25 Release Notes Deno 1.31: package.json support Deno 1.15 Release Notes Foreign Function Interface How to go from a JS string[] to a C array of pointers - Deno

ventgreyOP•5mo ago

I'm already at that stage. Using text encoders to feed Uint8Array into the function buffer parameters. However, this response troubles me as step 4 is still the exact same type specifically a Deno.PointerValue<unknown>. Deno still whines about types when doing a new UnsafePointerView. Trying to change the PointerValue from unknown to anything else makes the type checker angry as well.

Deno AI Helper•5mo ago

I apologize for the confusion. It seems there might be a misunderstanding regarding the usage of Deno.UnsafePointerView. Based on your description, you are receiving a Deno.PointerValue<unknown> which should be a valid argument for creating a new Deno.UnsafePointerView. Here's a quick recap of how you should be able to use Deno.UnsafePointerView with a pointer returned from an FFI call:

const resultPointer = dylib.symbols.searchByDinoSpecies(dinosaurStringArray, speciesArray);
const resultView = new Deno.UnsafePointerView(resultPointer);
const resultString = resultView.getCString();

const resultPointer = dylib.symbols.searchByDinoSpecies(dinosaurStringArray, speciesArray);
const resultView = new Deno.UnsafePointerView(resultPointer);
const resultString = resultView.getCString();

If you're getting a type error when trying to pass resultPointer to new Deno.UnsafePointerView, it's possible that there might be a version mismatch or an issue with the Deno types. Make sure you are using the latest version of Deno, as the FFI API is still considered unstable and may have undergone changes. If the issue persists, it could be a bug or a change in the FFI API that is not yet reflected in the documentation. In such cases, it's recommended to check the release notes for the version of Deno you are using or to look for any recent issues or discussions in the Deno repository on GitHub. Please note that this advice is not based on the Deno documentation, and you should double-check the accuracy of the advice given. If you continue to experience difficulties, consider reaching out to the Deno community for further assistance.

Helpful Links: Deno 1.25 Release Notes Deno 1.17 Release Notes Deno.UnsafePointerView.prototype.getPointer Deno.UnsafePointerView /io/mod.ts | std@0.207.0 | Deno index Deno 1.38: HTML doc generator and HMR Hex and Base64 Encoding - Deno by Example

ventgreyOP•5mo ago

I'm sorry but deno check states otherwise:

TS2345 [ERROR]: Argument of type 'PointerValue<unknown>' is not assignable to parameter of type 'PointerObject<unknown>'.
  Type 'null' is not assignable to type 'PointerObject<unknown>'.
    const ret = new Deno.UnsafePointerView(resultPtr);

TS2345 [ERROR]: Argument of type 'PointerValue<unknown>' is not assignable to parameter of type 'PointerObject<unknown>'.
  Type 'null' is not assignable to type 'PointerObject<unknown>'.
    const ret = new Deno.UnsafePointerView(resultPtr);

Deno AI Helper•5mo ago

I understand that you're encountering a TypeScript error when trying to pass a PointerValue<unknown> to the Deno.UnsafePointerView constructor. The error message suggests that TypeScript is expecting a PointerObject<unknown> instead, and it's not allowing null to be passed. From the error message you've provided, it seems like there might be a type mismatch or an issue with the handling of potential null values. Without the ability to consult the official Deno documentation for an answer to this specific issue, I can only provide general advice based on the error message itself. One thing you could try is to check if the resultPtr is not null before attempting to create a new UnsafePointerView. Here's an example of how you might guard against null:

if (resultPtr !== null) {
  const ret = new Deno.UnsafePointerView(resultPtr);
  // ... use `ret` as needed
} else {
  // Handle the case where `resultPtr` is `null`
}

if (resultPtr !== null) {
  const ret = new Deno.UnsafePointerView(resultPtr);
  // ... use `ret` as needed
} else {
  // Handle the case where `resultPtr` is `null`
}

Please note that this advice is not based on the Deno documentation, and you should double-check the accuracy of the advice given. If the issue persists, it may be beneficial to look for recent changes or discussions in the Deno repository on GitHub, or to reach out to the Deno community for further assistance.

Helpful Links: Deno 1.31: package.json support Deno 1.20 Release Notes Deno 1.17 Release Notes Foreign Function Interface Deno 1.23 Release Notes Key Space | Deno Docs Configuring TypeScript in Deno

ventgreyOP•5mo ago

Well..that solved it 🤡 . For anyone that may get stuck with this in the future, here is the whole code for Deno + Hono API endpoint using the dinosaurs example earlier:

// src/routes/denos/mod.ts
import { Context, Hono } from '@hono/hono'; // <-- Install with "deno add @hono/hono"
import { readDinosaursJSON } from '$lib/files'; // <-- Implement your own (use a JSON generator online)
import logger from '$lib/logger'; // <-- Implement your own (example uses winston)
import { libGoDenos } from '$lib/ffi';

const app: Hono = new Hono();

app.get('/species/:name', async (c: Context) => {
    const name: string = c.req.param('name');
    const denos: string = await readDinosaursJSON('denos');

    const encodedName: Uint8Array = new TextEncoder().encode(name);
    const encodedDenos: Uint8Array = new TextEncoder().encode(denos + '\0');

    const resultPtr: Deno.PointerValue<unknown> = libGoDenos.symbols.SearchByDenoSpecies(
        encodedDenos,
        encodedName,
    );

    // if (resultPtr) {...}
    if (resultPtr !== null) {
        const ret: Deno.UnsafePointerView = new Deno.UnsafePointerView(resultPtr);
        const resultString: string = ret.getCString();

        c.status(200);
        return c.json(JSON.parse(resultString));

    } else {
        logger.warn(
            `Got null result pointer when searching for denos by species: ${name} using the FFI`,
        );

        c.status(404);
        return c.json({
            error: 'No denos found',
            message: 'FFI error',
        });
    }
});

export default app;

// src/routes/denos/mod.ts
import { Context, Hono } from '@hono/hono'; // <-- Install with "deno add @hono/hono"
import { readDinosaursJSON } from '$lib/files'; // <-- Implement your own (use a JSON generator online)
import logger from '$lib/logger'; // <-- Implement your own (example uses winston)
import { libGoDenos } from '$lib/ffi';

const app: Hono = new Hono();

app.get('/species/:name', async (c: Context) => {
    const name: string = c.req.param('name');
    const denos: string = await readDinosaursJSON('denos');

    const encodedName: Uint8Array = new TextEncoder().encode(name);
    const encodedDenos: Uint8Array = new TextEncoder().encode(denos + '\0');

    const resultPtr: Deno.PointerValue<unknown> = libGoDenos.symbols.SearchByDenoSpecies(
        encodedDenos,
        encodedName,
    );

    // if (resultPtr) {...}
    if (resultPtr !== null) {
        const ret: Deno.UnsafePointerView = new Deno.UnsafePointerView(resultPtr);
        const resultString: string = ret.getCString();

        c.status(200);
        return c.json(JSON.parse(resultString));

    } else {
        logger.warn(
            `Got null result pointer when searching for denos by species: ${name} using the FFI`,
        );

        c.status(404);
        return c.json({
            error: 'No denos found',
            message: 'FFI error',
        });
    }
});

export default app;

This is the Golang code used to generate the dynamic library for the Denosaurs example:

package main

import (
    "encoding/json"
    "fmt"
    "unsafe"
)

/*
#include <stdlib.h>
*/
import "C"

//export Denosaur
type Denosaur struct {
    Species  string `json:"species"`
    Colour  string `json:"colour"`
    Discovered string `json:"discovered"`
}

//export SearchByDenoSpecies
func SearchByDenoSpecies(denosJSON *C.char, speciesName *C.char) *C.char {
    var denos []Denosaur

    err := json.Unmarshal([]byte(C.GoString(denosJSON)), &posts)
    if err != nil {
        return C.CString(fmt.Sprintf("Error: %s", err)) // <-- Memory leak :(
    }

    searchSpecies := C.GoString(speciesName)

    for _, deno := range denos {
        if deno.Species == searchSpecies {
            result, _ := json.Marshal(deno)
            return C.CString(string(result)) // <-- Another memory leak :((
        }
    }

    var notFountStr *C.char = C.CString(fmt.Sprintf("Not found: %s", searchSpecies))
    defer C.free(unsafe.Pointer(notFountStr)) // <-- Memory Leakn't :)
    return notFountStr
}

func main() {}

package main

import (
    "encoding/json"
    "fmt"
    "unsafe"
)

/*
#include <stdlib.h>
*/
import "C"

//export Denosaur
type Denosaur struct {
    Species  string `json:"species"`
    Colour  string `json:"colour"`
    Discovered string `json:"discovered"`
}

//export SearchByDenoSpecies
func SearchByDenoSpecies(denosJSON *C.char, speciesName *C.char) *C.char {
    var denos []Denosaur

    err := json.Unmarshal([]byte(C.GoString(denosJSON)), &posts)
    if err != nil {
        return C.CString(fmt.Sprintf("Error: %s", err)) // <-- Memory leak :(
    }

    searchSpecies := C.GoString(speciesName)

    for _, deno := range denos {
        if deno.Species == searchSpecies {
            result, _ := json.Marshal(deno)
            return C.CString(string(result)) // <-- Another memory leak :((
        }
    }

    var notFountStr *C.char = C.CString(fmt.Sprintf("Not found: %s", searchSpecies))
    defer C.free(unsafe.Pointer(notFountStr)) // <-- Memory Leakn't :)
    return notFountStr
}

func main() {}

you may compile this using: go build -o ./<output_dir> -buildmode=c-shared .<input_go_file.