Kalleby Santos Posts

Kalleby Santos

•Created by Kalleby Santos on 1/24/2025 in #help

How to extend `serde_v8` to handle custom logic?

Hi guys! I'm currently I have the following struct to represent my Tensor object:

// Type alias for TensorElementType (External lib proxy)
// https://serde.rs/remote-derive.html
#[derive(Debug, Serialize, Deserialize)]
#[serde(remote = "TensorElementType", rename_all = "lowercase")]
pub enum JsTensorType {
    /// 32-bit floating point number, equivalent to Rust's `f32`.
    Float32,
    /// Unsigned 8-bit integer, equivalent to Rust's `u8`.
    Uint8,
    /// String, equivalent to Rust's `String`.
    String,
    ...
}

#[derive(Serialize, Deserialize)]
struct JsTensorTypeSerdeHelper(#[serde(with = "JsTensorType")] TensorElementType);

#[derive(Debug, Serialize, Deserialize)]
pub struct JsTensor {
    #[serde(rename = "type", with = "JsTensorType")]
    data_type: TensorElementType,
    data: JsBuffer,
    dims: Vec<i64>,
}

// Type alias for TensorElementType (External lib proxy)
// https://serde.rs/remote-derive.html
#[derive(Debug, Serialize, Deserialize)]
#[serde(remote = "TensorElementType", rename_all = "lowercase")]
pub enum JsTensorType {
    /// 32-bit floating point number, equivalent to Rust's `f32`.
    Float32,
    /// Unsigned 8-bit integer, equivalent to Rust's `u8`.
    Uint8,
    /// String, equivalent to Rust's `String`.
    String,
    ...
}

#[derive(Serialize, Deserialize)]
struct JsTensorTypeSerdeHelper(#[serde(with = "JsTensorType")] TensorElementType);

#[derive(Debug, Serialize, Deserialize)]
pub struct JsTensor {
    #[serde(rename = "type", with = "JsTensorType")]
    data_type: TensorElementType,
    data: JsBuffer,
    dims: Vec<i64>,
}

It's working well with #[serde] attribute on my op function. The problem is that I need to extend it to also support string arrays like: ["foo", "bar"], and I can't handle it as JsBuffer so I need to explicit do pattern matching:

enum JsTensorData {
    StringArray(Vec<String>),
    TypedArrayBuffer(JsBuffer),
}

pub struct JsTensor {
    data_type: TensorElementType,
    data: JsTensorData,
    dims: Vec<i64>,
}

enum JsTensorData {
    StringArray(Vec<String>),
    TypedArrayBuffer(JsBuffer),
}

pub struct JsTensor {
    data_type: TensorElementType,
    data: JsTensorData,
    dims: Vec<i64>,
}

--- From the Js land I need to pass an object with the given format, where TypedArray could be any theses:

type Tensor = {
  type: 'float32' | 'uint8' | 'string' // etc...
  data: TypedArray | string[]
  dims: number[]
}

type Tensor = {
  type: 'float32' | 'uint8' | 'string' // etc...
  data: TypedArray | string[]
  dims: number[]
}

The problem is that even if I use #[serde(untagged)] :

#[derive(Debug, Serialize, Deserialize)]
#[serde(untagged)]
enum JsTensorData {
    StringArray(Vec<String>),
    TypedArrayBuffer(JsBuffer),
}

#[derive(Debug, Serialize, Deserialize)]
#[serde(untagged)]
enum JsTensorData {
    StringArray(Vec<String>),
    TypedArrayBuffer(JsBuffer),
}

I can't handle JsBuffer, cause of the following error:

TypeError: data did not match any variant of untagged enum JsTensorData

TypeError: data did not match any variant of untagged enum JsTensorData

2 replies

DDeno

•Created by Kalleby Santos on 9/24/2024 in #help

RustyV8: Example of creating a TypedArray?

Hey there, I'm currently trying to Zero-copy my ort::Tensor to a v8::TypedArray, but I can't figure out how to work with the byte_offset and length parameters of v8::Float32Array or other TypedArray variant. Here is an example of my code:

// ... model init
// ... v8 scope

let mut outputs = session.run(inputs![/* etc... */]);

// I do remove to get ownership of the value, discard the key since I don't need it
let (_key, last_hidden) = outputs.remove('last_hidden_state');

/** Then I get an `ArrayBuffer` from the inner `ptr` of my tensor **/
// I could try `Tensor::try_extract_raw_tensor_mut<T>` with `v8::ArrayBuffer::new_backing_store_from_bytes`
// but it only allows [u8] instead of [T], so we need to get into `unsafe` path.
let (_, raw_tensor) = last_hidden
    .try_extract_raw_tensor_mut::<f32>()
    .map_err(AnyError::from)?;

println!("tensor: len: {} -> {raw_tensor:?}", raw_tensor.len());
// $ "tensor: len: 3456 -> [0.22348395, 0.13147816, 0.32118404, 0.13782105, 0.10047962, -0.2339811, ...]"

// We zero-copy an ORT Tensor to a JS ArrayBuffer like:
// https://github.com/denoland/deno_core/blob/7258aa325368a8e2c1271a25c1b4d537ed41e9c5/core/runtime/ops_rust_to_v8.rs#L370
let tensor_ptr = raw_tensor.as_ptr();
let tensor_len = raw_tensor.len();
let tensor_rc = Rc::into_raw(Rc::new(raw_tensor)) as *const c_void;

extern "C" fn drop_tensor(_ptr: *mut c_void, _len: usize, data: *mut c_void) {
    // SAFETY: We know that data is a raw Rc from above
    unsafe { drop(Rc::from_raw(data.cast::<f32>())) }
}

let buf_store = unsafe {
    v8::ArrayBuffer::new_backing_store_from_ptr(
        tensor_ptr as _,
        tensor_len,
        drop_tensor,
        tensor_rc as _,
    )
}
.make_shared();

let array_buffer = v8::ArrayBuffer::with_backing_store(scope, &buf_store)

// ... model init
// ... v8 scope

let mut outputs = session.run(inputs![/* etc... */]);

// I do remove to get ownership of the value, discard the key since I don't need it
let (_key, last_hidden) = outputs.remove('last_hidden_state');

/** Then I get an `ArrayBuffer` from the inner `ptr` of my tensor **/
// I could try `Tensor::try_extract_raw_tensor_mut<T>` with `v8::ArrayBuffer::new_backing_store_from_bytes`
// but it only allows [u8] instead of [T], so we need to get into `unsafe` path.
let (_, raw_tensor) = last_hidden
    .try_extract_raw_tensor_mut::<f32>()
    .map_err(AnyError::from)?;

println!("tensor: len: {} -> {raw_tensor:?}", raw_tensor.len());
// $ "tensor: len: 3456 -> [0.22348395, 0.13147816, 0.32118404, 0.13782105, 0.10047962, -0.2339811, ...]"

// We zero-copy an ORT Tensor to a JS ArrayBuffer like:
// https://github.com/denoland/deno_core/blob/7258aa325368a8e2c1271a25c1b4d537ed41e9c5/core/runtime/ops_rust_to_v8.rs#L370
let tensor_ptr = raw_tensor.as_ptr();
let tensor_len = raw_tensor.len();
let tensor_rc = Rc::into_raw(Rc::new(raw_tensor)) as *const c_void;

extern "C" fn drop_tensor(_ptr: *mut c_void, _len: usize, data: *mut c_void) {
    // SAFETY: We know that data is a raw Rc from above
    unsafe { drop(Rc::from_raw(data.cast::<f32>())) }
}

let buf_store = unsafe {
    v8::ArrayBuffer::new_backing_store_from_ptr(
        tensor_ptr as _,
        tensor_len,
        drop_tensor,
        tensor_rc as _,
    )
}
.make_shared();

let array_buffer = v8::ArrayBuffer::with_backing_store(scope, &buf_store)

3 replies

DDeno

•Created by Kalleby Santos on 9/12/2024 in #help

Embedding Deno: How to add external dependencies to esm file?

1 replies