Rune•5mo ago

Parsing Typescript with deno_core and deno_ast fails on type annotations

Hey all, i'm trying to embed deno into my app so that users can write config file in typescript. I got inspired by the "Roll your own JavaScript runtime" blog series, but they seem to be a bit outdated... The examples in the deno_core repo are not perfect either. I'm trying to write the following code:

((globalThis) => {
    globalThis.Math = {
        sum: (numbers: number[]): number => Deno.core.ops.op_sum(numbers),
        min: (numbers: number[]): number => Deno.core.ops.op_min(numbers)
    }
})(globalThis);

((globalThis) => {
    globalThis.Math = {
        sum: (numbers: number[]): number => Deno.core.ops.op_sum(numbers),
        min: (numbers: number[]): number => Deno.core.ops.op_min(numbers)
    }
})(globalThis);

When running this as javascript, without type annotations, it works just fine. However, when i try to add the ts loader from the example (with some slight modifications), it errors on the type annotations:

called `Result::unwrap()` on an `Err` value: Uncaught SyntaxError: Unexpected token ':'
    at [runjs:runtime.ts]:3:22

called `Result::unwrap()` on an `Err` value: Uncaught SyntaxError: Unexpected token ':'
    at [runjs:runtime.ts]:3:22

These are my dependencies:

deno_core = { version = "0.291", features = ["include_icu_data"] }
deno_ast = { version = "0.39", features = ["emit", "transpiling", "sourcemap", "typescript", "transforms", "compat"] }
anyhow = "1"

deno_core = { version = "0.291", features = ["include_icu_data"] }
deno_ast = { version = "0.39", features = ["emit", "transpiling", "sourcemap", "typescript", "transforms", "compat"] }
anyhow = "1"

This is my main file

mod ts_loader;

use deno_core::error::AnyError;
use deno_core::*;
use std::cell::RefCell;
use std::collections::HashMap;
use std::rc::Rc;

const RUNTIME_SETUP: &str = include_str!("runtime.ts");
const EXAMPLE_SCRIPT: &str = include_str!("../example.ts");

#[op2]
fn op_sum(#[serde] nums: Vec<f64>) -> Result<f64, AnyError> {
    // Sum inputs
    let sum = nums.iter().fold(0.0, |a, v| a + v);
    // return as a Result<f64, AnyError>
    Ok(sum)
}

#[op2]
fn op_min(#[serde] nums: Vec<f64>) -> Result<f64, AnyError> {
    // Sum inputs
    let sum = nums.iter().min_by(|a, b| a.total_cmp(b));
    match sum {
        Some(x) => Ok(*x),
        None => Ok(f64::NAN),
    }
}

async fn run_js(script: String) -> Result<(), AnyError> {
    const SUM_EXT: OpDecl = op_sum();
    let sum_ext = Extension {
        name: "sum_ext",
        ops: std::borrow::Cow::Borrowed(&[SUM_EXT]),
        ..Default::default()
    };

    const MIN_EXT: OpDecl = op_min();
    let min_ext = Extension {
        name: "min_ext",
        ops: std::borrow::Cow::Borrowed(&[MIN_EXT]),
        ..Default::default()
    };

    let source_map_store = ts_loader::SourceMapStore(Rc::new(RefCell::new(HashMap::new())));

    // Initialize a runtime instance
    let mut runtime = JsRuntime::new(RuntimeOptions {
        extensions: vec![sum_ext, min_ext],
        module_loader: Some(Rc::new(ts_loader::TypescriptModuleLoader {
            source_maps: source_map_store.clone(),
        })),
        source_map_getter: Some(Rc::new(source_map_store)),
        ..Default::default()
    });

    runtime
        .execute_script("[runjs:runtime.ts]", RUNTIME_SETUP)
        .unwrap();
    runtime
        .execute_script("[evaluate_manifest]", script)
        .unwrap();

    Ok(())
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn it_works() {
        match run_js(EXAMPLE_SCRIPT.to_string()).await {
            Ok(_) => assert!(true),
            Err(reason) => assert!(false, "{}", reason),
        }
    }
}

mod ts_loader;

use deno_core::error::AnyError;
use deno_core::*;
use std::cell::RefCell;
use std::collections::HashMap;
use std::rc::Rc;

const RUNTIME_SETUP: &str = include_str!("runtime.ts");
const EXAMPLE_SCRIPT: &str = include_str!("../example.ts");

#[op2]
fn op_sum(#[serde] nums: Vec<f64>) -> Result<f64, AnyError> {
    // Sum inputs
    let sum = nums.iter().fold(0.0, |a, v| a + v);
    // return as a Result<f64, AnyError>
    Ok(sum)
}

#[op2]
fn op_min(#[serde] nums: Vec<f64>) -> Result<f64, AnyError> {
    // Sum inputs
    let sum = nums.iter().min_by(|a, b| a.total_cmp(b));
    match sum {
        Some(x) => Ok(*x),
        None => Ok(f64::NAN),
    }
}

async fn run_js(script: String) -> Result<(), AnyError> {
    const SUM_EXT: OpDecl = op_sum();
    let sum_ext = Extension {
        name: "sum_ext",
        ops: std::borrow::Cow::Borrowed(&[SUM_EXT]),
        ..Default::default()
    };

    const MIN_EXT: OpDecl = op_min();
    let min_ext = Extension {
        name: "min_ext",
        ops: std::borrow::Cow::Borrowed(&[MIN_EXT]),
        ..Default::default()
    };

    let source_map_store = ts_loader::SourceMapStore(Rc::new(RefCell::new(HashMap::new())));

    // Initialize a runtime instance
    let mut runtime = JsRuntime::new(RuntimeOptions {
        extensions: vec![sum_ext, min_ext],
        module_loader: Some(Rc::new(ts_loader::TypescriptModuleLoader {
            source_maps: source_map_store.clone(),
        })),
        source_map_getter: Some(Rc::new(source_map_store)),
        ..Default::default()
    });

    runtime
        .execute_script("[runjs:runtime.ts]", RUNTIME_SETUP)
        .unwrap();
    runtime
        .execute_script("[evaluate_manifest]", script)
        .unwrap();

    Ok(())
}

#[cfg(test)]
mod tests {
    use super::*;

    #[tokio::test]
    async fn it_works() {
        match run_js(EXAMPLE_SCRIPT.to_string()).await {
            Ok(_) => assert!(true),
            Err(reason) => assert!(false, "{}", reason),
        }
    }
}

The loader:

message.txt

2 Replies

Deno AI Helper•5mo ago

It seems like you're encountering issues with parsing TypeScript code using deno_core and deno_ast. The error message you're seeing indicates that the TypeScript type annotations are not being correctly handled, which is causing a syntax error. The documentation mentions that transpiling TypeScript to JavaScript is not a trivial task. It involves parsing the TypeScript file, creating an Abstract Syntax Tree (AST), removing optional typings that JavaScript doesn't understand, and then converting this tree back into a JavaScript text document. The deno_ast crate is used in the Deno ecosystem to handle this process. Here are a few steps and considerations based on the documentation that might help you resolve the issue: 1. Ensure Correct Versions: Make sure that the versions of deno_core and deno_ast you are using are compatible with each other and with the code examples you are following. 2. Module Loader Implementation: The module loader you implement must correctly handle TypeScript files. This involves determining the file's media type and whether transpiling is necessary, reading the file into a string, transpiling if necessary, and then returning the transpiled code as a module. 3. Use of deno_ast: Since you're using deno_ast with features like "transpiling" and "typescript", ensure that you're correctly using the crate to transpile TypeScript code to JavaScript. This might involve using the transpile function provided by deno_ast. 4. Error Handling: The error message you're seeing is the result of unwrapping a Result that contains an Err value. Make sure to handle errors gracefully instead of unwrapping them directly. This will help you debug issues more effectively. 5. Runtime Setup: When setting up the JsRuntime,

Deno AI Helper•5mo ago

ensure that all necessary extensions and configurations are properly initialized. This includes setting up the module loader and source map getter correctly. 6. Debugging: If you continue to encounter issues, consider adding more detailed error handling and logging to your module loader's load function to pinpoint where the failure occurs during the transpiling process. 7. Documentation and Examples: Review the latest examples and documentation for deno_core and deno_ast to ensure that your implementation aligns with the current recommended practices. Since you mentioned that the blog series might be outdated, it's important to refer to the most recent and official sources. Remember that embedding a JavaScript runtime and handling TypeScript transpilation can be complex, and it may require a deep understanding of the tools and libraries involved. If you continue to face difficulties, consider reaching out to the Deno community for more specific guidance on your implementation. 🦕

Helpful Links: Migrating to and from JavaScript | Deno Docs Roll your own JavaScript runtime, pt. 2 npm: specifiers | Deno Docs Deno 1.12 Release Notes Built-in TypeScript support - Deno by Example