All Articles
SubstrateLuaFFIPolkadotSystems Programming

The Unsafe Bridge: Architecting a High-Performance Substrate SDK in Lua

How we built SubLua—a high-performance FFI bridge connecting Lua's lightweight scripting world to Substrate's type-safe blockchain framework, solving memory ownership, async/await gaps, and runtime upgrade challenges.

By Abhiraj Mengade•

Lua is the "dark matter" of the software world. You don't always see it, but it holds the universe together. It powers the logic of World of Warcraft, manages traffic in Cloudflare's Nginx edge (OpenResty), and configures embedded IoT devices on factory floors. It is lightweight, ruthlessly fast, and dynamically typed.

Substrate (the framework behind Polkadot) is the opposite. It is built on Rust. It is statically typed, obsessed with correctness, and relies on heavy binary serialization (SCALE).

SubLua is our attempt to force these two worlds to shake hands without crashing the process. We didn't just want a wrapper; we wanted a high-performance pipe that allows a 90s scripting language to sign transactions on a next-gen blockchain, all within a few megabytes of RAM.

This is the engineering story of how we solved the memory wars, bridged the async/await chasm, and built a future-proof SDK by taming the FFI boundary.

Project Update: SubLua has successfully completed Milestone 1 of 2 for the Polkadot Open Source Bounties program, with core FFI implementation, dynamic metadata support, and comprehensive documentation now delivered.

I. The Architectural Choice: Why FFI is Non-Negotiable

The first question we faced was: Do we rewrite the Substrate SCALE codec and sr25519 cryptography in pure Lua?

It was tempting. But this approach is a trap for two reasons:

  1. Maintenance Death Sentence: Substrate evolves. Parity updates the SCALE codec and cryptographic primitives. Keeping a pure Lua re-implementation in parity with the Rust upstream would be an endless, bug-prone game of catch-up.

  2. Performance & Security: Bit-banging binary serialization and cryptographic signing in a scripting language burns CPU cycles that game loops (our primary target) can't spare. It also opens a massive surface area for security vulnerabilities.

We needed a "Single Source of Truth." The only sane path was to use the native Rust libraries (subxt and sp-core) and drive them from Lua. This forced us onto the treacherous ground of the Foreign Function Interface (FFI).

II. The Memory War: Taming the GC/RAII Chasm

The hardest engineering problem wasn't cryptography; it was memory ownership.

  • Rust uses RAII (Resource Acquisition Is Initialization). It drops memory when a variable goes out of scope. Immediately.
  • Lua uses a Garbage Collector (GC). It cleans up memory "eventually," when it feels like it.

If Rust passes a pointer to Lua and then drops the memory, Lua is left holding a dangling pointer. The result: a guaranteed Segmentation Fault.

To solve this, we implemented a manual "destructor pattern" by deliberately leaking memory in Rust and handing ownership to Lua's garbage collector.

The Rust Side (The Leak)

When creating a persistent object like a signer, we use Box::into_raw. This tells the Rust compiler: "Forget this memory exists. It's not your problem anymore."

#[no_mangle]
pub extern "C" fn create_signer(seed: *const c_char) -> *mut SignerContext {
    // ... create the signer pair ...

    // We deliberately "leak" the memory so Rust doesn't drop it 
    // when this function ends. Lua now owns this pointer.
    Box::into_raw(Box::new(SignerContext { pair }))
}

The Lua Side (The Cleanup)

On the Lua side, we use a powerful LuaJIT feature: ffi.gc(). This attaches a finalizer to the pointer.

-- Lua wrapper ensuring we don't leak RAM
function Signer:new(seed)
    local ptr = lib.create_signer(seed)
    
    -- This is the magic. We tell Lua's garbage collector:
    -- "When you eventually decide to free this object, 
    -- please call this Rust function to do it properly."
    ffi.gc(ptr, lib.free_signer)
    
    return ptr
end

This ensures that memory lives exactly as long as the Lua script needs it—no shorter, no longer. It's a delicate dance, but it keeps the memory footprint incredibly low.

III. The Killer Feature: Solving the Runtime Upgrade Nightmare with Dynamic Metadata

Most lightweight SDKs fall into the "Hardcoded Index Trap." They bake in assumptions like Balances.transfer is at call index [4, 0].

In the Substrate world, this is fatal. Chains undergo forkless runtime upgrades. An upgrade can remap the entire runtime, shifting Balances.transfer to [5, 3]. A hardcoded SDK would suddenly fail or, worse, call the wrong function entirely.

SubLua is architected to be metadata-aware. When it connects to a node, it doesn't just open a socket; it performs a handshake:

  1. Fetch: It downloads the chain's opaque, SCALE-encoded metadata blob.
  2. Decode: The Rust FFI layer uses subxt to parse this binary blob into a queryable structure.
  3. Resolve: Before constructing a transaction, it dynamically looks up the correct pallet and call indices.
-- NO hardcoding!
-- We ask the chain: "Where is the 'transfer_keep_alive' call inside the 'Balances' pallet?"
local indices = metadata.get_dynamic_call_index(
    "wss://westend-rpc.polkadot.io",
    "Balances",
    "transfer_keep_alive"
)
-- Returns the *current* correct indices, e.g., {4, 3}

This makes SubLua future-proof. A script written today will continue to work seamlessly even after the Polkadot network has upgraded its runtime ten times.

IV. The Asynchronous Abyss: Bridging Lua's Sync World with Rust's Async/Await

There was another, more subtle, architectural gap. Modern Rust is built on async/await. All network I/O in subxt is non-blocking. Lua, however, is almost universally synchronous.

We couldn't expose an async API to Lua. We had to create a synchronous "facade" over an asynchronous core.

The solution was to run a full tokio multi-threaded runtime inside our FFI library. Each FFI call that requires network I/O spins up a temporary async task and blocks until it completes.

// Simplified FFI function for querying balance
#[no_mangle]
pub extern "C" fn query_balance(url: *const c_char, address: *const c_char) -> FFIResult {
    // 1. Initialize the Tokio runtime for this thread.
    let rt = tokio::runtime::Runtime::new().unwrap();

    // 2. Use `block_on` to run the async code and wait for the result.
    // This bridges the gap between Lua's sync call and Rust's async world.
    let result = rt.block_on(async {
        let client = subxt::client::OnlineClient::<PolkadotConfig>::from_url(url_str).await?;
        // ... perform the actual async storage query ...
        Ok(balance_value)
    });
    
    // 3. Package the result and return it to Lua.
    // ...
}

This is a resource-intensive solution, but it provides the clean, blocking API that Lua developers expect, while still leveraging the high-performance, non-blocking I/O of modern Rust.

V. A Toolkit for Hackers: What This Unlocks

We didn't build SubLua just for wallet apps. We built it because Lua runs in places where Node.js and Python fear to tread.

  • The Blockchain-Aware Firewall (OpenResty): Write an Nginx worker that cryptographically authenticates incoming HTTP requests against a Substrate account at the edge, before it ever hits your backend.

  • The Self-Sovereign Machine (OpenWrt/IoT): A $20 home router running OpenWrt can now natively pay for its own bandwidth, manage IoT data attestations, or act as a DePIN node without any external hardware.

  • Truly Native Game Integration (LÖVE/Minetest): Forget web-based "metaverses." Write a server-side mod for an open-source engine like Minetest where mining a diamond block triggers a direct FFI call to mint a real asset on-chain, all within the game loop.

  • The Vim-Based Trader (Neovim): Configure your code editor to show the latest block hash in your status line or execute a governance vote directly from a terminal command, because Neovim's plugin system is pure Lua.

Conclusion: The Pipe is Open

SubLua is an opinionated piece of software. We prioritized performance and embeddability over ease of contribution. We chose unsafe blocks and manual memory management because we wanted raw speed and a minimal footprint.

We have built the pipe. It connects the rigid, secure world of Substrate to the fluid, chaotic world of embedded Lua. Now we want to see what you pump through it.

Clone the repo, build the FFI, and start breaking things.

GitHub: https://github.com/MontaQLabs/sublua

Want to learn more about our work? Check out our projects or get in touch with us.

View All Articles