n8n/packages/@n8n/expression-runtime/docs/deep-lazy-proxy.md

# Deep Lazy Proxy

## Overview

The Deep Lazy Proxy is a memory-efficient mechanism for providing workflow data to expression evaluation contexts. Instead of copying entire data structures upfront, it loads data on-demand as properties are accessed.

## Key Features

- **On-Demand Loading**: Only fetches data when accessed
- **Metadata-Driven**: Returns object structure (keys, length) without values
- **Caching**: Values are cached after first access to avoid redundant lookups
- **Type Support**: Handles objects, arrays, functions, and primitives correctly
- **Memory Efficient**: Large arrays and objects don't cause memory overhead

## Architecture

The deep lazy proxy is implemented in `src/runtime/lazy-proxy.ts`, which is bundled
together with the other runtime modules into `dist/bundle/runtime.iife.js` and injected
into the V8 isolate at startup.

Key functions exposed on `globalThis` inside the isolate:

- `createDeepLazyProxy(basePath)` — creates recursive object/array proxies
- `resetDataProxies()` — called before each evaluation to reinitialise `$json`,
  `$input`, `$node`, etc. as fresh lazy proxies backed by the three host callbacks
- `__sanitize(key)` — runtime property-access guard that blocks `__proto__`,
  `constructor`, `prototype`, etc.

Host-side callbacks registered by `IsolatedVmBridge` as `ivm.Reference` objects
(synchronous cross-isolate calls):

- `__getValueAtPath(path[])` — returns a primitive, array metadata, or object metadata
- `__getArrayElement(path[], index)` — returns a single array element (or its metadata)
- `__callFunctionAtPath(path[], ...args)` — invokes a host-side function and returns the result

## Usage

The proxy system runs **inside the V8 isolate** and is not directly importable from
host code. The host sets up the data context by calling `bridge.execute(code, data)`,
which internally:

1. Registers three `ivm.Reference` callbacks with the current `data` object
2. Calls `resetDataProxies()` in the isolate to create fresh lazy proxies for
   `$json`, `$binary`, `$input`, `$node`, `$parameter`, `$workflow`, `$prevNode`
3. Runs the tournament-transformed expression code with `this === __data`

From the expression's perspective it just sees normal objects:

```typescript
// Inside an expression (runs in isolate):
$json.user.email        // triggers getValueAtPath(['$json','user','email'])
$json.items[150].id     // triggers getArrayElement(['$json','items'], 150)
$items()                // triggers callFunctionAtPath(['$items'])
```

### Array metadata

Arrays are **never transferred in full** — only their length is returned. Elements
are loaded individually on demand. Length can be determined from the host object
in O(1), but serialization cost is proportional to the total byte size of all
elements, which cannot be bounded from length alone.

```typescript
// __getValueAtPath returns:
{ __isArray: true, __length: 1000 }            // always metadata only
{ __isObject: true, __keys: ['name','email'] } // object — lazy
42                                              // primitive
```

## How It Works

### Metadata Pattern

Instead of transferring entire objects/arrays, the proxy uses metadata:

**Arrays** (all sizes):
```typescript
{
  __isArray: true,
  __length: 1000  // Only length; elements loaded on demand via __getArrayElement
}
```

**Objects**:
```typescript
{
  __isObject: true,
  __keys: ['name', 'email', 'age']  // Only keys, not values
}
```

### Caching

Once a property is accessed, it's cached in the proxy's target object:

```typescript
proxy.$json.user.name  // First access: fetches via callback
proxy.$json.user.name  // Second access: returns cached value
```

### Recursive Proxies

When accessing nested objects or arrays, new proxies are created:

```typescript
proxy.$json.user  // Creates proxy for user object
proxy.$json.items[50]  // Creates proxy for object at index 50
```

## Security

### Function Handling

- **Custom Functions**: Allowed and passed directly
- **Native Functions**: Blocked for security (e.g., `Object.keys`)

```typescript
const customFn = (x: number) => x * 2;  // Allowed
const nativeFn = Object.keys;  // Blocked (returns undefined)
```

Detection is done by checking if `fn.toString()` contains `'[native code]'`.

### Symbol Properties

Symbol properties return `undefined` to prevent security issues.

## Performance

### Memory Efficiency

- **Arrays**: Always lazy-loaded — only length transferred, elements fetched on demand
- **Objects**: Always lazy-loaded — only keys transferred, values fetched on demand

### Access Patterns

Best performance when:
- Accessing few properties from large objects
- Accessing specific array elements (not iterating entire array)
- Accessing the same properties multiple times (caching means only the first access pays)

Suboptimal performance when:
- Iterating entire arrays (`.map()`, `.filter()`) — each element triggers a separate callback
- Accessing most properties of large objects
- No property reuse (no benefit from caching)

## Known Limitations

1. **Array Methods**: Methods like `.map()`, `.filter()` iterate all elements.
   Each element triggers a separate `__getArrayElement` callback call, which is slow
   for large arrays.
   - **Workaround**: Avoid iterating large arrays in expressions; access specific indices instead

2. **Circular References**: May cause infinite loops in the proxy handler.
   - **Current**: No cycle detection; circular structures should be avoided in expression data

## Testing

### Integration Tests

```bash
cd packages/@n8n/expression-runtime
pnpm test
```

Test coverage:
- ✅ Basic property access
- ✅ Nested properties
- ✅ Array element access (lazy-loaded via `__getArrayElement`)
- ✅ Object proxies
- ✅ Function handling
- ✅ Caching behavior
- ✅ Edge cases (circular refs, symbols, "in" operator)

## API Reference (inside the isolate bundle)

These functions are available on `globalThis` within the V8 isolate after the
runtime bundle (`dist/bundle/runtime.iife.js`) is loaded.

### `resetDataProxies()`

Called by the bridge before each expression evaluation. Reads `$json`, `$binary`,
`$input`, `$node`, `$parameter`, `$workflow`, `$prevNode`, `$runIndex`, `$itemIndex`,
and `$items` from `__data` (populated via host callbacks) and exposes them on both
`globalThis` and `__data` so tournament-transformed code can access them via
`this.$json`, `this.$input`, etc.

### `createDeepLazyProxy(basePath)`

Creates a recursive Proxy for a given property path. Intercepts property access and
calls back to the host via `__getValueAtPath` to fetch structure metadata, then
creates nested proxies for objects or arrays as needed.

**Parameter:**
- `basePath: string[]` — path from the root data object to the node this proxy represents

## Examples

### Accessing nested data (expression syntax)

```
{{ $json.order.customer.name }}    // lazy-loads order.customer.name
{{ $json.order.items[1].product }} // lazy-loads array element at index 1
{{ $json.items[0] }}               // fetches only the first element
```

### Array iteration is slow for large arrays

```
{{ $json.items.reduce((sum, x) => sum + x, 0) }}
// items has 10 000 elements → length transferred, then 10 000 callback
// calls to fetch each element. Prefer accessing specific indices.
```

Note: lodash (`_`) is not available in expressions — it is bundled internally for
use by extension functions but not exposed on `globalThis`.

## Contributing

When modifying the proxy implementation:

1. **Run tests**: `pnpm test proxy`
2. **Type check**: `pnpm typecheck`
3. **Build**: `pnpm build`
4. **Add tests** for new features
5. **Update this documentation**

## Related Files

- Proxy implementation: `packages/@n8n/expression-runtime/src/runtime/lazy-proxy.ts` — `createDeepLazyProxy`
- Reset: `packages/@n8n/expression-runtime/src/runtime/reset.ts` — `resetDataProxies`
- Security globals: `packages/@n8n/expression-runtime/src/runtime/safe-globals.ts` — `SafeObject`, `SafeError`, `__sanitize`
- Runtime entry: `packages/@n8n/expression-runtime/src/runtime/index.ts` — wires all modules to `globalThis`
- Bridge: `packages/@n8n/expression-runtime/src/bridge/isolated-vm-bridge.ts` — registers `ivm.Reference` callbacks, loads bundle, calls `resetDataProxies`
- Build: `packages/@n8n/expression-runtime/esbuild.config.js` — bundles runtime to `dist/bundle/runtime.iife.js`