Skip to main content

Boo! Now Get Off My Browser Tab!

On-Device WebGPU • Voice & Vision Agents • Halloween Build

Rebuilding NVIDIA’s Nemotron Haunted House 100% Locally on WebGPU

How an inspiring multi-GPU Python, FastRTC, and Omniverse driveway installation turned into a WebGPU haunted house—complete with 44-target ARKit pirate-plank deformation, TSL volumetric hellfire, YOLOv10s + ByteTrack costume roasting.

When the NVIDIA team published Boo! Now Get Off My Lawn! Building a Haunted House with NVIDIA Nemotron Open Models on Hugging Face, it immediately stood out as one of the most joyful physical AI builds of the year. Instead of putting another polite assistant inside a chat box, they projected a grumpy, splintered sunken pirate ship onto a real suburban garage door—giving it live speech recognition, webcam costume detection, stateful trick-or-treat comedic timing, neural lip-sync via Audio2Face-3D, and a physical candy chute that only dispensed full-size chocolate bars once trick-or-treaters showed some manners and said “please”.

Looking at their system architecture diagram, however, sparked the exact WebAI question I can never resist asking. Running a desktop haunted house setup normally means orchestrating local Python servers, FastRTC WebRTC audio relays, desktop CUDA containers, multi-GPU VRAM budgets, and an external Maya/Omniverse blendshape rig.

The Single-Tab WebGPU Challenge

What happens if you strip away every Python microservice, every WebRTC signaling hop, and every cloud API key, and rebuild the entire haunted house—streaming Silero VAD, speech recognition, multi-visitor costume tracking, pirate persona reasoning, zero-shot voice conditioning, 60 Hz 44-target ARKit facial animation, Dual-Stream Demon DSP, and volumetric furnace fire—to run 100% client-side inside a single browser tab over WebGPU?

That question turned into MonsterHouse WebGPU: an open-source, zero-backend haunted house studio where anyone with Chrome or Edge, a laptop GPU, a webcam, and a backyard projector can turn their garage door into an autonomous trick-or-treat monster in thirty seconds flat.

🎬 Full Narrated Live Demo Walkthrough (3m 45s • Audio On 🔊) Open Direct MP4 ↗
Chapters: 0:00 3D Stage & TSL Hellfire • 0:33 Trick-or-Treat & “Please” Breakthrough • 1:01 Soundboard, Inline Tags & Puppet Mic • 1:29 Profiles C/B/A & Demon DSP • 2:06 YOLOv10s + ByteTrack Costume Roast • 2:33 Prompts & Shader Tuner • 2:56 Zero-Shot Voice & ?projector=1 • 3:24 Cache, Metrics & Architecture
MonsterHouse WebGPU 3D Stage, Volumetric Hellfire, and Candy Barrage GIF

Figure 1: Three.js WebGPU Haunted House stage running at 60 FPS—24 splintered pirate planks deformed across 44 ARKit sound targets, 3-layer TSL volumetric hellfire, Close-Up Maw & Projector Mask views, and physicalized candy launches onto a rain-slick driveway.

1. Collapsing a Multi-GPU Python Stack into 5 Browser Web Workers

In a desktop haunted house setup, keeping 60 FPS graphics smooth while running speech, vision, and LLM inference is straightforward: you pin services to separate OS processes or separate GPUs. Inside a browser tab, naive main-thread inference freezes requestAnimationFrame the instant an ONNX session dispatches a heavy prefill shader.

To keep our Three.js WebGPU stage locked at 60 FPS with zero UI jank, we partitioned the entire application into five isolated execution lanes backed by cross-origin isolation headers (COOP: same-origin, COEP: require-corp) so ONNX Runtime Web can pair WebGPU compute queues with multi-threaded SharedArrayBuffer WASM workers:

MonsterHouse In-Browser 5-Lane WebGPU Architecture (Zero Backend Servers) COOP/COEP SharedArrayBuffer Isolation • 4 Dedicated AI Workers + 60 FPS Main Stage 1. speech.worker.ts Silero VAD (2MB, 300ms EOU) Parakeet 0.6B CTC / Gemma Audio + 350ms Speaker Echo Gate 2. vision.worker.ts YOLOv10s WebGPU (32MB) Two-Pass TS ByteTrack IoU 5-Visitor Synthetic or Webcam 3. brain.worker.ts EncounterController (10 States) Profile C: LFM3-VL 2B (1.24GB) Profile B: Gemma 4 E2B (3.22GB) Profile A: Nemotron E4B (4.68GB) Shared Costume Crop VLM (0MB) StreamingClauseAccumulator Serialized enqueueGpuTask() 4. tts-a2f.worker.ts Kokoro-82M / Zero-Shot Clone NVIDIA Audio2Face-3D v2.3 60Hz 44-Target ARKit Curves 5. Main 60Hz Stage + DSP Three.js TSL Hellfire + Planks GranularPitch AudioWorklet BroadcastChannel ?projector=1

Desktop workstations can casually load 16 GB of bf16 weights, whereas browser tabs operate inside Dawn/wgpu buffer limits and laptop VRAM. In src/ai/modelRegistry.ts, we built three switchable WebGPU hardware profiles—exposing Profile C and Profile B as instant co-primary toggles in the left sidebar while keeping NVIDIA’s larger Profile A in the Services dropdown for 8 GB+ desktop GPUs:

Profile Brain + Vision (q4f16) Speech Listener (VAD + ASR) Est. VRAM
Profile C (Default) Liquid AI LFM3-VL-2B-ONNX (1,236 MB shared Brain + Vision; 75% gated Short-Conv O(1) cache) Silero VAD (2 MB) + Parakeet 0.6B CTC (433 MB) ~1.7 GiB
Profile B (Co-Primary) Google Gemma 4 E2B-it ONNX (3,224 MB unified Text + Vision + Audio) Silero VAD + Gemma 4 Native 16 kHz Audio Encoder (0 MB extra ASR) ~3.4 GiB
Profile A (Dropdown) NVIDIA Nemotron 3.5 Bolt E4B (4,678 MB q4f16) + optional LFM3-VL subagent Silero VAD (2 MB) + Parakeet 0.6B CTC (433 MB) ~5.8 GiB

Profile C (LFM3-VL 2B) shines on integrated laptop GPUs because Liquid AI’s hybrid architecture replaces 75% of quadratic attention layers with gated Short-Conv blocks carrying a tiny fixed-size O(1) state—giving us multimodal vision + fast token streaming in 1.24 GB. Profile B (Gemma 4 E2B Omni) eliminates separate ASR weights completely by routing 16 kHz speech frames directly into Gemma 4’s native audio_encoder_q4f16 alongside porch camera crops. And to prevent cold-boot HTTP/2 contention, bootAllWorkers brings Silero VAD (2 MB) online first and queues Parakeet 0.6B until the Brain finishes streaming its quantized .onnx_data shards.

Switching WebGPU Model Profiles C, B, and A and Tuning Demon DSP

Figure 2: Flipping live between Profile C (LFM3-VL 2B) and Profile B (Gemma 4 E2B Omni), inspecting the 5 worker slots and Profile A (Nemotron Bolt E4B) in Services, and sculpting the Dual-Stream Demon DSP rack.

2. Deterministic Trick-or-Treat State Machine & Two Latency Shortcuts

One of the sharpest design lessons in NVIDIA’s original post is that small real-time models should improvise pirate flavor, not manage game state. Asking a 2B quantized model to track politeness, refusal counts, and candy inventory inside raw chat history falls apart as soon as three kids yell over each other on a porch.

In src/orchestrator/EncounterController.ts, we keep all conversational rules inside a deterministic 10-state finite-state machine (IdleSelfTalk, WarmGreeting, GrumpyGreeting, FirstRefusal, GrumpyBanter, AnswerPlusCandy, GivingIn, WarmFollowUp, GrudgingFollowUp, Goodbye). Demanding candy greedily triggers FirstRefusal (0 treats dispensed). The exact moment a visitor says “please”, EncounterController triggers an immediate GivingIn breakthrough, erupts golden furnace sparks, and fires a 3D candy barrage across the driveway:

Trick-or-Treat State Machine FirstRefusal to GivingIn Please Breakthrough

Figure 3: Live trick-or-treat arc—demanding candy triggers deterministic FirstRefusal, saying "please" unlocks an immediate GivingIn candy barrage, and thanking the house transitions to Goodbye.

To keep vocal turnaround snappy in the browser, we pair the state machine with two latency shortcuts:

  1. Prepared Dialogue Bank + Sub-50ms IndexedDB Curve Cache (src/cache/responseCache.ts): Canonical beats can draw directly from our authored pirate bank (piratePersonaBank.ts), caching both the 24 kHz Float32Array audio and pre-solved 60 Hz 44-target ARKit curves in IndexedDB for instant <50 ms replay.
  2. Streaming Clause Accumulator & Inline Emotion Tags (src/orchestrator/emotionTagParser.ts): When the LLM improvises fresh banter, StreamingClauseAccumulator strips any <think>...</think> blocks, parses inline tags like [grumpy], [short pause], [menacing], and [delighted], and flushes clause #1 to TTS + Audio2Face at the first punctuation boundary while clause #2 is still decoding—shifting vocal prosody and flame colors mid-sentence without ever speaking the bracketed tags aloud.
12-Tile Soundboard, Inline Emotion Tags, and Puppet Mic Mode

Figure 4: Cutting in with the 12-tile Soundboard, scripting multi-mood inline tags ([grumpy] → [short pause] → [menacing] → [delighted]), and toggling between Live, Demo, and real-time Puppet Mic possession.

3. Splitting Clean Lip-Sync from Demon DSP & 44 Eyeless ARKit Targets

Early on, we hit a classic audio-driven animation trap: if you pitch-shift TTS speech down -4.5 semitones, saturate it with tube drive, drench it in 1.85-second ship-hull reverb, and feed that monsterized signal into Audio2Face-3D or a spectral formant solver, lip-sync turns to mush. Sub-bass resonance and reverb tails smear plosives and keep the wooden jaws hanging open long after the syllable finishes.

Dual-Stream Audio Fork: Crisp 60Hz Lip-Sync + Booming Galleon Acoustics Clean 24kHz PCM Kokoro-82M / Zero-Shot or Live Puppet Mic Stream A: Clean Analytic Bus Audio2Face-3D Mark ([1,1,8320]) + Spectral F1/F2 Formant Solver Stream B: Demon WebAudio DSP GranularPitchWorklet (-4.5 st) +7.5dB Sub • Drive • Hull Convolver 44 Eyeless ARKit Poses 24 Split Oak Planks @ 60Hz Driveway Speakers Possessed Pirate Galleon

In src/audio/DualStreamAudioRouter.ts, clean speech drives the 60 Hz 44-target mouth solver while speaker audio flows through a custom zero-dependency GranularPitchProcessor AudioWorklet (dual 180°-out-of-phase Hann grains shifting pitch without altering tempo), followed by asymmetric ship-timber saturation and a procedural hull impulse response with oak slapback reflections at 38 ms and 79 ms.

Why 44 ARKit targets instead of 52? Because a garage door has no eyeballs! Masking the 8 ocular blink/gaze targets to 0 concentrates 100% of deformation energy into our 24 procedural weathered-oak planks, charred inner fangs, and magma tongue (src/stage/SplinteredMouthRig.ts), backed by a 3-layer Three.js Shading Language (TSL) volumetric hellfire node shader (src/stage/HellfireComputeTSL.ts) whose WGSL color and turbulence uniforms morph live with every emotion tag:

Editable Pirate Prompts and Live TSL Volumetric Hellfire Shader Tuner

Figure 5: Editing persona prompts and state-indexed pirate exemplars in Prompts, then previewing [menacing], [delighted], and [curious] TSL volumetric hellfire uniforms live on the 3D stage.

4. Driveway Echolocation: YOLOv10s, ByteTrack, and Shared-VRAM Costume Roasting

In src/ai/workers/vision.worker.ts and src/vision/ByteTracker.ts, fast multi-person detection runs on onnx-community/yolov10s (32 MB) paired with a two-pass TypeScript ByteTrack Kalman/IoU tracker that keeps persistent Visitor #ID boxes across driveway occlusions. Every 2 seconds (configurable 1s–6s), cropped visitor boxes hop over to brain.worker.ts to reuse the already-loaded LFM3-VL or Gemma 4 vision encoder for 0 MB of extra VRAM.

And so anyone can test multi-person tracking at their desk without putting on a cape, Camera Vision boots into a built-in 5-costume synthetic porch scene (Pirate, Vampire, Witch, Pumpkin, and Ghost) alongside a one-click Live Webcam toggle and an instant Echolocation Roast button:

YOLOv10s and ByteTrack Multi-Visitor Costume Tracking and Echolocation Roast

Figure 6: Multi-visitor tracking via YOLOv10s + ByteTrack in Camera Vision, VLM sampling cadence controls, and live Echolocation Costume Roasting.

5. Four Real Browser War Stories from Auditing the Stack

Fitting speech, vision, VLM generation, zero-shot voice conditioning, and 60 FPS WebGPU rendering into one tab uncovered four subtle bugs worth sharing:

// Bug #1 Fix in brain.worker.ts: slice off prompt input_ids before decoding VLM JSON!
// Otherwise batch_decode() includes the prompt's '{"costume": "<1-2 word label>"}' example.
const promptLen = inputs.input_ids?.dims?.[1] ?? 0;
const genIds = promptLen > 0 ? out.slice(null, [promptLen, null]) : out;
const decoded = processor.batch_decode(genIds, { skip_special_tokens: true })[0] ?? '';
  • Prompt-Echo Costume Labels: Decoding the full model.generate() output tensor matched the prompt’s literal placeholder "<1-2 word label>" before reaching the VLM’s answer until we sliced past promptLen.
  • WebGPU Session Collisions: Background 2-second webcam costume crops occasionally collided mid-decode with live voice turns on the shared ONNX WebGPU session. Wrapping dispatches in a serialized enqueueGpuTask promise queue (and skipping background crops while gpuBusyCount > 0) eliminated every race condition.
  • Cold-Boot Cache Poisoning: Clicking a tile during the first 3 seconds before Kokoro-82M finished loading cached the cold-boot fallback synth in IndexedDB permanently. Tagging each CachedPerformanceEntry with ttsEngine now auto-upgrades fallback entries as soon as WebGPU TTS turns green.
  • Acoustic Self-Triggering & Zero-Gain Mic Taps: Booming pirate lines over speakers re-triggered Silero VAD until we added SET_HOUSE_SPEAKING echo gating with a 350 ms room-reverb tail guard, plus a 0.0 GainNode terminator on ToolsTab’s voice-cloning recorder.

6. Garage-Door Projection (?projector=1), Zero-Shot Voice & Live Telemetry

For a real driveway installation, clicking Projector pops out a frameless pure-black /?projector=1 window for your HDMI garage projector, synced at 60 Hz over BroadcastChannel('monsterhouse_projector_v1'). Inside the Tools tab, you can record 3–10 second per-mood reference clips in Sample Your Voice for zero-shot timbre conditioning, run the 44-target ARKit phoneme sweep, and warp 4-corner keystone geometry live across both windows:

Zero-Shot Voice Studio, 44-Target ARKit Sweep, and Dual-Window Projector Keystone Sync

Figure 7: Recording a zero-shot pirate voice sample, sweeping all 44 ARKit plank targets, and warping 4-corner keystone calibration synced live to /?projector=1 (inset top-right).

IndexedDB 60Hz Curve Cache, Turnaround Latency Metrics, and 5-Tier Browser Architecture

Figure 8: Auditioning sub-50ms IndexedDB cached performances in Cache, inspecting EOU-to-First-Audio turnaround latency in Metrics, and verifying all 5 browser pipeline tiers in Architecture.

7. How to Play With Every Functionality in 5 Minutes

Clone the repository, run npm install && npm run dev, and open http://localhost:5173 in Chrome or Edge:

  1. 3D Viewport & Candy Physics: Toggle Full House, Close-Up Maw, and Projector Mask in the top-right stage overlay, expand Theater mode, and click Toss Candy.
  2. Trick-or-Treat Politeness Loop (CONVERSATION): Click Connect. Demand “Give us candy!” to trigger FirstRefusal, then say “May we please have some candy?” to trigger the GivingIn candy barrage, and finish with “Thank you, Happy Halloween!”
  3. Soundboard, Inline Emotion Tags & Puppet Mic: Click any of the 12 Soundboard tiles, script custom inline tags ([grumpy] ... [short pause] [menacing] ... [delighted]) in Put Words In My Mouth, or switch the Left Sidebar to Puppet Mic to drive the wooden jaws and Demon DSP live with your own voice.
  4. Model Profiles & Demon DSP (SERVICES + Right Sidebar): Flip between Profile C (LFM3-VL 2B), Profile B (Gemma 4 E2B Omni), and Profile A (Nemotron Bolt E4B), then drag the Right Sidebar’s pitch, hull-reverb, and 44 ARKit sliders.
  5. Live Webcam Costume Roasting (CAMERA VISION): Test the 5-visitor synthetic porch immediately, or click Live Webcam, step into frame until your green ByteTrack box appears, and hit Echolocation Roast.
  6. Shader Tuning & Garage Projection (PROMPTS & TOOLS): Preview mood fire shaders on the 3D stage in PROMPTS, record zero-shot voice clips in TOOLS, pop out Projector (/?projector=1), and align 4-corner keystone sliders onto your garage door.

Got a Projector and a Browser Tab? Wake Up the House.

No Python environments, no CUDA Docker containers, and no cloud API keys required. Clone the repo, open Chrome or Edge, pop out the frameless ?projector=1 mask onto your garage door, and see who remembers to say “please”.

💻 GitHub Repository: https://github.com/rabimba/monsterhouse-webgpu
🎬 Narrated Live Demo Video (MP4): monsterhouse-live-demo.mp4
🎃 Original Inspiration (NVIDIA Nemotron Post): Boo! Now Get Off My Lawn! Building a Haunted House with NVIDIA Nemotron Open Models

Comments

Popular posts from this blog

Racecraft (Project Koru) · Prologue — The Origin Story

Racecraft · Prologue , The Origin Story It Started With a Wine List and a Question About Racing How a happy-hour conversation in the Bay Area turned into a trustable AI race coach , and then into a second version that runs entirely on a phone, on the NPU. This is the prologue to a five-part series. Two years ago(1st November, 2024) I was in the Bay Area for a GDE Summit. If you've never been: it's a couple of days of talks among Google Developer Experts, the kind of people who get unreasonably excited about a new on-device runtime, and then , mercifully , a happy hour where everyone stops performing and just eats. We ended up at a restaurant(Puesto Santa Clara), a long table of GDEs, and I was doing the most important engineering of the evening: trying to decide which wine to order. Across the table was Ajeet Mirwani . I don't even remember how the wine talk turned into racing talk , these things drift , but the moment the word "racing" ...

The Throughput Trap: Benchmarking vLLM on OpenXLA and the Reality of Production LLM Serving

vLLM Systems · DevLab 2026, Deep Dive I was recently invited by the Google TPU team to speak at the OpenXLA Summer DevLab 2026 . This post breaks down our deep-dive evaluation of the matured vLLM + OpenXLA stack, the fundamental engineering mismatches between CUDA and XLA serving paths, and why traditional capacity metrics are lying to you. If you are operating large language models at enterprise scale right now, your platform architecture team is likely staring at a massive infrastructure crossroads: Should we migrate our core serving workloads from GPUs to TPUs? Historically, NVIDIA's CUDA ecosystem was the only serious option for user-facing, low-latency LLM generation. But here in 2026, the economics and infrastructure options have transformed. Google TPUs are highly available, cheaper per chip, and the open-source serving stack built around vLLM and OpenXLA has officially achieved absolute production readiness. Yet, when our infrast...

A Split‑Brain Neuro‑Symbolic Training Method for High‑Velocity Autonomous Coaching from Telemetry

 Author: Rabimba Karanjai Scope: Problem statement + data methodology + model training (no deployment discussion) Abstract Real‑time coaching in motorsport is a safety‑critical learning problem : a system must map noisy, high‑frequency telemetry to short, actionable guidance that remains physically consistent and avoids hazardous recommendations . This paper proposes a “Split‑Brain” training formulation that separates (i) a semantic coaching target (what action/critique should be expressed) from (ii) a reflexive interface (how actions are represented as compact, verifiable tokens). The approach trains a Small Language Model (SLM) in the Gemma family [1] using QLoRA fine‑tuning [2] , and introduces a telemetry tokenizer plus teacher‑student synthesis pipeline to generate instruction‑action pairs at scale. Core contribution: a reproducible method to convert “ golden lap ” differential tel...