Skip to content

Latest commit

 

History

History
54 lines (39 loc) · 1.59 KB

README.md

File metadata and controls

54 lines (39 loc) · 1.59 KB
Raw

Human: 3D Detection to VRM Avatar

WIP!!!

  • Input
    Video file
    Live webcam
  • Analyze and track all keypoints in a web worker thread
    Using Human library
    Face, Body, Hands
  • Display
    Video Overlay with keypoints
    VRM Avatar with pan/zoom/rotate functionality

Run


Online

Live Demo

Offline

  • Clone or fork the repository
  • Install dependencies: npm install
  • Start HTTPS server using precompiled code: npm start
  • Navigate to https://localhost:8001

Development Mode

  • Clone or fork the repository
  • Compile code on-demand & start HTTPS server: npm run dev
  • Navigate to https://localhost:8001

Notes

  • All heavy-lifting (machine learning processing) is performed using Human library
  • 2D visualization is performed using built-in draw methods in Human library
  • 3D visualization is performed using BabylonJS

Processing

Process flow is separated into two loops:

  • Process loop
    Runs in web-worker so its not blocking the main thread
    Supports WASM, WebGL and WebGPU backends
    Runs as fast as possible: frame is submited to worker and as soon as results are available a new frame is submitted
  • Render loop
    Runs at fixed 60 FPS
    Draws both 2D canvas overlay and 3D GL mesh
    Uses intepolated (for smoother output) results based on last known processed results from process loop