From 4e2224e9dc02ea88f6cade45258aaebba506f49b Mon Sep 17 00:00:00 2001 From: Pine Date: Tue, 18 Aug 2026 01:37:44 +0800 Subject: [PATCH] =?UTF-8?q?feat(voice):=20=E5=AE=9E=E6=97=B6=E8=AF=AD?= =?UTF-8?q?=E9=9F=B3=E5=AF=B9=E8=AF=9D=E9=A1=B5=EF=BC=88React=20=E5=AE=8C?= =?UTF-8?q?=E5=85=A8=E5=A4=8D=E5=88=BB=E5=8E=9F=E7=89=88=20UI=EF=BC=89?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - /voice 页面:中间圆球 + 左右按钮 + 状态图标 + 右下角气泡(进场/淡出/阶梯字号/上限8)+ 噪声门弧线电平,结构类名与原版一致 - s2s WebSocket 客户端 + worklet 音频(mic-capture/audio-playback)+ orb 可视化 - 摄像头实时预览(默认开启)+ 视觉识别状态条(后端 YOLO 推理) - 底部园区概览条(复用 ParkOverviewStrip),orb-wrap 视窗居中 - 页眉复用 ScreenLayout;左右键导航加入 /voice - 文档:docs/voice-integration-plan.md --- docs/voice-integration-plan.md | 168 ++ public/voice-demo/config.js | 3 + public/voice-demo/index.html | 353 +++ public/voice-demo/main.js | 1503 ++++++++++ public/voice-demo/style.css | 2773 ++++++++++++++++++ public/voice-demo/ui/account.js | 189 ++ public/voice-demo/ui/chat.js | 422 +++ public/voice-demo/ui/dom.js | 37 + public/voice-demo/worklets/audio-playback.js | 211 ++ public/voice-demo/worklets/mic-capture.js | 159 + public/voice-demo/ws/codec.js | 57 + public/voice-demo/ws/orb-visualizer.js | 98 + public/voice-demo/ws/s2s-ws-client.js | 1176 ++++++++ public/voice-worklets/audio-playback.js | 211 ++ public/voice-worklets/mic-capture.js | 159 + src/App.jsx | 3 + src/config.js | 2 +- src/pages/VoiceAssistant.jsx | 539 ++++ src/pages/voice-embed.css | 12 + src/pages/voice-original.css | 2773 ++++++++++++++++++ src/pages/voice-overrides.css | 135 + src/utils/pageNav.js | 1 + src/voice/codec.js | 57 + src/voice/orb-visualizer.js | 98 + src/voice/s2s-ws-client.js | 1185 ++++++++ src/voice/useVisionDetection.js | 243 ++ 26 files changed, 12566 insertions(+), 1 deletion(-) create mode 100644 docs/voice-integration-plan.md create mode 100644 public/voice-demo/config.js create mode 100644 public/voice-demo/index.html create mode 100644 public/voice-demo/main.js create mode 100644 public/voice-demo/style.css create mode 100644 public/voice-demo/ui/account.js create mode 100644 public/voice-demo/ui/chat.js create mode 100644 public/voice-demo/ui/dom.js create mode 100644 public/voice-demo/worklets/audio-playback.js create mode 100644 public/voice-demo/worklets/mic-capture.js create mode 100644 public/voice-demo/ws/codec.js create mode 100644 public/voice-demo/ws/orb-visualizer.js create mode 100644 public/voice-demo/ws/s2s-ws-client.js create mode 100644 public/voice-worklets/audio-playback.js create mode 100644 public/voice-worklets/mic-capture.js create mode 100644 src/pages/VoiceAssistant.jsx create mode 100644 src/pages/voice-embed.css create mode 100644 src/pages/voice-original.css create mode 100644 src/pages/voice-overrides.css create mode 100644 src/voice/codec.js create mode 100644 src/voice/orb-visualizer.js create mode 100644 src/voice/s2s-ws-client.js create mode 100644 src/voice/useVisionDetection.js diff --git a/docs/voice-integration-plan.md b/docs/voice-integration-plan.md new file mode 100644 index 0000000..7527e4c --- /dev/null +++ b/docs/voice-integration-plan.md @@ -0,0 +1,168 @@ +# DPM 实时语音对话整合方案(复用 live-avatar 云化栈) + +> 目标:砍掉数字人/口型/wav2lip 等全部视觉部分,只保留「语音识别 + 对话 + 实时交互」链路; +> 前端在 DPM React 应用中新增对话页(复用 live-avatar demo 前端的样式与逻辑); +> 后端并入 DPM `backend/pyproject.toml`。 +> 基线:live-avatar 云化栈已在 `/Volumes/Pine/mycode/opc/live` 跑通(s2s v0.2.11 + cloud-migration 补丁 + demo 前端)。 + +--- + +## 0. 现状对比 + +| | live-avatar(已跑通) | DPM(现状) | +|---|---|---| +| 前端 | 原生 JS demo(index.html / main.js / s2s-ws-client.js / 两个 worklet) | React 19 + Vite + Tailwind;`/ai` 页已有 `AiChatPanel`(文字 + **录音上传式**语音) | +| 后端 | s2s 独立进程 :8765(VAD 本地 + 云 ASR/LLM/TTS) | FastAPI :8000;已有 `app/asr.py`(dashscope paraformer)、`app/llm.py`(qwen + 工具)、`SSL_CERT_FILE=certifi` 同款处理 | +| 语音体验 | **实时全双工**:本地 VAD → 流式云 ASR → 流式 LLM → 流式 TTS → 音频回放 | 录音 → 上传 → 转写 → 文字回复(无流式、无 TTS 回音) | + +**DPM 缺的正是 live-avatar 栈的实时交互能力**;两边的云凭据/CA 处理已一致(dashscope + certifi),合并成本低。 + +--- + +## 1. 两个关键决策点 + +### 1.1 后端:内嵌进 DPM FastAPI(推荐) vs 独立 s2s 进程 + +- **推荐:内嵌**。s2s v0.2.11 的 `api/openai_realtime/websocket_router.create_app(pool, stop_event)` 本身就返回一个 FastAPI 应用, + 可在 DPM `app.main` 里构建管线池后 `app.mount("/v1", s2s_app)`,与现有 `/api/*` 共存、单进程单端口。 + Tauri 桌面打包只需一个后端进程,无子进程管理。 +- 备选:s2s 独立跑 :8765,DPM 负责拉起/守护。改动最小,但两进程两端口,打包与部署更重。 + +### 1.2 前端:iframe 内嵌(推荐 v1) vs React 组件移植(v2) + +- **推荐 v1:iframe 内嵌**。把 live-avatar `demo/` 目录拷入 DPM `public/voice-demo/`(去掉数字人 iframe/视觉元素), + React 新增 `VoiceAssistant.jsx`(路由 `/voice`)里 `