refactor(backend): 启动环境初始化与资源项目化（零跨项目依赖）

- main.py：uv run main.py 一键启动——读取 backend/.env、TORCH_HOME/NLTK_DATA 指向项目内缓存、certifi SSL + NO_PROXY 网络直连、默认开启 s2s，启动横幅
- vendor/：补丁后 s2s 云化栈移入项目（backend/vendor/s2s-cloud），pyproject sources 改相对路径
- .gitignore：忽略 .torch-cache / nltk_data 缓存
- AGENTS.md：进程管理铁律（AI 不启动/重启服务，由用户操作）

backend/vendor/s2s-cloud/src/speech_to_speech/STT/README.md

@@ -0,0 +1,163 @@
+# STT Summary
+
+This document summarizes the Speech-to-Text (STT) implementations in the `STT/` folder, including language support, language abbreviations, and usage in `s2s_pipeline.py`.
+
+## Available STT Modes (`--stt`)
+
+- `whisper` → `STT/whisper_stt_handler.py`
+- `whisper-mlx` → `STT/lightning_whisper_mlx_handler.py`
+- `mlx-audio-whisper` → `STT/mlx_audio_whisper_handler.py`
+- `faster-whisper` → `STT/faster_whisper_handler.py`
+- `parakeet-tdt` → `STT/parakeet_tdt_handler.py`
+- `paraformer` → `STT/paraformer_handler.py`
+
+## Language Support by Handler
+
+### 1) Whisper (`--stt whisper`)
+
+- Handler: `WhisperSTTHandler`
+- Language input flag: `--language` (from shared Whisper args)
+- Supports fixed language (e.g. `en`) or `auto`
+- Internal supported language list:
+  - `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
+- Behavior:
+  - Detects language from token output
+  - If detected language is outside the supported list, it falls back to the previous language
+
+### 2) Lightning Whisper MLX (`--stt whisper-mlx`)
+
+- Handler: `LightningWhisperSTTHandler`
+- Uses same shared `--language` argument as Whisper
+- Internal supported language list:
+  - `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
+- Behavior:
+  - If `--language auto`, model auto-detects each utterance
+  - If detected language is unsupported, falls back to last supported language
+
+### 3) MLX Audio Whisper (`--stt mlx-audio-whisper`)
+
+- Handler: `MLXAudioWhisperSTTHandler`
+- Model flag: `--mlx_audio_whisper_model_name`
+- Language still comes from shared `--language` flag (wired in pipeline)
+- Internal supported language list:
+  - `en`, `fr`, `es`, `zh`, `ja`, `ko`, `hi`, `de`, `pt`, `pl`, `it`, `nl`
+- Behavior:
+  - Uses fixed language unless `--language auto`
+  - Falls back to last known supported language when needed
+
+### 4) Faster-Whisper (`--stt faster-whisper`)
+
+- Handler: `FasterWhisperSTTHandler`
+- Language flag: `--faster_whisper_stt_gen_language`
+- Default language: `en`
+- Note:
+  - This handler passes generation kwargs directly to `faster_whisper.WhisperModel.transcribe(...)`
+  - Effective language coverage depends on selected Faster-Whisper/OpenAI Whisper model
+
+### 5) Parakeet TDT (`--stt parakeet-tdt`)
+
+- Handler: `ParakeetTDTSTTHandler`
+- Language flag: `--parakeet_tdt_language` (optional)
+- Supports auto language detection when language not specified
+- Declared supported language list (25 European languages):
+  - `en`, `de`, `fr`, `es`, `it`, `pt`, `nl`, `pl`, `ru`, `uk`, `cs`, `sk`, `hu`, `ro`, `bg`, `hr`, `sl`, `sr`, `da`, `no`, `sv`, `fi`, `et`, `lv`, `lt`
+- Backend behavior:
+  - On macOS/MPS: MLX (`mlx-community/parakeet-tdt-0.6b-v3`)
+  - On CUDA/CPU: nano-parakeet (`nvidia/parakeet-tdt-0.6b-v3`)
+
+### 6) Paraformer (`--stt paraformer`)
+
+- Handler: `ParaformerSTTHandler`
+- Model flag: `--paraformer_stt_model_name`
+- Default model: `paraformer-zh`
+- No dedicated language flag in current args class
+- Practical support:
+  - Depends on selected FunASR model checkpoint
+  - Default setup is Chinese-oriented (`zh`)
+
+## Language Abbreviations (ISO-style codes seen in STT handlers)
+
+| Code | Language |
+|---|---|
+| `en` | English |
+| `fr` | French |
+| `es` | Spanish |
+| `zh` | Chinese |
+| `ja` | Japanese |
+| `ko` | Korean |
+| `hi` | Hindi |
+| `de` | German |
+| `pt` | Portuguese |
+| `pl` | Polish |
+| `it` | Italian |
+| `nl` | Dutch |
+| `ru` | Russian |
+| `uk` | Ukrainian |
+| `cs` | Czech |
+| `sk` | Slovak |
+| `hu` | Hungarian |
+| `ro` | Romanian |
+| `bg` | Bulgarian |
+| `hr` | Croatian |
+| `sl` | Slovenian |
+| `sr` | Serbian |
+| `da` | Danish |
+| `no` | Norwegian |
+| `sv` | Swedish |
+| `fi` | Finnish |
+| `et` | Estonian |
+| `lv` | Latvian |
+| `lt` | Lithuanian |
+| `auto` | Per-utterance automatic language detection |
+
+## Usage Examples
+
+### Whisper (Transformers)
+
+```bash
+python s2s_pipeline.py --stt whisper --language en
+python s2s_pipeline.py --stt whisper --language auto
+```
+
+### Whisper MLX (LightningWhisperMLX)
+
+```bash
+python s2s_pipeline.py --stt whisper-mlx --language auto --device mps
+```
+
+### MLX Audio Whisper
+
+```bash
+python s2s_pipeline.py --stt mlx-audio-whisper \
+  --mlx_audio_whisper_model_name mlx-community/whisper-large-v3-turbo \
+  --language auto
+```
+
+### Faster-Whisper
+
+```bash
+python s2s_pipeline.py --stt faster-whisper \
+  --faster_whisper_stt_model_name large-v3 \
+  --faster_whisper_stt_gen_language en
+```
+
+### Parakeet TDT
+
+```bash
+python s2s_pipeline.py --stt parakeet-tdt --parakeet_tdt_device auto
+python s2s_pipeline.py --stt parakeet-tdt --parakeet_tdt_language de
+```
+
+With live transcription (MLX or CUDA/nano-parakeet backend):
+
+```bash
+python s2s_pipeline.py --stt parakeet-tdt \
+  --enable_live_transcription \
+  --live_transcription_update_interval 0.25
+```
+
+### Paraformer
+
+```bash
+python s2s_pipeline.py --stt paraformer --paraformer_stt_model_name paraformer-zh
+```

backend/vendor/s2s-cloud/src/speech_to_speech/STT/__init__.py

@@ -0,0 +1 @@
+

backend/vendor/s2s-cloud/src/speech_to_speech/STT/base_stt_handler.py

@@ -0,0 +1,212 @@
+from __future__ import annotations
+
+import logging
+from collections import Counter, OrderedDict
+from time import perf_counter
+from typing import Any
+
+from speech_to_speech.baseHandler import BaseHandler
+from speech_to_speech.pipeline.handler_types import STTIn, STTOut
+from speech_to_speech.pipeline.messages import PartialTranscription, Transcription, VADAudio
+from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker
+
+logger = logging.getLogger(__name__)
+
+
+class BaseSTTHandler(BaseHandler[STTIn, STTOut]):
+    """Base STT handler with speculative-turn stale input filtering."""
+
+    _MAX_COMPLETED_FINAL_REVISIONS = 2048
+
+    speculative_turns: SpeculativeTurnTracker | None = None
+    final_revision_settle_s: float = 0.0
+
+    def should_process_input(self, item: STTIn) -> bool:
+        mode = getattr(item, "mode", None)
+        turn_id = getattr(item, "turn_id", None)
+        turn_revision = getattr(item, "turn_revision", None)
+        if self._is_completed_final_revision(item):
+            queued_drops = self._drop_stale_queued_inputs()
+            self._log_stale_turn_item(item, "input-after-final", queued_drops=queued_drops)
+            return False
+        if mode == "progressive" and self._has_queued_final_for_revision(item):
+            self._log_stale_turn_item(item, "progressive-before-final")
+            return False
+
+        wait_for_stability = mode == "final"
+        gate_start = perf_counter()
+        is_latest = self._is_latest_turn_item(
+            item,
+            wait_for_pending_reopen=True,
+            wait_for_stability=wait_for_stability,
+        )
+        gate_wait_s = perf_counter() - gate_start
+        if gate_wait_s >= 0.05:
+            logger.info(
+                "%s: STT input gate waited %.3fs for turn=%s rev=%s mode=%s latest=%s age=%.3fs queue=%s",
+                self.__class__.__name__,
+                gate_wait_s,
+                turn_id,
+                turn_revision,
+                mode,
+                is_latest,
+                self._item_age_s(item),
+                self._safe_qsize(),
+            )
+
+        if not is_latest:
+            queued_drops = self._drop_stale_queued_inputs()
+            self._log_stale_turn_item(item, "input", queued_drops=queued_drops)
+            return False
+        return True
+
+    def should_emit_output(self, output: STTOut) -> bool:
+        if isinstance(output, PartialTranscription) and self._is_completed_final_revision(output):
+            self._log_stale_turn_item(output, "output-after-final")
+            return False
+
+        if not self._is_latest_turn_item(output, wait_for_pending_reopen=True, wait_for_stability=False):
+            self._log_stale_turn_item(output, "output")
+            return False
+        return True
+
+    def before_emit_output(self, output: STTOut) -> None:
+        if isinstance(output, Transcription):
+            self._mark_completed_final_revision(output)
+
+    def _is_latest_turn_item(
+        self,
+        item: object,
+        *,
+        wait_for_pending_reopen: bool,
+        wait_for_stability: bool,
+    ) -> bool:
+        if self.speculative_turns is None:
+            return True
+        turn_id = getattr(item, "turn_id", None)
+        turn_revision = getattr(item, "turn_revision", None)
+        if turn_id is None or turn_revision is None:
+            return True
+
+        if wait_for_stability:
+            is_latest = self.speculative_turns.is_latest_after_stability_window(
+                turn_id,
+                turn_revision,
+                self.final_revision_settle_s,
+            )
+        elif wait_for_pending_reopen:
+            is_latest = self.speculative_turns.is_latest_after_pending_reopen(turn_id, turn_revision)
+        else:
+            is_latest = self.speculative_turns.is_latest(turn_id, turn_revision)
+        return is_latest
+
+    def _drop_stale_queued_inputs(self) -> int:
+        if self.speculative_turns is None or not hasattr(self.queue_in, "mutex") or not hasattr(self.queue_in, "queue"):
+            return 0
+
+        dropped = 0
+        with self.queue_in.mutex:
+            kept: list[Any] = []
+            while self.queue_in.queue:
+                queued_item = self.queue_in.queue.popleft()
+                if isinstance(queued_item, VADAudio) and (
+                    self._is_completed_final_revision(queued_item)
+                    or (queued_item.mode == "progressive" and self._has_queued_final_for_revision_locked(queued_item))
+                    or not self._is_latest_turn_item(
+                        queued_item,
+                        wait_for_pending_reopen=False,
+                        wait_for_stability=False,
+                    )
+                ):
+                    dropped += 1
+                else:
+                    kept.append(queued_item)
+            self.queue_in.queue.extend(kept)
+            if dropped:
+                self.queue_in.not_full.notify_all()
+        return dropped
+
+    def _log_stale_turn_item(self, item: object, stage: str, *, queued_drops: int = 0) -> None:
+        turn_id = getattr(item, "turn_id", None)
+        turn_revision = getattr(item, "turn_revision", None)
+        if turn_id is None or turn_revision is None:
+            return
+
+        if not hasattr(self, "_stale_drop_counts"):
+            self._stale_drop_counts: Counter[tuple[str, str, int]] = Counter()
+        key = (stage, turn_id, turn_revision)
+        self._stale_drop_counts[key] += 1
+
+        message = "%s: dropping stale STT %s for turn=%s rev=%s age=%.3fs"
+        args: tuple[object, ...] = (
+            self.__class__.__name__,
+            stage,
+            turn_id,
+            turn_revision,
+            self._item_age_s(item),
+        )
+        if queued_drops:
+            message += " (+%d queued)"
+            args = (*args, queued_drops)
+
+        if self._stale_drop_counts[key] == 1:
+            logger.info(message, *args)
+        else:
+            logger.debug(message, *args)
+
+    def _item_age_s(self, item: object) -> float:
+        created_at_s = getattr(item, "created_at_s", None)
+        if not isinstance(created_at_s, float):
+            return 0.0
+        return max(0.0, perf_counter() - created_at_s)
+
+    def _safe_qsize(self) -> int | str:
+        try:
+            return self.queue_in.qsize()
+        except NotImplementedError:
+            return "unknown"
+
+    def _has_queued_final_for_revision(self, item: object) -> bool:
+        if not hasattr(self.queue_in, "mutex") or not hasattr(self.queue_in, "queue"):
+            return False
+        with self.queue_in.mutex:
+            return self._has_queued_final_for_revision_locked(item)
+
+    def _has_queued_final_for_revision_locked(self, item: object) -> bool:
+        key = self._revision_key(item)
+        if key is None:
+            return False
+        return any(
+            isinstance(queued_item, VADAudio) and queued_item.mode == "final" and self._revision_key(queued_item) == key
+            for queued_item in self.queue_in.queue
+        )
+
+    def _revision_key(self, item: object) -> tuple[str, int] | None:
+        turn_id = getattr(item, "turn_id", None)
+        turn_revision = getattr(item, "turn_revision", None)
+        if not isinstance(turn_id, str) or not isinstance(turn_revision, int):
+            return None
+        return (turn_id, turn_revision)
+
+    def _completed_final_revisions(self) -> OrderedDict[tuple[str, int], None]:
+        if not hasattr(self, "_completed_final_revision_keys"):
+            self._completed_final_revision_keys: OrderedDict[tuple[str, int], None] = OrderedDict()
+        return self._completed_final_revision_keys
+
+    def _is_completed_final_revision(self, item: object) -> bool:
+        key = self._revision_key(item)
+        return key is not None and key in self._completed_final_revisions()
+
+    def _mark_completed_final_revision(self, output: Transcription) -> None:
+        key = self._revision_key(output)
+        if key is None:
+            return
+        completed = self._completed_final_revisions()
+        completed[key] = None
+        completed.move_to_end(key)
+        while len(completed) > self._MAX_COMPLETED_FINAL_REVISIONS:
+            completed.popitem(last=False)
+
+    def on_session_end(self) -> None:
+        if hasattr(self, "_completed_final_revision_keys"):
+            self._completed_final_revision_keys.clear()

backend/vendor/s2s-cloud/src/speech_to_speech/STT/dashscope_asr_handler.py

@@ -0,0 +1,188 @@
+"""DashScope (Aliyun Bailian) streaming ASR STT handler — Omni realtime API.
+
+Uses dashscope.audio.qwen_omni.OmniRealtimeConversation in ASR-only mode
+(output_modalities=[TEXT] + input audio transcription) with the
+qwen3-asr-flash-realtime model. Each VAD segment is uploaded (streamed in
+chunks) and committed; partial/final transcriptions are yielded to the
+pipeline. VAD stays local, exactly as faster-whisper used to behave.
+"""
+
+from __future__ import annotations
+
+import base64
+import logging
+import os
+import queue
+import threading
+import time
+from typing import Any, Iterator
+
+import numpy as np
+
+from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
+from speech_to_speech.pipeline.handler_types import STTIn, STTOut
+from speech_to_speech.pipeline.messages import PartialTranscription, Transcription
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_MODEL = "qwen3-asr-flash-realtime"
+DEFAULT_SAMPLE_RATE = 16000
+CONNECT_RETRIES = 4
+CONNECT_BACKOFF_S = 2.0
+EVENT_TIMEOUT_S = 15.0
+APPEND_CHUNK_SAMPLES = 16000 * 2  # 2s chunks
+
+
+class DashScopeASRSTTHandler(BaseSTTHandler):
+    """Streaming Chinese ASR via DashScope Omni realtime conversation (ASR-only)."""
+
+    def setup(
+        self,
+        api_key: str = "",
+        model: str = DEFAULT_MODEL,
+        language: str = "zh",
+        sample_rate: int = DEFAULT_SAMPLE_RATE,
+        gen_kwargs: dict[str, Any] | None = None,
+    ) -> None:
+        import dashscope  # pip install dashscope>=1.20 (lazy: only when this backend is used)
+
+        dashscope.api_key = api_key or os.environ.get("DASHSCOPE_API_KEY", "")
+        if not dashscope.api_key:
+            raise ValueError("DASHSCOPE_API_KEY is not set (pass --dashscope_asr_api_key or set env).")
+        self.model = model
+        self.language = language
+        self.sample_rate = int(sample_rate)
+        logger.info(
+            "DashScopeASRSTTHandler ready (model=%s, lang=%s, sr=%d)",
+            self.model,
+            self.language,
+            self.sample_rate,
+        )
+
+    @staticmethod
+    def _float32_to_pcm16(audio: np.ndarray) -> bytes:
+        if audio.dtype == np.int16:
+            return audio.tobytes()
+        return (np.clip(audio, -1.0, 1.0) * 32767).astype(np.int16).tobytes()
+
+    def _connect_with_retry(self, conversation: Any) -> None:
+        last_exc: Exception | None = None
+        for attempt in range(CONNECT_RETRIES):
+            try:
+                conversation.connect()
+                return
+            except Exception as e:  # noqa: BLE001
+                last_exc = e
+                logger.warning("DashScopeASR connect attempt %d failed: %s", attempt + 1, e)
+                time.sleep(CONNECT_BACKOFF_S)
+        raise RuntimeError(f"DashScopeASR could not connect after {CONNECT_RETRIES} attempts: {last_exc}")
+
+    def process(self, vad_audio: STTIn) -> Iterator[STTOut]:
+        # v1: cloud ASR opens one websocket session per utterance. Progressive
+        # (live) segments are skipped — transcribing every 0.5s revision would
+        # re-open sessions serially and stall the pipeline. Launch with
+        # --enable_live_transcription false so VAD emits one final segment per
+        # utterance; keep this guard as defense-in-depth.
+        if vad_audio.mode == "progressive":
+            return
+
+        from dashscope.audio.qwen_omni import MultiModality, OmniRealtimeConversation
+        from dashscope.audio.qwen_omni.omni_realtime import TranscriptionParams
+
+        pcm = self._float32_to_pcm16(vad_audio.audio)
+
+        events: "queue.Queue[tuple[str, str]]" = queue.Queue()
+        partial_text = ""
+        final_text = ""
+
+        class Callback:
+            def on_open(self) -> None:
+                events.put(("open", ""))
+
+            def on_close(self, *args: Any) -> None:
+                events.put(("close", ""))
+
+            def on_error(self, error: Any) -> None:
+                events.put(("error", str(error)))
+
+            def on_event(self, message: Any) -> None:
+                event_type = message.get("type") if isinstance(message, dict) else str(message)
+                if event_type == "conversation.item.input_audio_transcription.text":
+                    stash = message.get("stash", "") if isinstance(message, dict) else ""
+                    events.put(("partial", str(stash)))
+                elif event_type == "conversation.item.input_audio_transcription.completed":
+                    transcript = message.get("transcript", "") if isinstance(message, dict) else ""
+                    events.put(("final", str(transcript)))
+
+        conversation = OmniRealtimeConversation(
+            model=self.model,
+            url="wss://dashscope.aliyuncs.com/api-ws/v1/realtime",
+            callback=Callback(),
+        )
+        self._connect_with_retry(conversation)
+        try:
+            conversation.update_session(
+                output_modalities=[MultiModality.TEXT],
+                enable_input_audio_transcription=True,
+                # Disable the server-side VAD: it races our explicit commit()
+                # (auto-consuming the audio buffer can make commit fail with
+                # "input messages do not contain elements with the role of user").
+                enable_turn_detection=False,
+                transcription_params=TranscriptionParams(
+                    language=self.language,
+                    sample_rate=self.sample_rate,
+                    input_audio_format="pcm",
+                ),
+            )
+            chunk_bytes = APPEND_CHUNK_SAMPLES * 2
+            for i in range(0, len(pcm), chunk_bytes):
+                conversation.append_audio(base64.b64encode(pcm[i : i + chunk_bytes]).decode())
+                time.sleep(0.05)
+            time.sleep(1.0)  # let the server ingest the trailing audio before commit
+            conversation.commit()
+
+            deadline = time.monotonic() + EVENT_TIMEOUT_S
+            while time.monotonic() < deadline:
+                try:
+                    kind, payload = events.get(timeout=1.0)
+                except queue.Empty:
+                    continue
+                if kind == "partial":
+                    partial_text = payload
+                    if vad_audio.mode == "progressive":
+                        yield PartialTranscription(
+                            text=payload,
+                            turn_id=vad_audio.turn_id,
+                            turn_revision=vad_audio.turn_revision,
+                        )
+                elif kind == "final":
+                    final_text = payload
+                    if not vad_audio.mode == "progressive":
+                        yield Transcription(
+                            text=payload,
+                            language_code=self.language,
+                            turn_id=vad_audio.turn_id,
+                            turn_revision=vad_audio.turn_revision,
+                            speech_stopped_at_s=vad_audio.created_at_s,
+                        )
+                        return
+                elif kind == "error":
+                    logger.error("DashScope ASR error: %s", payload)
+                    return
+                elif kind == "close":
+                    break
+        finally:
+            try:
+                conversation.close()
+            except Exception:  # noqa: BLE001
+                pass
+
+        # Fallback: progressive mode never got a final event.
+        if vad_audio.mode != "progressive" and final_text:
+            yield Transcription(
+                text=final_text,
+                language_code=self.language,
+                turn_id=vad_audio.turn_id,
+                turn_revision=vad_audio.turn_revision,
+                speech_stopped_at_s=vad_audio.created_at_s,
+            )

backend/vendor/s2s-cloud/src/speech_to_speech/STT/faster_whisper_handler.py

@@ -0,0 +1,75 @@
+from __future__ import annotations
+
+import logging
+import os
+from typing import Any, Iterator
+
+from faster_whisper import WhisperModel
+from rich.console import Console
+
+from speech_to_speech.pipeline.handler_types import STTIn, STTOut
+from speech_to_speech.pipeline.messages import PartialTranscription, Transcription
+from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
+
+console = Console()
+
+logger = logging.getLogger(__name__)
+
+
+class FasterWhisperSTTHandler(BaseSTTHandler):
+    """
+    Handles the Speech To Text generation using a Whisper model.
+    """
+
+    def setup(
+        self,
+        model_name: str = "tiny.en",
+        device: str = "auto",
+        compute_type: str = "auto",
+        gen_kwargs: dict[str, Any] = {},
+    ) -> None:
+        self.gen_kwargs = self.adapt_gen_kwargs(gen_kwargs)
+
+        os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
+        self.model = WhisperModel(model_name, device=device, compute_type=compute_type)
+
+    def process(self, vad_audio: STTIn) -> Iterator[STTOut]:
+        logger.debug("infering faster whisper...")
+
+        segments, info = self.model.transcribe(vad_audio.audio, **self.gen_kwargs)
+        output_text = []
+
+        for segment in segments:
+            logger.debug("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
+            output_text.append(segment.text)
+
+        pred_text = " ".join(output_text).strip()
+
+        logger.debug("finished whisper inference")
+        if pred_text:
+            console.print(f"[yellow]USER: {pred_text}")
+
+            if vad_audio.mode == "progressive":
+                yield PartialTranscription(
+                    text=pred_text,
+                    turn_id=vad_audio.turn_id,
+                    turn_revision=vad_audio.turn_revision,
+                )
+            else:
+                yield Transcription(
+                    text=pred_text,
+                    turn_id=vad_audio.turn_id,
+                    turn_revision=vad_audio.turn_revision,
+                    speech_stopped_at_s=vad_audio.created_at_s,
+                )
+        else:
+            logger.debug("no text detected. skipping...")
+
+    def cleanup(self) -> None:
+        print("Stopping FasterWhisperSTTHandler")
+        del self.model
+
+    def adapt_gen_kwargs(self, gen_kwargs: dict[str, Any]) -> dict[str, Any]:
+        gen_kwargs["without_timestamps"] = not gen_kwargs.pop("return_timestamps", True)
+
+        return gen_kwargs

backend/vendor/s2s-cloud/src/speech_to_speech/STT/lightning_whisper_mlx_handler.py

@@ -0,0 +1,103 @@
+from __future__ import annotations
+
+import logging
+from typing import Any, Iterator, Optional
+
+import numpy as np
+import torch
+from lightning_whisper_mlx import LightningWhisperMLX
+from rich.console import Console
+
+from speech_to_speech.pipeline.handler_types import STTIn, STTOut
+from speech_to_speech.pipeline.messages import Transcription
+from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
+
+logger = logging.getLogger(__name__)
+
+console = Console()
+
+SUPPORTED_LANGUAGES = [
+    "en",
+    "fr",
+    "es",
+    "zh",
+    "ja",
+    "ko",
+    "hi",
+    "de",
+    "pt",
+    "pl",
+    "it",
+    "nl",
+]
+
+
+class LightningWhisperSTTHandler(BaseSTTHandler):
+    """
+    Handles the Speech To Text generation using a Whisper model.
+    """
+
+    def setup(
+        self,
+        model_name: str = "distil-large-v3",
+        device: str = "mps",
+        torch_dtype: str = "float16",
+        compile_mode: Optional[str] = None,
+        language: Optional[str] = None,
+        gen_kwargs: dict[str, Any] = {},
+    ) -> None:
+        if len(model_name.split("/")) > 1:
+            model_name = model_name.split("/")[-1]
+        self.device = device
+        self.model = LightningWhisperMLX(model=model_name, batch_size=6, quant=None)
+        self.start_language = language
+        self.last_language = language
+
+        self.warmup()
+
+    def warmup(self) -> None:
+        logger.info(f"Warming up {self.__class__.__name__}")
+
+        # 2 warmup steps for no compile or compile mode with CUDA graphs capture
+        n_steps = 1
+        dummy_input = np.array([0] * 512)
+
+        for _ in range(n_steps):
+            _ = self.model.transcribe(dummy_input)["text"].strip()
+
+    def process(self, vad_audio: STTIn) -> Iterator[STTOut]:
+        logger.debug("infering whisper...")
+
+        audio = vad_audio.audio
+        if self.start_language != "auto":
+            transcription_dict = self.model.transcribe(audio, language=self.start_language)
+        else:
+            transcription_dict = self.model.transcribe(audio)
+            language_code = transcription_dict["language"]
+            if language_code not in SUPPORTED_LANGUAGES:
+                logger.warning(f"Whisper detected unsupported language: {language_code}")
+                if self.last_language in SUPPORTED_LANGUAGES:  # reprocess with the last language
+                    transcription_dict = self.model.transcribe(audio, language=self.last_language)
+                else:
+                    transcription_dict = {"text": "", "language": "en"}
+            else:
+                self.last_language = language_code
+
+        pred_text = transcription_dict["text"].strip()
+        language_code = transcription_dict["language"]
+        torch.mps.empty_cache()
+
+        logger.debug("finished whisper inference")
+        console.print(f"[yellow]USER: {pred_text}")
+        logger.debug(f"Language Code Whisper: {language_code}")
+
+        if self.start_language == "auto":
+            language_code += "-auto"
+
+        yield Transcription(
+            text=pred_text,
+            language_code=language_code,
+            turn_id=vad_audio.turn_id,
+            turn_revision=vad_audio.turn_revision,
+            speech_stopped_at_s=vad_audio.created_at_s,
+        )

backend/vendor/s2s-cloud/src/speech_to_speech/STT/mlx_audio_whisper_handler.py

@@ -0,0 +1,155 @@
+from __future__ import annotations
+
+import logging
+from typing import Any, Iterator, Optional
+
+import numpy as np
+from rich.console import Console
+
+from speech_to_speech.pipeline.handler_types import STTIn, STTOut
+from speech_to_speech.pipeline.messages import Transcription
+from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
+
+logger = logging.getLogger(__name__)
+
+console = Console()
+
+SUPPORTED_LANGUAGES = [
+    "en",
+    "fr",
+    "es",
+    "zh",
+    "ja",
+    "ko",
+    "hi",
+    "de",
+    "pt",
+    "pl",
+    "it",
+    "nl",
+]
+
+
+class MLXAudioWhisperSTTHandler(BaseSTTHandler):
+    """
+    Handles the Speech To Text generation using MLX Audio's Whisper implementation.
+    Optimized for Apple Silicon using the MLX framework.
+    """
+
+    def setup(
+        self,
+        model_name: str = "mlx-community/whisper-large-v3-turbo",
+        language: Optional[str] = None,
+        gen_kwargs: dict[str, Any] = {},
+    ) -> None:
+        from mlx_audio.stt.generate import load_model
+        from transformers import WhisperProcessor
+
+        self.model_name = model_name
+        self.start_language = language
+        self.last_language = language
+        self.gen_kwargs = gen_kwargs
+
+        # Load the model directly
+        logger.info(f"Loading model {model_name}...")
+        self.model = load_model(model_name)
+
+        # Check if processor was loaded, if not, load it manually from original model
+        if self.model._processor is None:
+            logger.info("Processor not found in MLX model, loading from original Whisper model...")
+            # Map MLX model names to their original Whisper counterparts
+            processor_model_map = {
+                "mlx-community/whisper-large-v3-turbo": "openai/whisper-large-v3",
+                "mlx-community/whisper-large-v3": "openai/whisper-large-v3",
+                "mlx-community/whisper-medium": "openai/whisper-medium",
+                "mlx-community/whisper-small": "openai/whisper-small",
+                "mlx-community/whisper-base": "openai/whisper-base",
+                "mlx-community/whisper-tiny": "openai/whisper-tiny",
+            }
+
+            # Get the appropriate processor model name
+            processor_model = processor_model_map.get(model_name, "openai/whisper-large-v3")
+            logger.info(f"Loading processor from {processor_model}...")
+
+            try:
+                self.model._processor = WhisperProcessor.from_pretrained(processor_model)
+                logger.info("Processor loaded successfully")
+            except Exception as e:
+                logger.error(f"Failed to load processor: {e}")
+                raise
+
+        logger.info(f"Model {model_name} loaded successfully")
+
+        self.warmup()
+
+    def warmup(self) -> None:
+        logger.info(f"Warming up {self.__class__.__name__}")
+
+        # Warmup with a dummy input
+        dummy_audio = np.zeros(16000, dtype=np.float32)
+
+        try:
+            # Pre-warm the model by running a transcription
+            _ = self.model.generate(dummy_audio, verbose=False)
+            logger.info("Model warmed up and ready")
+        except Exception as e:
+            logger.warning(f"Warmup failed: {e}")
+
+    def process(self, vad_audio: STTIn) -> Iterator[STTOut]:
+        logger.debug("inferring mlx-audio whisper...")
+
+        assert isinstance(vad_audio.audio, np.ndarray), "Audio must be a numpy array"
+        audio_input = vad_audio.audio.astype(np.float32)
+
+        # Prepare generation kwargs - only pass valid parameters
+        gen_kwargs = {}
+
+        # Add language if specified
+        if self.start_language and self.start_language != "auto":
+            gen_kwargs["language"] = self.start_language
+
+        try:
+            # Generate transcription directly using model.generate
+            result = self.model.generate(audio_input, verbose=False, **gen_kwargs)
+
+            # Extract text from result
+            pred_text = result.text.strip() if hasattr(result, "text") else str(result).strip()
+
+            # Try to detect language from result if available
+            if hasattr(result, "language"):
+                language_code = result.language
+            elif self.start_language and self.start_language != "auto":
+                language_code = self.start_language
+            else:
+                # Default to last known language or English
+                language_code = self.last_language if self.last_language else "en"
+
+            # Validate language code
+            if language_code not in SUPPORTED_LANGUAGES:
+                logger.warning(f"Detected unsupported language: {language_code}")
+                if self.last_language in SUPPORTED_LANGUAGES:
+                    language_code = self.last_language
+                else:
+                    language_code = "en"
+            else:
+                self.last_language = language_code
+
+        except Exception as e:
+            logger.error(f"MLX Audio Whisper inference failed: {e}")
+            pred_text = ""
+            language_code = self.last_language if self.last_language else "en"
+
+        logger.debug("finished mlx-audio whisper inference")
+        console.print(f"[yellow]USER: {pred_text}")
+        logger.debug(f"Language Code: {language_code}")
+
+        if self.start_language == "auto":
+            language_code += "-auto"
+
+        yield Transcription(
+            text=pred_text,
+            language_code=language_code,
+            turn_id=vad_audio.turn_id,
+            turn_revision=vad_audio.turn_revision,
+            speech_stopped_at_s=vad_audio.created_at_s,
+        )

backend/vendor/s2s-cloud/src/speech_to_speech/STT/paraformer_handler.py

@@ -0,0 +1,79 @@
+from __future__ import annotations
+
+import logging
+from typing import Any, Iterator
+
+import numpy as np
+import torch
+from rich.console import Console
+
+from speech_to_speech.pipeline.handler_types import STTIn, STTOut
+from speech_to_speech.pipeline.messages import PartialTranscription, Transcription
+from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
+
+logging.basicConfig(
+    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
+)
+logger = logging.getLogger(__name__)
+
+console = Console()
+
+
+class ParaformerSTTHandler(BaseSTTHandler):
+    """
+    Handles the Speech To Text generation using a Paraformer model.
+    The default for this model is set to Chinese.
+    This model was contributed by @wuhongsheng.
+    """
+
+    def setup(
+        self,
+        model_name: str = "paraformer-zh",
+        device: str = "cuda",
+        gen_kwargs: dict[str, Any] = {},
+    ) -> None:
+        print(model_name)
+        if len(model_name.split("/")) > 1:
+            model_name = model_name.split("/")[-1]
+        self.device = device
+        try:
+            from funasr import AutoModel
+        except ModuleNotFoundError as exc:
+            raise ModuleNotFoundError(
+                "Paraformer STT requires the optional 'paraformer' extra. "
+                "Install it with `pip install speech-to-speech[paraformer]`."
+            ) from exc
+        self.model = AutoModel(model=model_name, device=device)
+        self.warmup()
+
+    def warmup(self) -> None:
+        logger.info(f"Warming up {self.__class__.__name__}")
+
+        # 2 warmup steps for no compile or compile mode with CUDA graphs capture
+        n_steps = 1
+        dummy_input = np.array([0] * 512, dtype=np.float32)
+        for _ in range(n_steps):
+            _ = self.model.generate(dummy_input)[0]["text"].strip().replace(" ", "")
+
+    def process(self, vad_audio: STTIn) -> Iterator[STTOut]:
+        logger.debug("infering paraformer...")
+
+        pred_text = self.model.generate(vad_audio.audio)[0]["text"].strip().replace(" ", "")
+        torch.mps.empty_cache()
+
+        logger.debug("finished paraformer inference")
+        console.print(f"[yellow]USER: {pred_text}")
+
+        if vad_audio.mode == "progressive":
+            yield PartialTranscription(
+                text=pred_text,
+                turn_id=vad_audio.turn_id,
+                turn_revision=vad_audio.turn_revision,
+            )
+        else:
+            yield Transcription(
+                text=pred_text,
+                turn_id=vad_audio.turn_id,
+                turn_revision=vad_audio.turn_revision,
+                speech_stopped_at_s=vad_audio.created_at_s,
+            )

backend/vendor/s2s-cloud/src/speech_to_speech/STT/parakeet_tdt_handler.py

@@ -0,0 +1,648 @@
+"""
+Parakeet TDT Speech-to-Text Handler
+
+Supports NVIDIA Parakeet TDT model for high-quality multilingual ASR.
+- On Apple Silicon (MPS): Uses mlx-audio with mlx-community/parakeet-tdt-0.6b-v3
+- On CUDA/CPU: Uses nano-parakeet (pure PyTorch) with nvidia/parakeet-tdt-0.6b-v3
+
+Model supports 25 European languages with automatic language detection.
+"""
+
+from __future__ import annotations
+
+import logging
+from contextlib import contextmanager
+from sys import platform
+from threading import Lock
+from time import perf_counter
+from typing import Any, Iterator, Optional
+
+import numpy as np
+from rich.console import Console
+from rich.text import Text
+
+from speech_to_speech.pipeline.handler_types import STTIn, STTOut
+from speech_to_speech.pipeline.messages import PartialTranscription, Transcription
+from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
+from speech_to_speech.STT.smart_progressive_streaming import PartialTranscription as ProgressiveStreamPartial
+from speech_to_speech.utils.mlx_lock import MLXLockContext
+
+try:
+    from lingua import Language, LanguageDetectorBuilder
+
+    LINGUA_AVAILABLE = True
+except ImportError:
+    LINGUA_AVAILABLE = False
+
+logger = logging.getLogger(__name__)
+console = Console()
+
+# Parakeet TDT v3 supports 25 European languages
+SUPPORTED_LANGUAGES = [
+    "en",
+    "de",
+    "fr",
+    "es",
+    "it",
+    "pt",
+    "nl",
+    "pl",
+    "ru",
+    "uk",
+    "cs",
+    "sk",
+    "hu",
+    "ro",
+    "bg",
+    "hr",
+    "sl",
+    "sr",
+    "da",
+    "no",
+    "sv",
+    "fi",
+    "et",
+    "lv",
+    "lt",
+]
+
+# Lingua uses "nb" (Bokmål) for Norwegian instead of "no"
+_LINGUA_CODE_MAP = {"no": "nb"}
+
+if LINGUA_AVAILABLE:
+    _lingua_iso_to_code = {
+        lang.iso_code_639_1.name.lower(): lang for lang in Language.all() if lang.iso_code_639_1 is not None
+    }
+    _lingua_languages = [
+        _lingua_iso_to_code[_LINGUA_CODE_MAP.get(code, code)]
+        for code in SUPPORTED_LANGUAGES
+        if _LINGUA_CODE_MAP.get(code, code) in _lingua_iso_to_code
+    ]
+
+    def _build_lingua_detector():
+        # Preloading can take multiple seconds on some hardware, including the
+        # deployed server. Pay that cost at startup instead of on the first user
+        # request, where it would look like slow STT.
+        return LanguageDetectorBuilder.from_languages(*_lingua_languages).with_preloaded_language_models().build()
+
+    _lingua_detector = _build_lingua_detector()
+
+
+class ParakeetTDTSTTHandler(BaseSTTHandler):
+    """
+    Handles Speech-to-Text using NVIDIA Parakeet TDT model.
+
+    On Apple Silicon (MPS): Uses mlx-audio with the MLX-converted model.
+    On CUDA/CPU: Uses nano-parakeet (pure PyTorch) for NeMo-free inference.
+
+    Parakeet TDT 0.6B v3 is a 600M parameter multilingual ASR model
+    supporting 25 European languages with automatic language detection.
+    """
+
+    def setup(
+        self,
+        model_name: Optional[str] = None,
+        device: str = "auto",
+        compute_type: str = "float16",
+        language: Optional[str] = None,
+        gen_kwargs: dict[str, Any] = {},
+        enable_live_transcription: bool = False,
+        live_transcription_update_interval: float = 0.5,
+    ) -> None:
+        """
+        Initialize the Parakeet TDT model.
+
+        Args:
+            model_name: Model identifier. Defaults are:
+                - MPS: "mlx-community/parakeet-tdt-0.6b-v3"
+                - CUDA/CPU: "nvidia/parakeet-tdt-0.6b-v3"
+            device: Device to use ("auto", "cuda", "mps", "cpu")
+            compute_type: Compute precision ("float16", "float32")
+            language: Target language code (optional, model auto-detects)
+            gen_kwargs: Additional generation kwargs
+        """
+        self.gen_kwargs = gen_kwargs
+        self.start_language = language
+        self.last_language = language if language else "en"
+        self.enable_live_transcription = enable_live_transcription
+        self.live_transcription_update_interval = live_transcription_update_interval
+        self.compute_lock = Lock()
+        self.sample_rate = 16000
+
+        # Determine device
+        if device == "auto":
+            if platform == "darwin":
+                self.device = "mps"
+            else:
+                import torch
+
+                self.device = "cuda" if torch.cuda.is_available() else "cpu"
+        else:
+            self.device = device
+
+        # Set default model based on device
+        if model_name is None:
+            if self.device == "mps":
+                model_name = "mlx-community/parakeet-tdt-0.6b-v3"
+            else:
+                model_name = "nvidia/parakeet-tdt-0.6b-v3"
+
+        self.model_name = model_name
+        self.compute_type = compute_type
+
+        logger.info(f"Loading Parakeet TDT model: {model_name} on {self.device}")
+
+        if self.device == "mps":
+            self._setup_mlx(model_name)
+        else:
+            self._setup_nano_parakeet(model_name)
+
+        # Setup streaming handler if live transcription is enabled
+        self.streaming_handler = None
+        if self.enable_live_transcription:
+            from speech_to_speech.STT.smart_progressive_streaming import (
+                SmartProgressiveStreamingHandler,
+            )
+
+            self.streaming_handler = SmartProgressiveStreamingHandler(
+                self.model,
+                emission_interval=self.live_transcription_update_interval,
+                max_window_size=15.0,
+                sentence_buffer=2.0,
+            )
+            self.processing_final = False  # Track if we're processing final audio
+            logger.info(f"Live transcription enabled for Parakeet TDT ({self.backend})")
+        self._live_transcription_active = False
+        self._live_turn_key: tuple[str | None, int | None] | None = None
+
+        self.warmup()
+
+    def _setup_mlx(self, model_name: str) -> None:
+        """Setup for Apple Silicon using mlx-audio."""
+        try:
+            from mlx_audio.stt.generate import load_model
+
+            self.backend = "mlx"
+            self.model = load_model(model_name)
+            logger.info("MLX Audio Parakeet model loaded successfully")
+        except ImportError as e:
+            raise ImportError(
+                "mlx-audio is required for Parakeet TDT on Apple Silicon. Install with: pip install mlx-audio"
+            ) from e
+
+    def _setup_nano_parakeet(self, model_name: str) -> None:
+        """Setup for CUDA/CPU using nano-parakeet."""
+        try:
+            import torch
+            from nano_parakeet import from_pretrained
+
+            self.backend = "nano_parakeet"
+
+            if self.device == "cuda" and not torch.cuda.is_available():
+                logger.warning("CUDA requested but not available. Falling back to CPU for nano-parakeet.")
+                self.device = "cpu"
+
+            self.model = from_pretrained(model_name=model_name, device=self.device)
+
+            logger.info(f"nano-parakeet model loaded successfully on {self.device}")
+        except ImportError as e:
+            raise ImportError(
+                "nano-parakeet is required for Parakeet TDT on CUDA/CPU. Install with: pip install nano-parakeet"
+            ) from e
+
+    def warmup(self) -> None:
+        """Warm up the model with a dummy input."""
+        logger.info(f"Warming up {self.__class__.__name__}")
+
+        # Create 1 second of silence at 16kHz
+        dummy_audio = np.zeros(16000, dtype=np.float32)
+
+        try:
+            if self.backend == "mlx":
+                import mlx.core as mx
+
+                # Convert to mx.array and call decode_chunk directly
+                audio_mx = mx.array(dummy_audio, dtype=mx.float32)
+                _ = self.model.decode_chunk(audio_mx, verbose=False)
+            elif self.backend == "nano_parakeet":
+                _ = self.model.transcribe(dummy_audio)
+            else:
+                _ = self.model.transcribe([dummy_audio], batch_size=1, verbose=False)
+
+            logger.info("Model warmed up and ready")
+        except Exception as e:
+            logger.warning(f"Warmup failed: {e}")
+
+    def process(self, vad_audio: STTIn) -> Iterator[STTOut]:
+        """
+        Process audio and generate transcription.
+
+        Yields:
+            :class:`PartialTranscription` or :class:`Transcription`
+        """
+        process_start_s = perf_counter()
+        is_progressive = vad_audio.mode == "progressive"
+        audio_input = vad_audio.audio
+
+        # Ensure audio is float32 numpy array
+        if not isinstance(audio_input, np.ndarray):
+            audio_input = np.array(audio_input, dtype=np.float32)
+        else:
+            audio_input = audio_input.astype(np.float32)
+        audio_duration_s = len(audio_input) / getattr(self, "sample_rate", 16000)
+        item_age_s = self._item_age_s(vad_audio)
+
+        self._prepare_live_transcription_turn(vad_audio.turn_id, vad_audio.turn_revision)
+
+        # Handle progressive updates: yield tagged partial for TranscriptionNotifier
+        if self.enable_live_transcription and is_progressive:
+            # Ignore progressive updates if we're already processing final audio
+            if self.processing_final:
+                logger.debug("Skipping stale progressive update (final audio already received)")
+                return
+
+            # Try to acquire lock with short timeout - skip if busy
+            lock_scope_start_s = perf_counter()
+            with self._compute_lock_context(handler_name="ParakeetSTT-Progressive", timeout=0.01) as acquired:
+                if acquired:
+                    try:
+                        inference_start_s = perf_counter()
+                        progressive_text = self._show_progressive_transcription(audio_input)
+                        inference_s = perf_counter() - inference_start_s
+                        if inference_s >= 0.25:
+                            logger.info(
+                                "Parakeet progressive STT timing turn=%s rev=%s audio=%.3fs age=%.3fs "
+                                "lock_scope=%.3fs inference=%.3fs chars=%d",
+                                vad_audio.turn_id,
+                                vad_audio.turn_revision,
+                                audio_duration_s,
+                                item_age_s,
+                                perf_counter() - lock_scope_start_s,
+                                inference_s,
+                                len(progressive_text),
+                            )
+                        if progressive_text:
+                            yield PartialTranscription(
+                                text=progressive_text,
+                                turn_id=vad_audio.turn_id,
+                                turn_revision=vad_audio.turn_revision,
+                            )
+                            return
+                    except Exception as e:
+                        logger.debug(f"Progressive transcription failed: {e}")
+                else:
+                    logger.debug("Skipping progressive update (compute busy)")
+            return
+
+        # Handle final transcription (send to LLM)
+        logger.info(
+            "Parakeet final STT start turn=%s rev=%s audio=%.3fs age=%.3fs",
+            vad_audio.turn_id,
+            vad_audio.turn_revision,
+            audio_duration_s,
+            item_age_s,
+        )
+        inference_s = 0.0
+        lock_scope_s = 0.0
+        try:
+            if self.enable_live_transcription:
+                # Mark that we're processing final audio (ignore stale progressive updates)
+                self.processing_final = True
+
+            # Acquire lock with longer timeout for final transcription
+            lock_scope_start_s = perf_counter()
+            with self._compute_lock_context(handler_name="ParakeetSTT-Final", timeout=5.0) as acquired:
+                lock_scope_s = perf_counter() - lock_scope_start_s
+                if not acquired:
+                    logger.error("Failed to acquire compute lock for final transcription")
+                    pred_text = ""
+                    language_code = self.last_language
+                else:
+                    inference_start_s = perf_counter()
+                    if self.backend == "mlx":
+                        pred_text, language_code = self._process_mlx_final(audio_input)
+                    else:
+                        pred_text, language_code = self._process_nano_parakeet(audio_input)
+                    inference_s = perf_counter() - inference_start_s
+                    lock_scope_s = perf_counter() - lock_scope_start_s
+
+            # Validate and update language
+            if language_code and language_code in SUPPORTED_LANGUAGES:
+                self.last_language = language_code
+            else:
+                language_code = self.last_language
+
+        except Exception as e:
+            logger.error(f"Parakeet TDT inference failed: {e}")
+            pred_text = ""
+            language_code = self.last_language
+
+        total_s = perf_counter() - process_start_s
+        logger.info(
+            "Parakeet final STT done turn=%s rev=%s total=%.3fs lock_scope=%.3fs inference=%.3fs chars=%d",
+            vad_audio.turn_id,
+            vad_audio.turn_revision,
+            total_s,
+            lock_scope_s,
+            inference_s,
+            len(pred_text),
+        )
+        logger.debug("Finished Parakeet TDT inference")
+        self._clear_live_transcription_line()
+        if pred_text.strip():
+            console.print(f"[yellow]USER: {pred_text.strip()}")
+            if language_code:
+                console.print(f"[dim]Language: {language_code}[/dim]")
+
+        # Reset per-utterance live transcription state only after final STT
+        # completes. The streaming handler carries fixed sentence timing within
+        # an utterance, and stale timing must not leak into the next turn.
+        if self.enable_live_transcription:
+            self.processing_final = False
+            self._reset_live_transcription_state(clear_turn=True)
+
+        yield Transcription(
+            text=pred_text,
+            language_code=language_code,
+            turn_id=vad_audio.turn_id,
+            turn_revision=vad_audio.turn_revision,
+            speech_stopped_at_s=vad_audio.created_at_s,
+        )
+
+    @property
+    def timing_log_level(self) -> int:
+        return logging.INFO
+
+    def should_log_timing(self, output: STTOut) -> bool:
+        return isinstance(output, Transcription) and self.last_time > self.min_time_to_debug
+
+    def _detect_language_from_text(self, text: str) -> Optional[str]:
+        """
+        Detect language from transcribed text using lingua-py.
+
+        Args:
+            text: Transcribed text string
+
+        Returns:
+            Detected language code or None if detection fails
+        """
+        if not LINGUA_AVAILABLE:
+            logger.warning("lingua-py not available, cannot detect language from text")
+            return None
+
+        # Skip very short utterances where language ID is still too noisy.
+        if not text or len(text.strip()) < 20:
+            return None
+
+        detected = _lingua_detector.detect_language_of(text)
+        if detected is None:
+            return None
+
+        code = detected.iso_code_639_1.name.lower()
+        # Map back lingua-specific codes to our supported codes
+        return {v: k for k, v in _LINGUA_CODE_MAP.items()}.get(code, code)
+
+    @contextmanager
+    def _compute_lock_context(self, handler_name: str, timeout: float) -> Iterator[bool]:
+        if self.backend == "mlx":
+            with MLXLockContext(handler_name=handler_name, timeout=timeout) as acquired:
+                yield acquired
+            return
+
+        lock_start_s = perf_counter()
+        acquired = self.compute_lock.acquire(timeout=timeout)
+        wait_s = perf_counter() - lock_start_s
+        hold_start_s: float | None = None
+        if acquired:
+            if wait_s >= 0.25:
+                logger.info("%s: compute lock acquired after %.2fs", handler_name, wait_s)
+            else:
+                logger.debug("%s: compute lock acquired after %.3fs", handler_name, wait_s)
+            hold_start_s = perf_counter()
+        else:
+            logger.warning("%s: Failed to acquire compute lock after %.3fs (timeout=%s)", handler_name, wait_s, timeout)
+        try:
+            yield acquired
+        finally:
+            if acquired:
+                assert hold_start_s is not None
+                self.compute_lock.release()
+                hold_s = perf_counter() - hold_start_s
+                if hold_s >= 0.25:
+                    logger.info("%s: compute lock released after holding %.2fs", handler_name, hold_s)
+                else:
+                    logger.debug("%s: compute lock released after holding %.3fs", handler_name, hold_s)
+
+    def _show_progressive_transcription(self, audio_input: np.ndarray) -> str:
+        """Run progressive transcription, print to console, and return the text."""
+        result = self.streaming_handler.transcribe_incremental(audio_input)
+        rich_text = Text()
+        if result.fixed_text:
+            rich_text.append("Live: ", style="dim")
+            rich_text.append(result.fixed_text, style="yellow")
+            if result.active_text:
+                rich_text.append(" ", style="dim")
+
+        if result.active_text:
+            if not result.fixed_text:
+                rich_text.append("Live: ", style="dim")
+            rich_text.append(result.active_text, style="cyan dim")
+
+        progressive_text = self._build_progressive_text(result)
+        if progressive_text:
+            self._print_live_transcription(rich_text, progressive_text)
+
+        return progressive_text
+
+    def _print_live_transcription(self, rich_text: Text, progressive_text: str) -> None:
+        is_terminal = bool(getattr(console, "is_terminal", False))
+        if is_terminal:
+            self._write_live_control("\r\x1b[2K")
+            if rich_text:
+                console.print(self._truncate_live_transcription(rich_text), end="")
+            else:
+                fallback = Text("Live: ", style="dim")
+                fallback.append(progressive_text, style="cyan dim")
+                console.print(self._truncate_live_transcription(fallback), end="")
+            self._write_live_control("\r")
+            self._live_transcription_active = True
+            return
+
+        if rich_text:
+            console.print(rich_text)
+        else:
+            console.print(f"[dim]Live: [/dim]{progressive_text}")
+
+    def _clear_live_transcription_line(self) -> None:
+        if not getattr(self, "_live_transcription_active", False):
+            return
+        self._write_live_control("\r\x1b[2K")
+        self._live_transcription_active = False
+
+    def _write_live_control(self, sequence: str) -> None:
+        file = getattr(console, "file", None)
+        if file is None:
+            return
+        file.write(sequence)
+        file.flush()
+
+    def _truncate_live_transcription(self, text: Text) -> Text:
+        text = text.copy()
+        width = getattr(console, "width", 80)
+        try:
+            max_width = max(1, int(width) - 1)
+        except (TypeError, ValueError):
+            max_width = 79
+        text.truncate(max_width, overflow="ellipsis")
+        return text
+
+    def _prepare_live_transcription_turn(self, turn_id: str | None, turn_revision: int | None) -> None:
+        if not self.enable_live_transcription:
+            return
+        turn_key = (turn_id, turn_revision)
+        if getattr(self, "_live_turn_key", None) == turn_key:
+            return
+        self._reset_live_transcription_state(clear_turn=False)
+        self._live_turn_key = turn_key
+
+    def _reset_live_transcription_state(self, clear_turn: bool) -> None:
+        self._clear_live_transcription_line()
+        streaming_handler = getattr(self, "streaming_handler", None)
+        if streaming_handler is not None:
+            streaming_handler.reset()
+        if clear_turn:
+            self._live_turn_key = None
+
+    def _build_progressive_text(self, result: ProgressiveStreamPartial) -> str:
+        parts = []
+        if result.fixed_text:
+            parts.append(result.fixed_text.strip())
+        if result.active_text:
+            parts.append(result.active_text.strip())
+        return " ".join(part for part in parts if part).strip()
+
+    def _process_mlx_final(self, audio_input: np.ndarray) -> tuple[str, str]:
+        """Process final audio using MLX backend with streaming handler."""
+        # If we have fixed sentences from progressive updates, only transcribe the new part
+        if (
+            self.streaming_handler is not None
+            and hasattr(self.streaming_handler, "fixed_sentences")
+            and self.streaming_handler.fixed_sentences
+        ):
+            self._clear_live_transcription_line()
+
+            # Get fixed text from previous progressive updates
+            fixed_text = " ".join(self.streaming_handler.fixed_sentences).strip()
+
+            # Calculate where fixed part ends in audio
+            fixed_end_time = self.streaming_handler.fixed_end_time
+            sample_rate = 16000
+            fixed_end_sample = int(fixed_end_time * sample_rate)
+            if fixed_end_sample > len(audio_input):
+                logger.warning(
+                    "Ignoring stale progressive fixed text: fixed_end_sample=%d exceeds final audio samples=%d",
+                    fixed_end_sample,
+                    len(audio_input),
+                )
+                pred_text, language_code = self._process_mlx(audio_input)
+                return pred_text, language_code
+
+            # Only transcribe the part after fixed sentences
+            if fixed_end_sample < len(audio_input):
+                remaining_audio = audio_input[fixed_end_sample:]
+
+                import mlx.core as mx
+
+                audio_mx = mx.array(remaining_audio, dtype=mx.float32)
+                result = self.model.decode_chunk(audio_mx, verbose=False)
+
+                if hasattr(result, "text"):
+                    new_text = result.text.strip()
+                else:
+                    new_text = str(result).strip()
+
+                # Combine fixed + new
+                pred_text = f"{fixed_text} {new_text}".strip() if new_text else fixed_text
+            else:
+                # All audio already transcribed in progressive updates
+                pred_text = fixed_text
+
+            # Reset streaming handler for next utterance
+            self.streaming_handler.reset()
+        else:
+            # No progressive updates, transcribe everything
+            pred_text, language_code = self._process_mlx(audio_input)
+            return pred_text, language_code
+
+        # Determine language
+        if self.start_language and self.start_language != "auto":
+            language_code = self.start_language
+        else:
+            detected_lang = self._detect_language_from_text(pred_text)
+            if detected_lang:
+                language_code = detected_lang
+            else:
+                language_code = self.last_language
+
+        return pred_text, language_code
+
+    def _process_mlx(self, audio_input: np.ndarray) -> tuple[str, str]:
+        """Process audio using MLX backend."""
+        import mlx.core as mx
+
+        # Convert numpy array to mx.array
+        audio_mx = mx.array(audio_input, dtype=mx.float32)
+
+        # Call decode_chunk directly with the audio array
+        result = self.model.decode_chunk(audio_mx, verbose=False)
+
+        # Extract text from result
+        if hasattr(result, "text"):
+            pred_text = result.text.strip()
+        else:
+            pred_text = str(result).strip()
+
+        # Determine language:
+        # 1. Use fixed language if specified by user
+        # 2. Try to detect from transcribed text using langdetect
+        # 3. Fall back to last known language
+        if self.start_language and self.start_language != "auto":
+            language_code = self.start_language
+        else:
+            # Detect language from transcribed text
+            detected_lang = self._detect_language_from_text(pred_text)
+            if detected_lang:
+                language_code = detected_lang
+            else:
+                language_code = self.last_language
+
+        return pred_text, language_code
+
+    def _process_nano_parakeet(self, audio_input: np.ndarray) -> tuple[str, str]:
+        """Process audio using nano-parakeet backend."""
+        pred_text = self.model.transcribe(audio_input).strip()
+
+        if self.start_language and self.start_language != "auto":
+            language_code = self.start_language
+        else:
+            detected_lang = self._detect_language_from_text(pred_text)
+            if detected_lang:
+                language_code = detected_lang
+            else:
+                language_code = self.last_language
+
+        return pred_text, language_code
+
+    def cleanup(self) -> None:
+        """Clean up model resources."""
+        logger.info(f"Cleaning up {self.__class__.__name__}")
+        if hasattr(self, "model"):
+            del self.model
+
+    def on_session_end(self) -> None:
+        super().on_session_end()
+        self.last_language = self.start_language if self.start_language else "en"
+        if self.enable_live_transcription:
+            self.processing_final = False
+            self._reset_live_transcription_state(clear_turn=True)
+        logger.debug("Parakeet TDT session state reset")

backend/vendor/s2s-cloud/src/speech_to_speech/STT/smart_progressive_streaming.py

@@ -0,0 +1,343 @@
+#!/usr/bin/env python3
+"""
+Smart Progressive Streaming Handler
+
+Provides frequent partial transcriptions (every 500ms) with:
+- Growing window up to 15s for accuracy
+- Sentence-boundary-aware window sliding for audio > 15s
+- Fixed sentences + active transcription
+"""
+
+from dataclasses import dataclass
+from types import SimpleNamespace
+from typing import Any, Generator
+
+import numpy as np
+
+
+@dataclass
+class PartialTranscription:
+    """Result from progressive streaming."""
+
+    fixed_text: str  # Sentences that won't change
+    active_text: str  # Current partial transcription
+    timestamp: float  # Current position in audio
+    is_final: bool  # True if this is the last update
+
+
+class SmartProgressiveStreamingHandler:
+    """
+    Smart progressive streaming with sentence-aware window management.
+
+    Strategy:
+    1. Emit partial transcriptions every 500ms
+    2. Use growing window (up to 15s) for better accuracy
+    3. When audio > 15s, slide window using sentence boundaries:
+       - Keep completed sentences as "fixed"
+       - Only re-transcribe the "active" portion
+    """
+
+    def __init__(
+        self,
+        model: Any,
+        emission_interval: float = 0.5,
+        max_window_size: float = 15.0,
+        sentence_buffer: float = 2.0,
+    ) -> None:
+        """
+        Args:
+            model: Parakeet model with sentence alignment
+            emission_interval: Emit partial transcription every N seconds (default 500ms)
+            max_window_size: Maximum window size before sliding (default 15s)
+            sentence_buffer: Keep last N seconds of sentences in active window (default 2s)
+        """
+        self.model = model
+        self.emission_interval = emission_interval
+        self.max_window_size = max_window_size
+        self.sentence_buffer = sentence_buffer
+        self.sample_rate = 16000
+
+        # State for incremental streaming
+        self.reset()
+
+    def reset(self) -> None:
+        """Reset state for new streaming session."""
+        self.fixed_sentences: list[str] = []
+        self.fixed_end_time: float = 0.0
+        self.last_transcribed_length: int = 0
+
+    def _decode_window(self, audio_window: np.ndarray) -> Any:
+        """
+        Decode an audio window and return an object with:
+          - text: full transcript for the window
+          - sentences: list of objects with .text and .end (seconds)
+        """
+        if hasattr(self.model, "decode_chunk"):
+            import mlx.core as mx
+
+            audio_mx = mx.array(audio_window, dtype=mx.float32)
+            return self.model.decode_chunk(audio_mx, verbose=False)
+
+        # nano-parakeet: use timestamps from transcribe
+        result = self.model.transcribe(audio_window, timestamps=True)
+        segments = result.timestamp.get("segment", []) if hasattr(result, "timestamp") else []
+        sentences = [SimpleNamespace(text=seg.get("segment", "").strip(), end=seg.get("end", 0.0)) for seg in segments]
+        return SimpleNamespace(text=getattr(result, "text", ""), sentences=sentences)
+
+    def transcribe_incremental(self, audio: np.ndarray) -> PartialTranscription:
+        """
+        Transcribe audio incrementally (for live streaming).
+
+        Call this repeatedly with growing audio buffer.
+        Returns a single PartialTranscription for current state.
+        """
+        # Skip if not enough new audio
+        current_length = len(audio)
+        if current_length < self.sample_rate * 0.5:  # Need at least 500ms
+            return PartialTranscription(
+                fixed_text=" ".join(self.fixed_sentences),
+                active_text="",
+                timestamp=current_length / self.sample_rate,
+                is_final=False,
+            )
+
+        # Skip if no new audio since last transcription
+        if current_length == self.last_transcribed_length:
+            return PartialTranscription(
+                fixed_text=" ".join(self.fixed_sentences),
+                active_text="",
+                timestamp=current_length / self.sample_rate,
+                is_final=False,
+            )
+
+        self.last_transcribed_length = current_length
+
+        # Extract window for transcription (from last fixed sentence to end)
+        window_start_samples = int(self.fixed_end_time * self.sample_rate)
+        audio_window = audio[window_start_samples:]
+
+        # Transcribe current window
+        result = self._decode_window(audio_window)
+
+        # Check if window exceeds max_window_size
+        window_duration = len(audio_window) / self.sample_rate
+
+        if window_duration >= self.max_window_size and len(result.sentences) > 1:
+            # Window is too large - fix some sentences
+            cutoff_time = window_duration - self.sentence_buffer
+
+            # Find sentences to fix
+            new_fixed_sentences = []
+            new_fixed_end_time = self.fixed_end_time
+
+            for sentence in result.sentences:
+                sentence_abs_time = self.fixed_end_time + sentence.end
+
+                if sentence.end < cutoff_time:
+                    # Fix this sentence
+                    new_fixed_sentences.append(sentence.text.strip())
+                    new_fixed_end_time = sentence_abs_time
+                else:
+                    break
+
+            if new_fixed_sentences:
+                self.fixed_sentences.extend(new_fixed_sentences)
+                self.fixed_end_time = new_fixed_end_time
+
+                # Re-transcribe from new fixed point
+                window_start_samples = int(self.fixed_end_time * self.sample_rate)
+                audio_window = audio[window_start_samples:]
+                result = self._decode_window(audio_window)
+
+        # Build output
+        fixed_text = " ".join(self.fixed_sentences)
+        active_text = result.text.strip()
+        timestamp = len(audio) / self.sample_rate
+
+        return PartialTranscription(fixed_text=fixed_text, active_text=active_text, timestamp=timestamp, is_final=False)
+
+    def transcribe_progressive(self, audio: np.ndarray) -> Generator[PartialTranscription, None, None]:
+        """
+        Transcribe audio with smart progressive emissions.
+
+        Yields PartialTranscription with:
+        - fixed_text: Completed sentences (won't change)
+        - active_text: Current partial transcription
+        - timestamp: Current position
+        - is_final: True for last update
+        """
+        position = 0  # Start of current window (in samples)
+        fixed_sentences = []  # List of completed sentence texts
+        fixed_end_time = 0.0  # End time of last fixed sentence
+
+        while position < len(audio):
+            # Determine current window end
+            remaining = (len(audio) - position) / self.sample_rate
+
+            if remaining <= self.emission_interval:
+                # Last chunk - process everything remaining
+                window_end = len(audio)
+                is_final = True
+            else:
+                # Regular chunk
+                window_end = min(len(audio), position + int(self.emission_interval * self.sample_rate))
+                is_final = False
+
+            # Extract window for transcription
+            # Window includes: fixed_end_time to current position
+            # (we re-transcribe from last fixed sentence to get better context)
+            window_start_samples = int(fixed_end_time * self.sample_rate)
+            audio_window = audio[window_start_samples:window_end]
+
+            # Transcribe current window
+            result = self._decode_window(audio_window)
+
+            # Check if window exceeds max_window_size
+            window_duration = (window_end - window_start_samples) / self.sample_rate
+
+            if window_duration >= self.max_window_size and len(result.sentences) > 1:
+                # Window is too large - need to fix some sentences
+                # Strategy: Keep sentences up to (max_window - sentence_buffer) as fixed
+                cutoff_time = window_duration - self.sentence_buffer
+
+                # Find sentences that are completed and before cutoff
+                new_fixed_sentences = []
+                new_fixed_end_time = fixed_end_time
+
+                for sentence in result.sentences:
+                    sentence_abs_time = fixed_end_time + sentence.end
+
+                    if sentence.end < cutoff_time:
+                        # This sentence is complete and old enough to fix
+                        new_fixed_sentences.append(sentence.text.strip())
+                        new_fixed_end_time = sentence_abs_time
+                    else:
+                        # Stop here - keep remaining sentences as active
+                        break
+
+                if new_fixed_sentences:
+                    fixed_sentences.extend(new_fixed_sentences)
+                    fixed_end_time = new_fixed_end_time
+
+                    # Re-transcribe from new fixed_end_time
+                    window_start_samples = int(fixed_end_time * self.sample_rate)
+                    audio_window = audio[window_start_samples:window_end]
+                    result = self._decode_window(audio_window)
+
+            # Build output
+            fixed_text = " ".join(fixed_sentences)
+            active_text = result.text.strip()
+            timestamp = window_end / self.sample_rate
+
+            yield PartialTranscription(
+                fixed_text=fixed_text, active_text=active_text, timestamp=timestamp, is_final=is_final
+            )
+
+            # Move to next position
+            position = window_end
+
+            if is_final:
+                break
+
+
+def demo_smart_progressive() -> None:
+    """Demonstrate smart progressive streaming."""
+    import time
+
+    import soundfile as sf
+    from mlx_audio.stt.generate import load_model
+
+    print("=" * 80)
+    print("SMART PROGRESSIVE STREAMING DEMO")
+    print("=" * 80)
+    print("\nFeatures:")
+    print("  • Partial updates every 500ms")
+    print("  • Growing window up to 15s")
+    print("  • Sentence-aware window sliding for long audio")
+    print()
+
+    # Load model
+    print("Loading model...")
+    model = load_model("mlx-community/parakeet-tdt-0.6b-v3")
+
+    # Load audio
+    audio, sr = sf.read("reachy-voice-test.wav")
+    if len(audio.shape) > 1:
+        audio = audio.mean(axis=1)
+    if sr != 16000:
+        from scipy import signal
+
+        audio = signal.resample(audio, int(len(audio) * 16000 / sr))
+    audio = audio.astype(np.float32)
+
+    # Test cases
+    test_cases = [
+        (audio, "Short (5.8s)"),
+        (np.concatenate([audio] * 3), "Medium (17.5s) - exceeds 15s window"),
+        (np.concatenate([audio] * 5), "Long (29.1s) - multiple sentence fixings"),
+    ]
+
+    for test_audio, label in test_cases:
+        duration = len(test_audio) / 16000
+
+        print(f"\n{'=' * 80}")
+        print(f"TEST: {label} - Duration: {duration:.1f}s")
+        print("=" * 80)
+
+        handler = SmartProgressiveStreamingHandler(
+            model,
+            emission_interval=0.5,  # 500ms updates
+            max_window_size=15.0,  # Max 15s window
+            sentence_buffer=2.0,  # Keep 2s of sentences in active window
+        )
+
+        print("\nProgressive transcriptions (showing every 4th update for readability):")
+        print()
+
+        update_count = 0
+        start_time = time.perf_counter()
+
+        for result in handler.transcribe_progressive(test_audio):
+            update_count += 1
+
+            # Show every other update (every 1s) to keep output manageable
+            if update_count % 2 == 0 or result.is_final:
+                elapsed = time.perf_counter() - start_time
+                marker = "FINAL" if result.is_final else f"Update {update_count}"
+
+                print(f"[{result.timestamp:5.1f}s | {elapsed:.3f}s elapsed] {marker}:")
+
+                if result.fixed_text:
+                    print(f"  Fixed:  {result.fixed_text[:70]}...")
+                print(f"  Active: {result.active_text[:70]}...")
+                print()
+
+        total_time = time.perf_counter() - start_time
+        print(f"Total updates: {update_count}")
+        print(f"Total time: {total_time:.3f}s ({total_time / duration:.3f}s per audio second)")
+
+    print("\n" + "=" * 80)
+    print("KEY INSIGHTS")
+    print("=" * 80)
+    print("""
+1. Frequent updates (500ms):
+   - User sees transcription building in real-time
+   - Low latency feel
+
+2. Growing window (up to 15s):
+   - Better accuracy with more context
+   - No arbitrary chunking for short audio
+
+3. Sentence-aware sliding (for audio > 15s):
+   - Completed sentences become "fixed"
+   - Only active portion re-transcribed
+   - Maintains accuracy while managing window size
+
+4. Performance:
+   - Processing happens DURING user speaking
+   - Final result available immediately when user stops
+    """)
+
+
+if __name__ == "__main__":
+    demo_smart_progressive()

backend/vendor/s2s-cloud/src/speech_to_speech/STT/transcription_notifier.py

@@ -0,0 +1,103 @@
+from __future__ import annotations
+
+import logging
+from queue import Queue
+from threading import Event
+from typing import Iterator, Union
+
+from speech_to_speech.api.openai_realtime.runtime_config import RuntimeConfig
+from speech_to_speech.baseHandler import BaseHandler
+from speech_to_speech.LLM.chat import make_user_message
+from speech_to_speech.pipeline.events import PartialTranscriptionEvent, TranscriptionCompletedEvent
+from speech_to_speech.pipeline.handler_types import LLMIn, STTOut
+from speech_to_speech.pipeline.messages import GenerateResponseRequest, PartialTranscription, Transcription
+from speech_to_speech.pipeline.queue_types import TextEventItem
+
+logger = logging.getLogger(__name__)
+
+
+class TranscriptionNotifier(BaseHandler[STTOut, Union[STTOut, LLMIn]]):
+    """Sits between STT and LLM.
+
+    For **realtime mode** (no ``runtime_config``): emits transcription events
+    on ``text_output_queue`` for protocol translation but yields nothing -- the
+    ``RealtimeService`` builds ``GenerateResponseRequest`` directly.
+
+    For **legacy mode** (``runtime_config`` provided): appends the user
+    message to ``runtime_config.chat`` and yields a
+    ``GenerateResponseRequest`` so the LLM handler receives a uniform input
+    type regardless of pipeline mode.
+    """
+
+    def setup(
+        self,
+        text_output_queue: Queue[TextEventItem] | None = None,
+        runtime_config: RuntimeConfig | None = None,
+        should_listen: Event | None = None,
+    ) -> None:
+        self.text_output_queue = text_output_queue
+        self.runtime_config = runtime_config
+        self.should_listen = should_listen
+
+    def process(self, transcription: STTOut) -> Iterator[Union[STTOut, LLMIn]]:
+        if isinstance(transcription, PartialTranscription):
+            if self.text_output_queue and transcription.text:
+                self.text_output_queue.put(
+                    PartialTranscriptionEvent(
+                        delta=str(transcription.text),
+                        turn_id=transcription.turn_id,
+                        turn_revision=transcription.turn_revision,
+                    )
+                )
+                logger.debug("Partial transcription: %s", str(transcription.text)[:80])
+            return
+
+        if isinstance(transcription, Transcription):
+            text = transcription.text
+            language_code = transcription.language_code
+            turn_id = transcription.turn_id
+            turn_revision = transcription.turn_revision
+            speech_stopped_at_s = transcription.speech_stopped_at_s
+        else:
+            text = transcription
+            language_code = None
+            turn_id = None
+            turn_revision = None
+            speech_stopped_at_s = None
+
+        transcript = str(text)
+        # Always close the client-visible transcription item. Empty final STT
+        # results should not trigger the LLM, but clients may already have
+        # received partial deltas and still need a completed event.
+        if self.text_output_queue is not None:
+            self.text_output_queue.put(
+                TranscriptionCompletedEvent(
+                    transcript=transcript,
+                    language_code=language_code,
+                    turn_id=turn_id,
+                    turn_revision=turn_revision,
+                    speech_stopped_at_s=speech_stopped_at_s,
+                )
+            )
+
+        if not transcript:
+            logger.debug("Transcription completed with empty transcript")
+            if self.should_listen is not None:
+                self.should_listen.set()
+                logger.debug("Empty transcription completed; listening re-enabled")
+            return
+
+        if language_code:
+            logger.info("Transcription completed (language=%s): %s", language_code, transcript)
+        else:
+            logger.info("Transcription completed: %s", transcript)
+
+        if self.runtime_config is not None:
+            self.runtime_config.chat.add_item(make_user_message(transcript))
+            yield GenerateResponseRequest(
+                runtime_config=self.runtime_config,
+                language_code=language_code,
+                turn_id=turn_id,
+                turn_revision=turn_revision,
+                speech_stopped_at_s=speech_stopped_at_s,
+            )

backend/vendor/s2s-cloud/src/speech_to_speech/STT/whisper_stt_handler.py

@@ -0,0 +1,147 @@
+from __future__ import annotations
+
+import logging
+from copy import copy
+from typing import Any, Iterator, Optional
+
+import numpy as np
+import torch
+from rich.console import Console
+from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
+
+from speech_to_speech.pipeline.handler_types import STTIn, STTOut
+from speech_to_speech.pipeline.messages import Transcription
+from speech_to_speech.STT.base_stt_handler import BaseSTTHandler
+
+logger = logging.getLogger(__name__)
+console = Console()
+
+SUPPORTED_LANGUAGES = [
+    "en",
+    "fr",
+    "es",
+    "zh",
+    "ja",
+    "ko",
+    "hi",
+    "de",
+    "pt",
+    "pl",
+    "it",
+    "nl",
+]
+
+
+class WhisperSTTHandler(BaseSTTHandler):
+    """
+    Handles the Speech To Text generation using a Whisper model.
+    """
+
+    def setup(
+        self,
+        model_name: str = "distil-whisper/distil-large-v3",
+        device: str = "cuda",
+        torch_dtype: str = "float16",
+        compile_mode: Optional[str] = None,
+        language: Optional[str] = None,
+        gen_kwargs: dict[str, Any] = {},
+    ) -> None:
+        self.device = device
+        self.torch_dtype = getattr(torch, torch_dtype)
+        self.compile_mode = compile_mode
+        self.gen_kwargs = gen_kwargs
+        self.start_language = language
+        self.last_language = language if language != "auto" else None
+        if self.last_language is not None:
+            self.gen_kwargs["language"] = self.last_language
+
+        self.processor = AutoProcessor.from_pretrained(model_name)
+        self.model = AutoModelForSpeechSeq2Seq.from_pretrained(
+            model_name,
+            torch_dtype=self.torch_dtype,
+        ).to(device)
+
+        # compile
+        if self.compile_mode:
+            self.model.generation_config.cache_implementation = "static"
+            self.model.forward = torch.compile(self.model.forward, mode=self.compile_mode, fullgraph=True)
+        self.warmup()
+
+    def prepare_model_inputs(self, audio: np.ndarray) -> torch.Tensor:
+        input_features = self.processor(audio, sampling_rate=16000, return_tensors="pt").input_features
+        input_features = input_features.to(self.device, dtype=self.torch_dtype)
+
+        return input_features
+
+    def warmup(self) -> None:
+        logger.info(f"Warming up {self.__class__.__name__}")
+
+        # 2 warmup steps for no compile or compile mode with CUDA graphs capture
+        n_steps = 1 if self.compile_mode == "default" else 2
+        dummy_input = torch.randn(
+            (1, self.model.config.num_mel_bins, 3000),
+            dtype=self.torch_dtype,
+            device=self.device,
+        )
+        if self.compile_mode not in (None, "default"):
+            # generating more tokens than previously will trigger CUDA graphs capture
+            # one should warmup with a number of generated tokens above max tokens targeted for subsequent generation
+            # hence, having min_new_tokens < max_new_tokens in the future doesn't make sense
+            warmup_gen_kwargs = {
+                "min_new_tokens": self.gen_kwargs["max_new_tokens"],  # Yes, assign max_new_tokens to min_new_tokens
+                "max_new_tokens": self.gen_kwargs["max_new_tokens"],
+                **self.gen_kwargs,
+            }
+        else:
+            warmup_gen_kwargs = self.gen_kwargs
+
+        if self.device == "cuda":
+            start_event = torch.cuda.Event(enable_timing=True)
+            end_event = torch.cuda.Event(enable_timing=True)
+            torch.cuda.synchronize()
+            start_event.record()
+
+        for _ in range(n_steps):
+            _ = self.model.generate(dummy_input, **warmup_gen_kwargs)
+
+        if self.device == "cuda":
+            end_event.record()
+            torch.cuda.synchronize()
+
+            logger.info(
+                f"{self.__class__.__name__}:  warmed up! time: {start_event.elapsed_time(end_event) * 1e-3:.3f} s"
+            )
+
+    def process(self, vad_audio: STTIn) -> Iterator[STTOut]:
+        logger.debug("infering whisper...")
+
+        input_features = self.prepare_model_inputs(vad_audio.audio)
+        pred_ids = self.model.generate(input_features, **self.gen_kwargs)
+        language_code = self.processor.tokenizer.decode(pred_ids[0, 1])[2:-2]  # remove "<|" and "|>"
+
+        if language_code not in SUPPORTED_LANGUAGES:  # reprocess with the last language
+            logger.warning("Whisper detected unsupported language: %s", language_code)
+            gen_kwargs = copy(self.gen_kwargs)
+            gen_kwargs["language"] = self.last_language
+            language_code = self.last_language
+            pred_ids = self.model.generate(input_features, **gen_kwargs)
+        else:
+            self.last_language = language_code
+
+        pred_text = self.processor.batch_decode(pred_ids, skip_special_tokens=True, decode_with_timestamps=False)[0]
+        language_code = self.processor.tokenizer.decode(pred_ids[0, 1])[2:-2]  # remove "<|" and "|>"
+
+        logger.debug("finished whisper inference")
+        console.print(f"[yellow]USER: {pred_text}")
+        logger.debug(f"Language Code Whisper: {language_code}")
+
+        if self.start_language == "auto":
+            language_code += "-auto"
+
+        yield Transcription(
+            text=pred_text,
+            language_code=language_code,
+            turn_id=vad_audio.turn_id,
+            turn_revision=vad_audio.turn_revision,
+            speech_stopped_at_s=vad_audio.created_at_s,
+        )