MediaPipe 網頁測試

MediaPipe 簡介#

MediaPipe 是一款由 Google 開發並開源的多媒體機器學習模型應用框架。
下面表格是它支持的功能和平台

	Android	iOS	C++	Python	JS	Coral
人臉檢測	✅	✅	✅	✅	✅	✅
人臉網格	✅	✅	✅	✅	✅
虹膜	✅	✅	✅
手部	✅	✅	✅	✅	✅
姿勢	✅	✅	✅	✅	✅
全身	✅	✅	✅	✅	✅
自拍分割	✅	✅	✅	✅	✅
頭髮分割	✅		✅
物體檢測	✅	✅	✅			✅
框追蹤	✅	✅	✅
即時運動追蹤	✅
Objectron	✅		✅	✅	✅
KNIFT	✅
自動翻轉			✅
媒體序列			✅
YouTube 8M			✅

MediaPipe 測試#

我這裡調用 Pose 的 JS API 來在網頁中進行測試
通過 React-Webcam 庫來從相機獲取視頻，再通過 canvas 繪製識別結果

import Webcam from "react-webcam";
import React, { useRef, useEffect, useState } from "react";
import { drawConnectors, drawLandmarks } from "@mediapipe/drawing_utils";
import { Camera } from "@mediapipe/camera_utils";

import { Pose, POSE_CONNECTIONS, POSE_LANDMARKS } from "@mediapipe/pose/pose";

const MPHolistic = () => {
  const webcamRef = useRef(null);
  const canvasRef = useRef(null);

  useEffect(() => {
    const pose = new Pose({
      locateFile: (file) => {
        return `pose/${file}`;
      },
    });
    pose.setOptions({
      modelComplexity: 1,
      smoothLandmarks: true,
      enableSegmentation: true,
      smoothSegmentation: true,
      minDetectionConfidence: 0.5,
      minTrackingConfidence: 0.5,
    });

    pose.onResults(onResults);

    if (
      typeof webcamRef.current !== "undefined" &&
      webcamRef.current !== null
    ) {
      const camera = new Camera(webcamRef.current.video, {
        onFrame: async () => {
          await pose.send({ image: webcamRef.current.video });
          // await holistic.send({ image: webcamRef.current.video })
        },
        width: 1280,
        height: 720,
      });
      camera.start();
    }

  }, []);

  const onResults = async (results) => {
    const videoWidth = webcamRef.current.video.videoWidth;
    const videoHeight = webcamRef.current.video.videoHeight;
    canvasRef.current.width = 1280;
    canvasRef.current.height = 720;

    const canvasElement = canvasRef.current;
    const canvasCtx = canvasElement.getContext("2d");

    canvasCtx.save();
    canvasCtx.clearRect(0, 0, videoWidth, videoHeight);
    canvasCtx.translate(videoWidth, 0)
    canvasCtx.scale(-1, 1)
    canvasCtx.drawImage(
      results.image,
      0,
      0,
      canvasElement.width,
      canvasElement.height
    );


    drawConnectors(canvasCtx, results.poseLandmarks, POSE_CONNECTIONS, { color: "#00FF00", lineWidth: 4 })
    drawLandmarks(canvasCtx, results.poseLandmarks, { color: "#FF0000", lineWidth: 2 })

    canvasCtx.restore();
  };

  const videoConstraints = {
    width: 1280,
    height: 720,
    facingMode: "user",
  };

  return (
    <>
      <div
        style={{
          position: "relative",
          width: "100%",
          height: "100%",
        }}
      >
        <Webcam
          audio={false}
          mirrored={true}
          ref={webcamRef}
          style={{
            position: "absolute",
            marginLeft: "auto",
            marginRight: "auto",
            left: 0,
            right: 0,
            textAlign: "center",
            zindex: 9,
            width: 1280,
            height: 720,
          }}
          videoConstraints={videoConstraints}
        />
        <canvas
          ref={canvasRef}
          style={{
            position: "absolute",
            marginLeft: "auto",
            marginRight: "auto",
            left: 0,
            right: 0,
            textAlign: "center",
            zindex: 9,
            width: 1280,
            height: 720,
          }}
        ></canvas>
      </div>
    </>
  );
};

export default MPHolistic;

效果#

一個測試截圖如下

我這裡也在本機的網頁中測試了一下幀率，大概可以穩定 fps 在 100 左右，識別效果也完全可以接受～