NekoMio

NekoMio

telegram
github

MediaPipe 网页测试

MediaPipe 简介#

MediaPipe 是一款由 Google 开发并开源的多媒体机器学习模型应用框架。
下面表格是它支持的功能和平台

AndroidiOSC++PythonJSCoral
人脸检测
人脸网格
虹膜
手部
姿势
整体
自拍分割
头发分割
物体检测
框追踪
即时运动追踪
Objectron
KNIFT
自动翻转
媒体序列
YouTube 8M

MediaPipe 测试#

我这里调用姿势的 JS API 来在网页中进行测试
通过 React-Webcam 库来从相机获取视频,再通过 canvas 绘制识别结果

import Webcam from "react-webcam";
import React, { useRef, useEffect, useState } from "react";
import { drawConnectors, drawLandmarks } from "@mediapipe/drawing_utils";
import { Camera } from "@mediapipe/camera_utils";

import { Pose, POSE_CONNECTIONS, POSE_LANDMARKS } from "@mediapipe/pose/pose";

const MPHolistic = () => {
  const webcamRef = useRef(null);
  const canvasRef = useRef(null);

  useEffect(() => {
    const pose = new Pose({
      locateFile: (file) => {
        return `pose/${file}`;
      },
    });
    pose.setOptions({
      modelComplexity: 1,
      smoothLandmarks: true,
      enableSegmentation: true,
      smoothSegmentation: true,
      minDetectionConfidence: 0.5,
      minTrackingConfidence: 0.5,
    });

    pose.onResults(onResults);

    if (
      typeof webcamRef.current !== "undefined" &&
      webcamRef.current !== null
    ) {
      const camera = new Camera(webcamRef.current.video, {
        onFrame: async () => {
          await pose.send({ image: webcamRef.current.video });
          // await holistic.send({ image: webcamRef.current.video })
        },
        width: 1280,
        height: 720,
      });
      camera.start();
    }

  }, []);

  const onResults = async (results) => {
    const videoWidth = webcamRef.current.video.videoWidth;
    const videoHeight = webcamRef.current.video.videoHeight;
    canvasRef.current.width = 1280;
    canvasRef.current.height = 720;

    const canvasElement = canvasRef.current;
    const canvasCtx = canvasElement.getContext("2d");

    canvasCtx.save();
    canvasCtx.clearRect(0, 0, videoWidth, videoHeight);
    canvasCtx.translate(videoWidth, 0)
    canvasCtx.scale(-1, 1)
    canvasCtx.drawImage(
      results.image,
      0,
      0,
      canvasElement.width,
      canvasElement.height
    );


    drawConnectors(canvasCtx, results.poseLandmarks, POSE_CONNECTIONS, { color: "#00FF00", lineWidth: 4 })
    drawLandmarks(canvasCtx, results.poseLandmarks, { color: "#FF0000", lineWidth: 2 })

    canvasCtx.restore();
  };

  const videoConstraints = {
    width: 1280,
    height: 720,
    facingMode: "user",
  };

  return (
    <>
      <div
        style={{
          position: "relative",
          width: "100%",
          height: "100%",
        }}
      >
        <Webcam
          audio={false}
          mirrored={true}
          ref={webcamRef}
          style={{
            position: "absolute",
            marginLeft: "auto",
            marginRight: "auto",
            left: 0,
            right: 0,
            textAlign: "center",
            zindex: 9,
            width: 1280,
            height: 720,
          }}
          videoConstraints={videoConstraints}
        />
        <canvas
          ref={canvasRef}
          style={{
            position: "absolute",
            marginLeft: "auto",
            marginRight: "auto",
            left: 0,
            right: 0,
            textAlign: "center",
            zindex: 9,
            width: 1280,
            height: 720,
          }}
        ></canvas>
      </div>
    </>
  );
};

export default MPHolistic;

效果#

一个测试截图如下

image

我这里也在本机的网页中测试了一下帧率,大概可以稳定 fps 在 100 左右,识别效果也完全可以接受~

加载中...
此文章数据所有权由区块链加密技术和智能合约保障仅归创作者所有。