Drawing onto Frame Processors

Drawing onto Frame Processors

It is technically pretty difficult to draw onto a Camera Frame and have it rendered into the resulting photo or video in realtime, but I have built a working proof of concept of that using Metal/OpenGL and Skia straight in VisionCamera.

This allows you to draw onto a Frame using simple JavaScript APIs:

const frameProcessor = useSkiaFrameProcessor((frame) => {
  'worklet'

  // Detect faces using GPU-accelerated ML API
  const faces = detectFaces()

  // Draw faces using GPU-accelerated Skia drawing API
  for (const face of faces) {
    const rect = Skia.XYWHRect(face.x, face.y, face.width, face.height)
    const paint = Skia.Paint()
    paint.setColor(Skia.Color('red'))
    frame.drawRect(rect, paint)
  }
}, [])

..or even apply color-filters in realtime:

const invertColorsFilter = Skia.RuntimeEffect.Make(`
  uniform shader image;
  half4 main(vec2 pos) {
    vec4 color = image.eval(pos);
    return vec4((1.0 - color).rgb, 1.0);
  }
`)
const paint = Skia.Paint(invertColorsFilter)

const frameProcessor = useSkiaFrameProcessor((frame) => {
  'worklet'

  // Draw frame using Skia Shader to invert colors
  frame.render(paint)
}, [])

While this API was originally planned for V3, I decided to remove it again because VisionCamera got insanely complex with that code integrated (custom Metal/OpenGL rendering pipeline with intermediate steps and tons of build setup) and I wanted to keep VisionCamera lean so other users aren't affected by build issues caused by Skia or those GPU APIs. See this PR for more information.

In my agency, Margelo, we have worked a lot with 2D and 3D graphics and Camera realtime processing (see the Snapchat-style mask filter on our website for example - that is running in VisionCamera/React Native!), if you need this feature reach out to us and we'll build a customized/tailored solution for your company! :)

🚀 Next section: Zooming (or creating a Frame Processor Plugin)