EffecTV: An entertainment application of real-time video image processing

EffecTV is a real-time software video effect processor based on motion detection and image processing techniques. It enables to add a visual effect to an input video image from a video camera in real-time. The visual effect reflects the motion of the input image so that participants in front of the camera see themselves with various visuals effect - burning, lightning, scratching or being invisible - along their body motion. Our real-time visual effect processing techniques are used for attractions of musical event or a beer bar, background visual for live performances, and special effect for some theatrical plays.

The 1st version of EffecTV was released in 2001 as an open source software, and the number of features and effects increased through contributions by people in the open source community. Please visit www.effectv.dev for its manual and source code. My personal works based on EffecTV are published in my works page.

EffecTV

EffecTV bases on optimization techniques of real-time graphics processing, developed in "demo scene", and motion recognition techniques (and some little hack). EffecTV achieves 1) real-time and 2) amazing and exciting visual effect by using them. This kind of system has the following advantages.

Interactivity
Participants can learn and enjoy by seeing themselves with visual effects reflects their motion in real-time.
Easy synchronization
It is easy to synchronize the visuals to the motion of the actors or the performers.

Existing visual effects on a video cameras or a video editing system are too simple such as color distortion or mirror effect. Non-linear video editing systems provide various quality visual effects, but it is hard to check the result soon because they require a time to render and record the visual. Those latency discourages the user's trial-and-error.

Basic System Configuration

We introduce two basic system configurations of EffecTV. In the left figure, participants stand in front of the camera and sees themselves on the display with visual effect along their body motion. We used this setting for "interactive-toy" style installation on streets or clubs. In the right figure, the performers are on the stage and the camera follows their motion. Visuals are displayed on the screen behind the stage, so the audience sees both the performers and the visual at once. This setting is used for various stage performances.

System overview: Toy-style

System overview: Stage-style

Application

EffecTV is used for a wide variety of applications for its high interactivity. Here are sample use-cases.

EffecTV provides a perfect synchronization between performers and visual effects. This advantage was appreciated for stage performances feature improvised actions. On a live music performance, EffecTV makes visuals synchronized to the music.

Samples

Publications