Hey guys,
I'm bringing back this tread with some testing of my own.
Lately I've been doing experiments with augmented reality, so I guess I was ready to try this thing too.
Before thinking about how you're going to include that feature inside Synthesia, I think it's better to do some practical testing to see if that's technically possible or not.
One important thing that I could notice with augmented reality is the video latency. That's probably the biggest thing you have to worry about, as a video feed got a lot more to carry than an audio one, and I'm afraid there won't be magical Asio-like drivers to save the day this time.
So here is what I get for the moment with Synthesia, VLC, and a HDV camera.
1.Hardware setup
My computer is adjusted as a quite compact layout inside what used to be a wardrobe (^^) so I got plenty of space at the top of it to place a camera.
I'm using a monopod placed horizontally, so I can have some control over the angle and stuff.
At least I'm sure the whole thing won't fall on my head while I'm playing :p
My compact setup (and a crappy cellphone pic...)
2.Software setup
No need for fancy additional software, VLC can handle.
Go to Media > Open capture device

You can find you DV or web camera there.
I reducing the caching a bit to get a lil less delay.
Select everything inside the "Edit options" box and copy it.
You can close VLC.
Open Synthesia config tool

You have to run it in a window to be able to see VLC on top of that.
For some reason OpenGL works a lot better that DirectX in windowed mode for me.
3.Script setup
Open notepad and paste the following VLC command line:
Code: Select all
"C:\Program Files\VideoLAN\VLC\vlc.exe" dshow:// :dshow-vdev="Microsoft DV Camera and VCR" :dshow-adev=none :dshow-caching=100 --no-video-deco --no-embedded-video --video-on-top --deinterlace=1 --video-x=1 --video-y=580 --width=1680 --height=336 --crop=5:1 ---video-filter=rotate{angle=180}
You need to replace the content with your own settings:
Path to VLC | the camera options you copied earlier from VLC | stuff to make VLC borderless | video size and placement | rotation because my camera is upside down
(The crop ratio should be the same as width/height ->1680/336=5/1)
Save this as VLCsynthesia.bat or whatever you like.
You can convert that into an .exe later if you like with
that.
Run the script.
Launch Synthesia.
Play.
Synthesia window with VLC live frame on top of it
Random facts & limitations:
-I could use a better camera placement/alignment
Hard to go farther though with a short firewire cable
-Need to alt/tab at the beginning of each song to bring back the live frame
Also you can't control synthesia while VLC is on focus
-Delay issues for the video feed
I'm getting almost half a second of delay, which kills part of the concept
You may have a good preview of your hands positions with slow songs, but 0,5sec is for example
way too long to check your placement when switching between fast notes
I also had to switch my HDV camera to DV mode to get a faster response
Now I'll be looking for a way to improve that delay. Hopefully some of you may also give that setup a try.
I used to have a crappy low def webcam a few years ago, but apparently they got some nice HD cheap ones nowadays,
maybe those got a better support and a shorter delay than firewire systems.