Auxiliary display?

[bjp]

I'm interested in making an auxiliary display for Synthesia and I was wondering how feasible this would be, with regard to interfacing with Synthesia.

What I'm planning to do, for context:
I'm going to put a big piece of white paper just above the keys of my keyboard and point one of those cheap LED video projectors at it. Then, I'll use a webcam pointed at both the keys of the keyboard and the projection screen to automatically 1) identify the keys of the keyboard and 2) observe some test patterns with the video projector to figure out what physical space relative to the keys each video projector pixel lights up. With this knowledge, I can now light up any area above (or on) any desired key of the keyboard. With that knowledge, I'd then like to essentially duplicate Synthesia's note roll above my real-life keyboard so that the roll lines up perfectly with each key.

I could perhaps approximate this effect by having the projector display the Synthesia screen as-is and placing the projector in exactly the right spot so that the projected keys line up with the real keys. But, it's very difficult (and sometimes impossible) to get this alignment just right, so the thing I want to make would essentially be an automatic distortion calibration tool that would make projector placement very easy/tolerant.

What I need from Synthesia:
In my ideal world, Synthesia would transmit the full song contents as a song was loaded/started (all the notes, their start timestamps, their end timestamps, and any hand/finger labels that might exist), perhaps via UDP broadcast, and then continuously transmit the current timecode as the user played the song. I imagine that's not currently available, so I was wondering if there was anything similar available that might meet or partially meet my needs. Is there any general SDK documentation available? A quick glance at some of the posts on this forum seems to suggest that there is a substantial amount of functionality that I'm unaware of documentation for.

[Nicholas]

The closest thing we've got to that volume of extra information is through the "Channel-based key lights" MIDI stream that communicates notes coming up in the next second or so. (I.e., not much!)

all the notes, their start timestamps, their end timestamps, [...] then continuously transmit the current timecode...

It sounds like you're looking for a MIDI loading/playing library.

I'm not sure the distortion problem is as bad as it sounds like you're worried it's going to be. In the past, I've had some success with just a large'ish touch-screen monitor hanging over the keys with a monitor arm.

Using the "Custom" zoom mode in Synthesia, it remembers your keyboard scale down to the pixel level (instead of snapping to whole white keys) to help with lining things up with a nearby display.

Beyond that, we can't add any more projector-specific features because someone already patented "Synthesia plus a projector" back in 2009. (They even went so far as to include a screenshot from Synthesia in their supplemental images that help describe how the "invention" works.

[bjp]

It sounds like you're looking for a MIDI loading/playing library.

Well yes, but the key thing would be being able to accept Synthesia's real-time timecode updates, plus having Synthesia say which file was loaded. I don't remember how things are encoded in MIDI exactly, but perhaps just outputs of "MIDI file XXX was loaded" and "now at time YYY in file" would be all that is necessary.

Man, that patent is very on-point and therefore super unfortunate So I guess never mind. Do you know if that product is available/in development for anything, or is the owner patent trolling?

[jimhenry]

Beyond that, we can't add any more projector-specific features because someone already patented "Synthesia plus a projector" back in 2009. (They even went so far as to include a screenshot from Synthesia in their supplemental images that help describe how the "invention" works.

Well have at it! The patent expired on 05/22/2017 and you are now free to use any and all ideas disclosed therein.

[Nicholas]

Oh, hey, you're right. Neat. I guess it was just (IP) ambulance chasing after all.

This is the same one that originally used the Synthesia screenshot right on their cover page in an earlier rejection. Recall: the last name of the inventor was shared with one of the partners at the otherwise-BioMed IP attorney place he used to file it. That, coupled with all of the bumbling evident from the PAIR log has always smelled like a family member calling in a favor to help him patent something. It looks like it was on -- and stayed on -- both of their back-burners this whole time.

Well good then. My general distaste for patents (and the system as a whole) couldn't reverse harder when one expires and instead of an ever-present looming shadow, it's essentially a guarantee of safety (so long as you stick to closely enough to the implementation they described in the first place).

[bjp]

Hey, that's awesome! Well ok then, project back on

In the mean time, I tried lining up my projector with the plain Synthesia display and it sort of worked ok. So, maybe instead of going full pixel-by-pixel calibration (a previous project I did professionally was a structured light 3D scanner using two cheap webcams so that's how I was thinking about it), I could maybe achieve 90% of the utility by doing a simpler transform on the graphics to be displayed. I'm not familiar with the timeline -- how far diverged is Synthesia's current codebase from, for instance, linthesia, or some other codebase I can have access to? Or, more directly: if I mod KeyboardDisplay to do what I want, would you be able to incorporate those changes into the current closed-source version of Synthesia, or would that be a lot of work because the codebases have diverged so much? If I write the mod (no promises; I'm pretty busy), you would be welcome to incorporate it free and royalty-free -- I just want the feature.

[Nicholas]

... how far diverged is Synthesia's current codebase from, for instance, linthesia...

10 years and 4000 commits.

If I squint hard enough, I can almost see one or two lines of code in a file here-and-there that looks like the old, open-source version.

You may have better luck (and this would be more useful in the general case for lots of things) making something that used Windows graphics hooks (the way screen-recording apps like Fraps do things) to grab any app's rendering surface and perform a transform on it.

Then you don't just get Synthesia for projectors... you get everything for projectors. (That actually sounds like a cool project.)

[bjp]

You may have better luck (and this would be more useful in the general case for lots of things) making something that used Windows graphics hooks (the way screen-recording apps like Fraps do things) to grab any app's rendering surface and perform a transform on it.

Yeah, that does sound cool, but substantially harder (I don't know how I would go about stream-capturing the screen in near real-time) :/ So that I can make sure I'm not researching a dead end, how does Synthesia draw to the screen? OpenGL? Something else?

Are you at all open to the idea of adding the feature where 1) upon start of the song (when keyboard is first shown), MIDI (or whatever format) data for the whole song is broadcast over UDP and 2) the time offset is continuously (~30 fps) broadcast over UDP while the keyboard is showing? This is how X-Plane (http://www.nuclearprojects.com/xplane/info.shtml) makes it easy for people to use multiple computers, interact with it as a simulator, etc. If you agree in principle, I would of course complete all my work first with mocks to make sure your time would be worth it. I would also be open to paying for the development of this feature, though of course it would depend on how much.

[bjp]

I'm still very interested in this feature (being able to see the approaching notes aligned with my physical keyboard using a projector), but I tried just throwing up a projector on a stand and manually aligning the on-screen keyboard with the real keys and that actually works surprisingly well. So, maybe I'll just try that out for a while before trying something more complicated.

[Nicholas]

That's a cool setup!

If you had entered this into an innovation competition in Finland back in 2013, (even though it had already been demonstrated for years), you could have earned 3rd place and received $6000.

Maybe "Synthesia + projector" was an invention worthy of patenting after all.

[bjp]

Haha, it seems as if this idea has been independently discovered by a very large number of people

My current setup is really just the quickest thing I could throw together for proof of concept -- I'll making something better now that I know it works.

I do have one major feature request for this even nearly as-is: the "Custom" keyboard in the view selection seems to be per-song rather than global. I have to readjust my projector pretty much every time I sit down, which then means I have to realign the keyboard on every song I want to play. That's lame; I would like to have a custom keyboard view that is global across all songs. I would have expected that to be the "Custom" option given that there is also a "Song Only" option. What does the "Song Only" option uniquely do, given that "Custom" seems to be per-song as well?

It would also be nice to have keyboard shortcuts to "move keyboard left", "move keyboard right", "zoom in", and "zoom out". They would start auto-incrementing upon KeyDown and stop upon KeyUp (as opposed to PgUp and PgDown with regard to the notes which increment once per KeyPress). But, those keyboard shortcuts aren't too important as long as there is a keyboard/screen layout view that applies to all songs.

[Nicholas]

Hrm... the "Custom" setting should be global. Your use case was exactly why it was added, so global is the only way it makes sense. I'll take a look to see if something has gone amiss.

("Song Only" will automatically zoom into the extents of the visible song notes, for each song. So, songs with a smaller range -- like beginner pieces -- will be zoomed in much farther than something that uses most of the octaves of the keyboard.)

Regarding zoom, the best interface is pinching on a touchscreen (in the falling note area). Even after six years, I'm still really happy with how natural that feels. It's only very recently that the little plus and minus icons were added to the zoom menu to manually adjust it on non-touchscreen devices. It'd be easy to add a shortcut for it. (The hardest part is that it's two new text entries to have translated into eleven languages.)