Using Synthesia with analog sounds!

Have questions? Just saying hello? This is the place.
No explicit, hateful, or hurtful language. Nothing illegal.
Post Reply
bluesdance
Posts: 13

Post by bluesdance » 05-07-15 7:42 am

I am an elementary school music teacher. Synthesia is very successful for teaching piano to my students but I wanted to go beyond that. I figured out a solution involving the digital audio workstation Reaper.


Nicholas
Posts: 12343

Post by Nicholas » 05-07-15 10:55 am

That is very cool! Coincidentally, I was just brainstorming about pitch-detection a couple nights ago. Someone showed me a similar app for the iPad made specifically for guitar. It does the same thing where it just pretends to be a MIDI device.

While it is geared very tightly for guitar (which opens a few weaknesses for piano), it was kind of magical just walking over to the piano, placing the iPad on the music stand, playing some notes, and watching them show up on the screen. It felt like the future.

The sensitivity (also shown in your video) is definitely a rough edge that would have to be sanded off. One solution that I want to try would be a one-time calibration step. You'd have to play all of your keyboard's notes instead of just the lowest/highest, but after that Synthesia would know exactly what it was looking for. Because piano has a very consistent pitch for each key (vs. something bendy like a guitar) I think this would work well. (Accuracy would come at the price of being able to use arbitrary instruments like your voice, and the calibration step is decidedly about 3 minutes of the opposite of fun.)

Anyway, thanks for sharing and taking the time to put together that video. Very cool!

bluesdance
Posts: 13

Post by bluesdance » 05-08-15 9:35 am

Thanks!!!

I agree that pitch detection in Synthesia would be wonderful, because when my young students ask me if they can get your app, I say yeah but you also need a cable and an electric keyboard and if you have a keyboard it might be this type of cable and it might be that type of cable, and with an iPad you need another dongle too and at that point I know there's no way that it's going to happen... you would have to be able to do multiphonic pitch detection, of course, but I know that it's possible because this thing exists: http://www.tcelectronic.com/polytune-2/

The open source dance game Stepmania, which I think of as a cousin to your app, has an option hiding deep inside an INI file to adjust the timing sensitivity. For education purposes I adjusted the timing sensitivity to be as forgiving as possible. It would be great to be able to adjust Stepmania so it would be more likely to say "Great!" vs "Barely", or "Barely" vs. no-score for a late tone.
Last edited by bluesdance on 05-08-15 9:38 am, edited 1 time in total.

bluesdance
Posts: 13

Post by bluesdance » 05-08-15 9:37 am

By the way while I'm on the subject... I think the name of your app is TERRIBLE at communicating what your app is and does. Something like "Piano Genius" or "Piano Wizard" or "Piano Star" or whatever would be MUCH better, at least for the type of people (kids and parents) I talk to. In my classroom we just call it "The Piano Game"

bluesdance
Posts: 13

Post by bluesdance » 05-08-15 9:39 am

One more vid


Nicholas
Posts: 12343

Post by Nicholas » 05-09-15 12:46 am

bluesdance wrote:.... you would have to be able to do multiphonic pitch detection, of course...
Yeah, polyphonic detection is the hard part. There is a trail of PhD theses that have been done on the subject. Requiring the (inconvenient) calibration step is how I'm hoping to bring the challenge down to an actually doable level.

In a pinch, Synthesia can cheat even more: because you're in the context of a song, it knows which notes you're supposed to be playing, so it can just anticipate how that frequency spectrum will look, and then wait until it sees it. This is how that guitar tuner works. Because it knows your "destination" for each string, all it has to do is characterize how far away you are from it. (Again, the general-purpose "which set of notes am I playing right now on this arbitrary instrument?" problem is super-hard.)
bluesdance wrote:... to adjust the timing sensitivity.
Fire up the Synthesia Configuration utility and change the "Gameplay.NoteWindowUs" advanced setting to a larger number. Doubling it should double the window of time you have to hit notes (before or after they've passed).
bluesdance wrote:I think the name of your app is TERRIBLE at communicating what your app is and does.
I agree. There is a lot of confusion about pronunciation, too. (You said it the same as I do in your first video.)

It's main strength is the neat play on "synesthesia", the medical condition where your brain can form associations between different kinds of stimuli; say between musical pitches and color! But, your point still stands 100%. "Clever" doesn't always mean "good".

Although, at this point... I'd be loathe to change it. There is already a lot of brand awareness out on the Internet. Whenever I see a new competitor emerge, forum comments are usually along the lines of "isn't this just the same as Synthesia?" Changing the name would effectively pit us against ourselves.

Maybe a descriptive subtitle would help. Historically I've enjoyed "Piano for Everyone". It might be time to put that sound-byte back on the front page.
bluesdance wrote:... "Piano Genius" or "Piano Wizard" or "Piano Star"...
For what it's worth, each of those (well, only sort-of for "Piano Genius") was recommended during our name contest after Activision told us we couldn't use the name "Piano Hero" anymore. (The same contest where I picked Synthesia as the winning entry. Although Oktav and Claviro were runner-ups.)

Worse than all of those being suggested, Piano Wizard was already a competing product, predating Synthesia by several years! :lol:

Dave12
Posts: 9

Post by Dave12 » 05-11-15 6:21 pm

Maybe merging Transcribe (http://www.seventhstring.com) and Synthesia together?

Nicholas
Posts: 12343

Post by Nicholas » 05-11-15 6:56 pm

Yeah, the pitch-listening feature will be using the same technology that Transcribe uses (analyzing the energy in the FFT of an audio stream). Although, doing it in real-time with effectively zero latency is another challenging part of the problem.

bluesdance
Posts: 13

Post by bluesdance » 05-12-15 4:49 pm

I did a free play singing thing with my students... I also put an octave chart to go below the keys:

Image

bluesdance
Posts: 13

Post by bluesdance » 05-12-15 4:51 pm

Nicholas wrote:For what it's worth, each of those (well, only sort-of for "Piano Genius") was recommended during our name contest after Activision told us we couldn't use the name "Piano Hero" anymore.
Piano Hombre, Cap'n Piano, Pianola, Go Go Gadget Piano, Piano Party, Piano-Playing Platypus...

User avatar
AndImStillAlive
Posts: 30
Location: US

Post by AndImStillAlive » 06-11-15 2:03 pm

This is cool! I would love to be able to use Synthesia with my melodica. :)
AndImStillAlive

User avatar
jimhenry
Posts: 1800
Location: Illinois

Post by jimhenry » 06-11-15 3:39 pm

How does the key size of the melodica compare to the Korg nanoKEY?
http://www.korg.com/us/products/controllers/nanokey2/

Could you use the nanoKEY to practice keyboard technique for melodica?
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/

User avatar
AndImStillAlive
Posts: 30
Location: US

Post by AndImStillAlive » 06-11-15 4:49 pm

I've never seen those before. They are really cool! Hmmm... might have to think about getting one...
I just got the melodica, so for practice, I have just been going back and forth between my digital piano and the melodica, but the nanoKey does look like it would be a more similar experience. I have this one:

http://www.amazon.com/Hohner-32B-Piano- ... elodica+32

So, not exactly the same thing, but I really think the nanoKey would be cool anyhow!
Now, if I only had a decent melodica soundfont...

Thanks for the link!
AndImStillAlive

User avatar
AndImStillAlive
Posts: 30
Location: US

Post by AndImStillAlive » 06-12-15 2:34 pm

It looks like the nanoKey would be great for entering music into Musescore!
I'm really thinking about getting one. Do you know if it will work directly with Synthesia?

Thanks,

Teresa
AndImStillAlive

Nicholas
Posts: 12343

Post by Nicholas » 06-13-15 2:55 am

AndImStillAlive wrote:Do you know if it will work directly with Synthesia?
It should. I have an older Korg nanoKontrol (the first one) and it detects as a vanilla MIDI device with no driver install required. The probability that the nanoKeys2 will work right out of the box is like 99%.

User avatar
AndImStillAlive
Posts: 30
Location: US

Post by AndImStillAlive » 06-15-15 3:18 pm

Awesome!

Thank you Nicholas :)
AndImStillAlive

User avatar
AndImStillAlive
Posts: 30
Location: US

Post by AndImStillAlive » 10-28-15 4:35 pm

I decided to go with the AKAI LPK25 MIDI Controller.
It is almost the same size and shape as my melodica (minus 7 keys) and, other than the limited range, it works great with Synthesia, in case anyone was wondering... I know I was.
It is also excellent for entering music into Musescore. :)
AndImStillAlive

Post Reply