microphone pitch listeneing for non MIDI keyboards!

Synthesia is a living project. You can help by sharing your ideas.
Search the forum before posting your idea. :D

No explicit, hateful, or hurtful language. Nothing illegal.
Soulfire
Posts: 1

Post by Soulfire »

It'd be awesome to have the possibility to control the game without a MIDI keyboard.

I have a keyboard that doesn't have MIDI support. However, with a microphone, it'd be possible to listen to the pitch of the sound of what it plays and actually discern the note, so I was thinking it'd be awesome if you could control the keyboard in the game from a standard keyboard from a microphone.

What do you think?
Nicholas
Posts: 13135

Post by Nicholas »

So, pitch detection is pretty straight-forward when you're only playing one note at a time, but it seems like it would become a lot more complex when you had to play more than one (e.g. any song that requires both hands, or even single-hand chords). Then, if you take into account the "attack" (the harder sound when you first hit the note), "falloff" and "sustain" (how the note fades out after you release the note), and the various losses and background noise between the keyboard and the mic, detecting lots of simultaneous/overlapping notes quickly and accurately seems like it'd be really difficult.

I'm not sure how widespread the problem is. Are there many keyboards without MIDI or USB connections? It may be worth looking into if that's the case. But, it wouldn't be for a while. There is still quite a bit of very high priority work to be done.
maccer
Posts: 222

Post by maccer »

Nicholas wrote:Are there many keyboards without MIDI or USB connections? It may be worth looking into if that's the case. But, it wouldn't be for a while. There is still quite a bit of very high priority work to be done.
I think the point was that you could then use a real piano that doesn't have any MIDI ports. At least that's what I would use it for.

But I can agree that it is probably not so easy to recognise multiple simultaneous key presses... or maybe it is, can't you just make a fourier analysis of the incoming signal in order to get the frequencies? By looking at the "amplitude" of the fourier graph it should be possible to distinguish the actual notes played from any overtones. I don't know, however, how much CPU performance would be needed to do this in real time.

But for simple songs where you play a track with only one note playing at a time pitch recognition shouldn't be too hard to do. For these trachs you could also use your voice just like in Sing Star (which I have actually never played)! :-)
Songs learned using Synthesia:
CT: Wind Scene, The Trial | FF7: Prelude | SMB: Overworld, Underwater | Tetris: Theme A | Zelda: Lost Woods | Other: Für Elise
Nicholas
Posts: 13135

Post by Nicholas »

It might be worth a shot. MIDI input from a real piano is an interesting idea. Also, the FFT and associated analysis wouldn't take much (relative) CPU time at all. There are some incredibly optimized libraries available for that kind of work these days.

I was worried more about the physical interaction of the notes (e.g. overtones) simply causing ambiguous input. But, maybe it wouldn't be so bad. Still, it doesn't seem like the recording/analysis "input device" could be a simple drop-in replacement. It would always have to have a little extra special-purpose code that say, wouldn't break your combo if some combination of notes made it difficulty to tell which octave you were playing in. It would have to drop the game into a more forgiving mode.
Michael
Synthesia Donor
Posts: 13

Post by Michael »

Yeah, there are plenty of algorithms around for detecting the pitch of single note, however, there is no way to analyse for chords and such (unless we are talking pure sinewaves here ;)).

Anyway, Soulfire, here is what you could try:
Get a program called REAPER (http://www.reaper.fm ; it's got a 30 days trail period) and use its internal ReaTune VST fx to turn the audio stream into MIDI data. Now use the app called MIDI Yoke (http://www.midiox.com/myoke.htm; a virtual MIDI driver) to route the MIDI stream from REAPER to Synthesia.

Though there are still problems, due to the fact that the accuracy of every pitch detection algorithm is proportional to the window size it uses and therefore the delay it causes. So you need to trade off between latency and how accurate the notes get detected. Also another crucial factor is the detection of low notes, since the lower you want to detect the bigger a window size you need also, which also again equals more latency, which again needs to be traded off.
I mean for voice its easy, detect you'd just detect the fundamentals (which are somewhere above 80Hz and under 1000Hz), but a piano has a much bigger ranger to be "look" at.

Anyway the REAPER -> MIDI Yoke -> Synthesia workaround might work pretty good actually, though I haven't tried it.
Nicholas
Posts: 13135

Post by Nicholas »

That does sound like it might be worth trying. If anybody gets a chance -- successful or not -- please mention it here.

In addition to MIDI Yoke (maybe if it gives you trouble?), there is another free MIDI loopback driver I've used in the past called LoopBe 1. (The latest version is for-pay, but 1 works just fine.)
Jande22
Posts: 8

Post by Jande22 »

im not sure if its the same thing but theres a game called guitar rising that uses the line-in or mic port on a pc to play the game using a real guitar.
shouldn't the same be possible with the headphone port on the keyboard and the mic in on the pc using an aux-aux chord.
Michael
Synthesia Donor
Posts: 13

Post by Michael »

Jande22 wrote:im not sure if its the same thing but theres a game called guitar rising that uses the line-in or mic port on a pc to play the game using a real guitar.
shouldn't the same be possible with the headphone port on the keyboard and the mic in on the pc using an aux-aux chord.
Yes and also no, from there product teasers, it seems as if it can only recognise single notes, but no chords (so I got reasonable doubt it works for chords, too).

Though there are MIDI pickups for guitars that get mnounted right after the bridge and measure each string on its own and convert the notes into MIDI data. So I think that this game works with something like that, and the whole plug the jack into your guitar and go of the video is just a marketing scam (especially since it only states in the video "USB Audio Device" and a generic MIDI device also gets listed as such).

Hmm.. that makes me think, I should get a guitar midi pickup and see how it works with Synthesia :P.
Jande22
Posts: 8

Post by Jande22 »

michael if you can tell me how to do the retune vst fx in reaper i'll more than happily play around with it. I have never used reaper before so I'm a litle list in it
kinnik
Posts: 13

Post by kinnik »

I tried using afew pitch detection VSTs to convert singing into midi but they were all far too slow to be usable with Synthesia. I tried "widi", "midifier","midikonv" and "the extractor". The Extractor was the quickest and most accurate from my tries but it was still to slow to use with the game... But it was very fun trying to sing in prefect pitch when the slightest bit off would detect as a note lower or higher. :)
Nicholas
Posts: 13135

Post by Nicholas »

This just in. Apparently there are ways to pick out the notes in a (live, recorded) chord.

That's an awesome video, too.

Too bad it sounds super-hard and proprietary. The "What doesn't work in theory can still work in reality..." quote is pretty good.
Jande22
Posts: 8

Post by Jande22 »

I'd be happy getting simple pitch detection for single notes working so i can somewhat play this game lol..
Michael
Synthesia Donor
Posts: 13

Post by Michael »

michael if you can tell me how to do the retune vst fx in reaper i'll more than happily play around with it. I have never used reaper before so I'm a litle list in it
Well its a bit complex.
* Get REAPER, install it.
* Open it.
* Hit CTRL+T (to create a new track).
* Then on that track click on the little 'R' (to record arm that track)
* Press the FX button of that track, FX chain view
* Press "ADD" (buttom left), FX browser pops up
* Search for the 'ReaTune' FX in there, and add it
* In the ReaTune window select 'send MIDI events when pitch changes' (edit: and set the 'Overlap' to '8x' to get a better pitch detection, also fudge with the window size once you got it all setup to see what works best)
* Click on the 'I/O' of the track disable 'Master/Parent Send' and as 'MIDI Hardware output' select the MIDI Joke midi device (or you could try the 'MS SW Wavetable Synth' first to see if it works, might need to enable it in the prefs (Preferences->Audio->Midi Devices/Midi outputs to make available (or something) first though)
* Then open Synthesia and select the MIDI joke driver as input and it should work

It is highly adviced to use a ASIO sound driver for low latency.
Michael
Synthesia Donor
Posts: 13

Post by Michael »

Nicholas wrote:This just in. Apparently there are ways to pick out the notes in a (live, recorded) chord.

That's an awesome video, too.

Too bad it sounds super-hard and proprietary. The "What doesn't work in theory can still work in reality..." quote is pretty good.
Yeah that's pretty nuts, though I think it will only contribute the increase of untalented artist going into studios and totally screw up there performance and then ask the engineer "you can fix that, right?". I mean, where have we come? No artistic nor musical talent, but engineers fixing it all and make it shine :(. Welcome to the new music industry.

EDIT: I also fear that many people will take Neubäcker too literal and expect the engineer to tune their guitar AFTER recording IN THE MIX!
Nicholas
Posts: 13135

Post by Nicholas »

Michael wrote:I also fear that many people will take Neubäcker too literal and expect the engineer to tune their guitar AFTER recording IN THE MIX!
Yeah, that also struck me really strangely. I couldn't imagine asking someone to do that. Not only don't they have to be talented, they don't even have to take the time to set up their instruments. It really does just shift the work (and skill requirement) to the engineer.

Still, it sounded really good in that demo video. ;)
aria1121
Posts: 1505

Post by aria1121 »

Never mind, I just changed something but my piano is 20 cents off pitch so if I play [A4] I get [Ab4] on my laptop. Changing Hertz Reference in the FX screen only changes the note name, not the sound.
aria1121
Posts: 1505

Post by aria1121 »

(Content moved to new post)
Last edited by aria1121 on 09-11-11 4:22 pm, edited 1 time in total.
User avatar
DC64
Posts: 830

Post by DC64 »

This should be in the FAQ.
"And now for something completely different."
aria1121
Posts: 1505

Post by aria1121 »

BTW, I am going to make a new topic for this so it will be a bit easier to follow.
Post Reply