Can Synthesia support VGM files?

tommai78101 · Post by **tommai78101** » 08-10-07 9:21 am

Can Synthesia support Sega Genesis/Master System music?

Here's their website: http://www.project2612.org/list.php

They said Winamp can decode the files to either WAV or MP3. I figure it is in "MIDI-likeness" codes or scripts. I tested it and they were fine.

Post by **Nicholas** » 08-11-07 11:18 pm

So, a while back someone suggested the same for NSF and SPC files (which I believe are the same as what you're talking about for the NES and SNES).

I looked into it a bit, and it looked like they were fundamentally different from MIDI. In particular, there is no way to get a list of "notes" out of the files. If I understand correctly, those files are actually snippets of CPU assembly instructions for whatever processor those old consoles used to use. They're instructions for turning on and off the built-in sound channels at particular frequencies. Because it's machine code, they have full control over frequency, note blending, and a bunch of other things that can't be easily picked out. If you search around, you can find a few programs that try to extract MIDI music from those types of files but they don't usually do a good job. It's a recognition problem like voice recognition.

If a really good engine existed to do an auto-conversion, I would use it. But, for now, it looks like that I might be going too far outside the scope of my project to add support for those kinds of video game music files.

tommai78101 · Post by **tommai78101** » 08-18-07 12:32 am

Oh, so those machines way back in time has a sound controller device inside the core.

I thought it was in an old type MIDI format. Must be me thinking too much.

Thank you.

mntorankusu · Post by **mntorankusu** » 09-12-07 10:40 pm

Hm. Is it really that hard? I originally suggested playing NSF files, and thought it would be pretty simple, because of this Winamp plugin. It accurately shows what notes are currently being played on each channel on an on-screen keyboard. It helped me learn to play a lot of NES music.

Post by **Nicholas** » 09-12-07 11:43 pm

mntorankusu wrote:It accurately shows what notes are currently being played on each channel on an on-screen keyboard.

That is pretty accurate. It looks like they just band the frequencies to the nearest note. (In say, the title music for Legend of Zelda, there is actually some vibrato on the square waves, but the keyboard display just shows a single note. Using the same method (one note that utilizes pitch bends rather than dozens of tiny notes) the music could be playable and still maintain a lot of the same nuances.

Still, to be completely accurate, Synthesia would have to output PCM wave signals instead of MIDI. I could try and find the closest GM MIDI instruments to the ones used in the NES, but it would never be perfect.

So I guess the new answer to the question is: it can be done (at least for NES files, due to the availability of the file formats -- and especially the source code for that plugin), in two ways. First, parse out the NSF and generate more conceptual "notes" (with pitch bends and volume control behind the scenes much like in the current MIDI parser) in the four available tracks, play them as the most-similar MIDI instrument. The second method extends the first by actually building a PCM renderer to generate the NES-accurate sounds instead of using MIDI instruments.

Now at the same time, either of those would take a great deal of time (several months of my sporadic free time) and rob from stuff that really needs to be completed sooner.

(But, man! That would be cool. It's certainly on my list again, but it is unfortunately really low priority.)

Thank you very much for the link. At least I know it's possible now. It would be really nice to get around to doing it some day. I'm sure similar methods are available for the other consoles. Except the quality of the "digitization" would get worse, the better the sound hardware on the console. Once you reach the Xbox/PS2/Dreamcast/Gamecube era (or maybe already in the Playstation/N64/Saturn generation) where studio-recorded soundtracks came into use, there wouldn't be much hope. I suspect sample-based music (like MOD on the Nintendo DS) would also be difficult.

mntorankusu · Post by **mntorankusu** » 09-13-07 12:49 am

I think it would be fine if you just decoded the notes and played them using the nearest midi instrument. It doesn't matter if it sounds exactly the same, because the point is to learn to play it on the piano, anyway.

Nicholas wrote:Once you reach the Xbox/PS2/Dreamcast/Gamecube era (or maybe already in the Playstation/N64/Saturn generation) where studio-recorded soundtracks came into use, there wouldn't be much hope. I suspect sample-based music (like MOD on the Nintendo DS) would also be difficult.

Actually, though CD-audio and otherwise recorded music goes back as far as the Sega CD and other early CD-systems, a whooole lot of music on every platform since, up to and including the PS2, Xbox and Gamecube, is sample-based. And, similar to NSF/SPC/etc, the PSX, N64 and PS2 have music formats available (and plugins for winamp and such) to play music from them, PSF, USF and PSF2 respectively.

Also, something related that might interest you. I posted a midi file in the bug report forum that won't play in Synthesia. The interesting part is that it was generated by a program that converts the Nintendo DS music format into midi. I can't find the particular program I used (it was called sseq2mid, and came packed with an SSeq extractor, which is the DS sound format), but I just found out there's a better one here: http://nintendon.s6.xrea.com/pc/dl.php

The source code is available for that too. It extracts all of the audio samples, streams and music files from a DS game into either the original formats (.swar for samples, .strm for streams, and .sseq for music, I think), or converts them to .wav and .midi. I don't know if there's any simple way of automatically linking the .sseq files to the proper samples in the .swar files, though. There's a lot of technical information about the formats that I can't begin to understand here: http://kiwi.ds.googlepages.com/sdat.html

Post by **Nicholas** » 09-13-07 1:05 am

Wow, I just tried the "better one" you mentioned on a DS ROM I had laying around and that works wonders! Again, thanks for the link. The source will be incredibly instructive.

(Thanks for all the other background info too. I didn't realize the newer consoles still had their own formats.)

tommai78101 · Post by **tommai78101** » 09-13-07 11:39 am

To mntorankusu:

Nintendo DS songs in MIDI?

If that is true, then GBA, GBC games should also be able to extract their MIDI with a plugin.

I also noticed that some songs in vgmusic.com (i.e. Pokemon series), they made the songs so accurate, that it is hard for me to believe that the song is made into MIDI. I thought it was extracted from the ROMs and BIOs.

And to Nicholas:

If everything is possible, would you like to try adding some patches or updating plugin to Synthesia so that it can play with these files can let others learn how to play?