Play along with the Original Song

Synthesia is a living project. You can help by sharing your ideas.
Search the forum before posting your idea. :D

No explicit, hateful, or hurtful language. Nothing illegal.
Post Reply
Posts: 25
Location: Germany

Post by AllNotesCount » 11-02-19 5:20 am


Usually, I use Synthesia with midi files, which I've bought from a midi shop and which include the full instrumentation of the songs. Many of those midi files are quite well transcribed nowadays (or at least often need only small adjustments to sufficiently match the originals). In Synthesia, I keep up the sound of all instruments, in order to get as close to a real play-along experience as possible.

Anyway, it still would be better to also be able to practice with the sound of the original songs (plus the option to add the midi sound of the instrument that I want to practice) - for different reasons, e.g.:

1) As a musician, you are always in the dilemma that you have to share your time between listening to the original music (original sound of the instruments as well as the voice/vocals of the original artist, for song analysis purposes or simply as a regular music listener) and making music yourself. That's why I love tools like Rocksmith and Go Playalong, where you can do both at the same time. Anyway, those tools have disadvantages in other aspects (e.g. Rocksmith is only for string instruments, and Go Playalong doesn't have the falling notes feature).
2) If you want to practice for playing a cover of a specific tune with your band, you can use Synthesia with the midi sound for the initial practicing, but you need to play with the original tune outside of Synthesia much sooner than with the tools mentioned above (where you play without the software only in the "final stage" when you try to remember all notes by yourself).
3) The vocals/lyrics can also give some orientation for your instrument, plus they help to memorize the song structure as early as possible.
4) Realizing that the original performance often is the reason why someone wants to learn to play that song at all: Playing with the Original is simply more fun than playing with midi sounds.

Anyway, I think that scripting a suitable feature up to a perfect stage (e.g. including editing options to match the timelines of the original tune and the midi, like it exists in Go Playalong) would be quite a lot of work. Therefore, I'd like to suggest a leaner concept:

Let's say that the user already has matched the timeline of his midi file with the original song outside of Synthesia, and that he has added the mp3 of the original song to his midi folder. Then

1) in Synthesia (or in the Synthesia Metadata Editor) he would only need an option to link the mp3 with the midi
2) and Synthesia could then start the mp3 at the same time as it starts the midi
3) but the user should still be able to add the sound of midi tracks within Synthesia as well as to occasionally mute the mp3 sound within Synthesia
4) Anyway, there's at least one more thing to be considered: If the user scrolls a song forward or backward or loops a part within Synthesia, this would also have to effect the mp3 file

Well ... would it be complicated to implement something like this?

Posts: 12285

Post by Nicholas » 11-11-19 6:38 am

This has been a recurring suggestion over the years, for the good reasons that you provided and more.

There would be a couple tricky parts involved:

1. Until very recently, Synthesia hasn't taken it upon itself to actually produce sound. It's been dealing in MIDI messages exclusively, which would have made regular sound output hard. Now that it does (everywhere but Mac, so far) and we have a robust, easy-to-use sound component built into the app, this is less scary. The only trouble remaining is reconciling the MIDI latency with the sound playback latency. With background notes and a user's notes sharing the same output device today, there wasn't ever the question of synchronizing the two. With different output methods, there would be. The whole process would be very sensitive to latency in any/all of the steps involved. (This is surmountable. It would just take a lot of work.)

2. In the same way that combining the two versions of the song would be very sensitive to latency, they would be even more sensitive to any microscopic deviations in the song's tempo. This just came up when a user very meticulously crafted a MIDI to be tempo-matched with a recording but their editor saved the result with 1 part in 1000 less timing precision. By the end of the (admittedly long) song, the two were out of sync by a quarter second.

It is a very high bar to assume you'll have an expertly crafted tempo map of the song available to perfectly join the MIDI and MP3 versions. I wouldn't want to leave that to chance. Instead, some kind of automatic method might be best. I remember seeing this impressive video from Google a number of years ago demonstrating a kind of automatic, non-linear solver that might be suitable for the job. The MP3's frequency spectrum could be inspected (see top image, here) to find the biggest spikes of energy. Those could then be correlated to the positions of the notes in the MIDI file, which should (hopefully) result in a perfectly-matching tempo map.

Again, it's probably surmountable (given an awesome, free non-linear solver like Google has on offer here), but would still be quite a bit of work!

Posts: 25
Location: Germany

Post by AllNotesCount » 11-11-19 1:52 pm

Hey Nicholas,

many thanks for your diffenciated reply!

I see, it's more complicated than I hoped it would be :roll:

I've explored a bit some methods of creating Rocksmith Custom Songs for my personal use, a while ago (including matching a Guitar Pro File with the MP3 in Go Playalong, then creating an XML-File that includes the tempo map by Go Playalong's export feature and then importing the MP3 and the XML into Editor on Fire), and I was just hoping that something similar might work for Synthesia as well.

Well, maybe the whole "Original Song Play-Along" idea might simply be kept in mind as a possible long term goal, especially since others also seem to have something like that on their wishlist.

And ... by the way ... although every tool can always get further improved, I love Synthesia already as it is. Great tool, terrific work!!

Posts: 39

Post by raynebc » 11-11-19 3:22 pm

Tempo mapping isn't as hard as many people think. As was mentioned, some tools like Go PlayAlong and Sonic Visualizer can try to automate it somewhat, otherwise you have mechanisms in rhythm game chart editors such as a waveform graph and a metronome to test sync.

Posts: 25
Location: Germany

Post by AllNotesCount » 11-11-19 4:24 pm

To avoid misunderstandings, I should be more precise:

I matched the Guitar pro file and the MP3 file manually in Go Playalong (this part often doesn't work good enough if it's done automatically by this software, at least for Rocksmith purposes), and then I used the automatic export to XML feature.

User avatar
Posts: 1789
Location: Illinois

Post by jimhenry » 11-12-19 2:15 am

Maybe it isn't as hard as it looks. If we say the premise is to help the musician play along with an MP3, then the MIDI/XML file is only driving the note display, not actually accompanying the MP3. It is the musicians responsibility to stay in sync with the MP3, not Synthesia's. Of course Synthesia needs to be fairly close if the note display is to stay meaningful. But the tolerance is probably something like +/- 1/32 note. The scoring, if it is even done in this mode, would be loosened up considerably because the musician knows better than Synthesia what the right note timing is.
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ

Post Reply