Synthesia Talk

Wow. It's been another long while (hello again, Nicholas! Hope your Thanksgiving was good!), but I'm glad to see development on Synthesia is still continuing. I haven't caught up with it lately, so I was pleased to see that MusicXML import is a feature now! Better sheet music is on the way, yay!

With that said, I'd like to know how the MusicXML import works. I have recently upgraded to the latest version of Dorico, so I should be able to produce MusicXML to the latest specifications. However, both the MIDI file and XML file show up as two separate files in the Song Library. (For reference, I am working with The Entertainer by Scott Joplin. I have entered it into Dorico and saved out a MIDI file with the proper splits, and I've saved a separate score with the correct staff notation and saved that as an uncompressed MusicXML file, although I'm aware that compressed files work as well.) The MIDI file plays back with the proper tempo and channel information (including hand splits etc.), but the MusicXML file doesn't play at the right tempo, and doesn't have the correct hand splits.

My current question concerns the coupling between the MIDI file and the MusicXML file. Can you load up a MIDI file in the Metadata Editor and then attach the MusicXML to display the sheet music along with the MIDI file? Or does the MusicXML file necessarily have to be a completely separate entity to the MIDI file?

Wow, long time no see!

Electrode wrote: ↑11-25-22 11:30 amMy current question concerns the coupling between the MIDI file and the MusicXML file...

There is no coupling between files. MusicXML files are loaded as their own, separate song just like MIDI files are.

In what way are the MusicXML splits showing differently than you were expecting? Right now Synthesia is splitting on (every possible combination of) instrument and staff (but not by voice, by default). To get it to also split on voice, you can enable the "MusicXML.SplitVoices" advanced option. That makes lots of extra "parts" on the Hands, Colors, and Instruments screen though, depending on the complexity of the song.

Nicholas wrote: ↑11-26-22 12:43 pm Wow, long time no see!

I know! It's been so long! (Plus I have a habit of dropping off the face of the earth for a bit and then randomly managing to find my way back! )

Nicholas wrote: ↑11-26-22 12:43 pm There is no coupling between files. MusicXML files are loaded as their own, separate song just like MIDI files are.

So does this mean that MusicXML files will eventually completely replace MIDI files?

Nicholas wrote: ↑11-26-22 12:43 pm In what way are the MusicXML splits showing differently than you were expecting? Right now Synthesia is splitting on (every possible combination of) instrument and staff (but not by voice, by default). To get it to also split on voice, you can enable the "MusicXML.SplitVoices" advanced option. That makes lots of extra "parts" on the Hands, Colors, and Instruments screen though, depending on the complexity of the song.

I have uploaded four files. The first one is the MIDI file, which is split correctly. The second one is the output from the MusicXML file. The MIDI file is split by channel as usual, and displays the split correctly. The MusicXML file looks the same, but doesn't split the hands correctly as it is going by the XML data rather than the MIDI data. (The same MIDI data is used to export the MusicXML file from Dorico, and in Dorico, I have the music notated exactly as the sheet music, with the cross-staff notes.)

The third one is the original public domain sheet music from IMSLP. You can see it begins with the left hand in the treble clef with stems pointing down, while the right hand has the stems pointing up. So both the MIDI and the XML display correctly in-game, but the XML doesn't have any knowledge of how the split should actually be handled. The fourth file is the Dorico source file, for reference.

This is why I was asking about coupling, since you could get the split from the MIDI and use the XML for sheet music display. Exported MIDI files already have repeats realised in Dorico (and Sibelius too, so presumably MuseScore has this too), so the MIDI would automatically match up to the score anyway. Why reinvent the wheel when the MIDI playback code is already implemented? Coupling would be the easiest way to implement the MusicXML import (in my humble opinion), considering that if you're exporting from a notation program, you would already be able to export a MIDI file with all repeats properly handled, and you'd have a proper tempo guide in the MIDI file to sync up the MusicXML display to.

I realise that cross-staff notes are an edge case, but to solve this, it might work to check for splits on stem direction also, assuming you already know that the top line has stems up and the bottom line has stems down. This could potentially make things like counterpoint/polyphonic music (such as Bach inventions or fugues) easier to parse from the XML. As it stands though, determining the MIDI data from the MusicXML file could be a bit of a headache, as the XML is primarily intended for the reproduction of the visual sheet music, and doesn't have a good chance in complex cases of aligning with what a MIDI file should contain with regards to channel or note information, so that it can work properly with Synthesia. For example, how would you handle trills and other ornaments that are just displayed via symbols on the music? I'm concerned that this solution would end up requiring a lot of custom parsing logic that could become almost as much of a headache as the original sheet music display was to make.

UPDATE: I have found out that I can fix the split using the Custom split option in "Hands, Colors and Instruments", so that's a relatively tiny amount of work, and it may be the best way to handle this in future.

Electrode wrote: ↑11-26-22 1:33 pmSo does this mean that MusicXML files will eventually completely replace MIDI files?

The things Synthesia can do with a MusicXML file will become a proper superset of the things it can do with MIDI files. Now, whether that means they completely replace MIDI is up to users. Today there are many more MIDI files out in the wilderness than there are MusicXML, but yes, the idea is that your Synthesia music collection could be exclusively MusicXML files. And just in case there were any question, Synthesia will always support MIDI files (until the end of time), so there isn't any worry about that format becoming deprecated or anything like that. MusicXML files will just give you better sheet music on top of what MIDI already provides.

Electrode wrote: ↑11-26-22 1:33 pmThe MusicXML file looks the same, but doesn't split the hands correctly as it is going by the XML data rather than the MIDI data.

Are we talking about the split across sheet staffs? Or the split between the colors of the falling note blocks?

Today (through 10.9), Synthesia still doesn't make any distinction between treble and bass staff w.r.t. hands. It simply puts notes where they're most convenient, regardless of hand. (This is obviously incorrect behavior that will be fixed in Synthesia 11.)

When I talk about splitting hands, I'm usually referring to how the falling note blocks have been separated. As long as those are showing the hands correctly, things should start to look better when the app begins using the extra sheet music information stored in the MusicXML file in Synthesia 11.

Electrode wrote: ↑11-26-22 1:33 pmWhy reinvent the wheel when the MIDI playback code is already implemented?

I was actually pretty happy with my repeat unrolling code that I wrote in 10.8 for MusicXML. It's been done for over a year, so the wheel has already been reinvented.

To the best of my knowledge, outside of codas (D.S. al coda, etc.), I have most of those things implemented correctly. And because MusicXML has most of the expressivity of MIDI (outside of maybe RPN/NRPN messages), you don't gain that much by requiring both files.

Electrode wrote: ↑11-26-22 1:33 pm... assuming you already know that the top line has stems up and the bottom line has stems down. This could potentially make things like counterpoint/polyphonic music (such as Bach inventions or fugues) easier to parse from the XML.

It's even better than relying on stem direction: MusicXML has explicit support for denoting which notes belong to which "voice". So there isn't any ambiguity or guessing at all.

Electrode wrote: ↑11-26-22 1:33 pmFor example, how would you handle trills and other ornaments that are just displayed via symbols on the music? I'm concerned that this solution would end up requiring a lot of custom parsing logic that could become almost as much of a headache as the original sheet music display was to make.

So far I've punted on things like trills and grace notes. You're right that it's more work, but I think it's worth the simplicity of only needing a single file. (There would also be a lot of work--even more?--involved in trying to reconcile the differences between two separate versions of the same song. Because of things like trills, the number of notes between files would differ and I'd need a lot of fiddly code to try and match things back up.)

Electrode wrote: ↑11-26-22 1:33 pmUPDATE: I have found out that I can fix the split using the Custom split option in "Hands, Colors and Instruments", so that's a relatively tiny amount of work, and it may be the best way to handle this in future.

With all of the expressiveness in MusicXML (instruments, staffs, and voices), this should be automatic. It might just be a matter of tweaking the way things are encoded in Dorico, but you should be able to get this for free without requiring the extra step in Synthesia.