MusicXML Coding/Research

Synthesia is a living project. You can help by sharing your ideas.
Search the forum before posting your idea. :D

No explicit, hateful, or hurtful language. Nothing illegal.
JimNYC
Posts: 8

Post by JimNYC »

I know MusicXML support in on the horizon for some future release. There may be resources on this board who also have some coding skills, so is there any initial research or background framework setup that would help you ultimately in the implementation? It could be approached in a modular way, such that it is separate from the actual app source code. I think many people would be willing to help if they could!
Nicholas
Posts: 12572

Post by Nicholas »

This is an interesting idea, although, I worry that I wouldn't really know how to get an effort like that started.

It reminds me of this discussion, which I was also enthusiastic about. An interesting lesson I've learned each of the three or four times that I've let someone else help with the project is that it always seems to take just about as long to spin someone up on the technical details (and how they related to Synthesia's particularities) as it would to sit down and do the work myself. After a short while, they seem to lose interest and move onto other things. (This is despite it being a reasonably well-paid position, too!)

That said, for larger projects the "just as long to spin up as doing it myself" isn't necessarily the case. (I'm probably overstating my own speed, too; all of this probably reads like someone that has a problem giving up any control at all. Because I do.) :lol:

The real question would be how the MusicXML work could be sliced up to maximize this sort of opportunity (assuming I could find someone ready, able, and willing to help). There's still some rendering work (drawing arbitrary, closed Bezier shapes for things like slurs and ties). That stuff is probably too deeply integrated with the rest of the drawing code to pull it out as a modular task. Loading a MusicXML file and (through one or more steps) turning it into a list of drawing commands is a nice, big modular chunk, I suppose.

There's a question that goes with it, though: today Synthesia loads MIDI files into it's own custom internal musical representation. The existing sheet music feature takes that internal representation and turns it into a list of glyphs that need to be drawn. With MusicXML, we now have two paths to sheet music. So should MusicXML get boiled down to that same internal representation (which would need to be substantially extended to be able to express all the new things that come with MusicXML)? Or should it go the other way: MusicXML directly into glyphs, and then insert an intermediate step in the old MIDI-handling infrastructure where the internal representation is converted to basic MusicXML?

Today
MIDI → [internal] → glyphs

Possibility A
MIDI → [expanded internal] → glyphs
MusicXML → [expanded internal] → glyphs

Possibility B
MusicXML → glyphs
MIDI → [internal] → simple, interpreted MusicXML → glyphs

Possibility B seems to be the more modular (and frankly, useful) of the two. For only the price of a simple MIDI-to-MusicXML converter, everything goes through the same path, the internal representation (for things like falling notes and keeping track of the user's performance) doesn't need to change at all, and I'd get to keep the fuzzy "MIDI to sheet music" guestimate stuff as far away from the "pure" sheet music drawing code as possible.
User avatar
jimhenry
Posts: 1836

Post by jimhenry »

Perhaps this MIT licensed project on GitHub would be of interest for Possibility B:
My guess is that MIDI -> MusicXML will be too slow to do it on the fly though.
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/
JimNYC
Posts: 8

Post by JimNYC »

I would think it would be preferable to render from MusicXML directly to sheet music, since that is it's core construct. Plus, there is lots of documentation on that and open-source examples of rendering (like MuseScore). So Option B seems like it would be the better long-term implementation.

However, having said that, it boils down also to what may be a quicker way to achieve the end goal and what benefit MusicXML provides. I can't speak for others, but one of the main benefits that MusicXML support would bring is a more accurate display of basic note position. Right now, Synthesia is great to use to practice classical files that I can find in midi format. But, the sheet music display in Synthesia is often too "jumbled" with notes on different clefs to learn or play from directly, so that I end up using both a printed copy of the original sheet music along with Synthesia next to it to track my playing. MusicXML would take some of the guess work out by allowing you to know which clef notes really belonged on or clef changes within each hand's part. I know there is a lot more that MusicXML can express, but just drawing the notes together as intended would make it significantly more usable than it is now for actual note reading. So if it's actually easier to just extend the internal format to maintain note positioning/clef changes and the glyph routines can just use that, then it would be a massive improvement.
JimNYC
Posts: 8

Post by JimNYC »

I saw this as well which deals with parsing MusicXML and then converting to Vexflow for rendering in a browser:

https://opensheetmusicdisplay.org/about/
User avatar
jimhenry
Posts: 1836

Post by jimhenry »

Agreed that MusicXML rendered directly to sheet music is the way to go for the best sheet music display. That is the centerpiece of the Synthesia v11 release.

The question is what to do about sheet music display from MIDI files, which are going to be the vast majority of files used in Synthesia for the foreseeable future.

If MIDI can be converted to MusicXML quickly, then Synthesia only needs to have a way to display MusicXML as sheet music. Obviously, MIDI to MusicXML is not going to yield perfect sheet music because there isn't enough information in a MIDI file to do that. It just needs to be at least as good the sheet music Synthesia now generates from MIDI files.

Or maybe Synthesia can demand that there be a MusicXML file if you want to display sheet music and provide an off-line MIDI to MusicXML converter.
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/
Nicholas
Posts: 12572

Post by Nicholas »

jimhenry wrote: 04-06-21 8:39 am Perhaps this MIT licensed project on GitHub would be of interest for Possibility B...
Poking around in the repository, I started wondering where all the "real" code was--most of the files in the meat of it only have a dozen or two lines--but once I understood the description on the project page home, it sounds like their converter exists because they wanted to run MIDI files through NEUTRINO (a "Neural Singing Synthesizer") but it only accepts MusicXML files.

Their MIDI parser only understands Note On, Note Off, and Tempo messages (discarding the rest). :grimace: So the output format is essentially MusicXML in name only, with no regard for how it might look on a printed page. It only carries along enough information to (barely) be played through a synth, not unlike MIDI itself.

(Nice find either way. I'm sure there are other efforts out there like this one that care more about the particulars.)
jimhenry wrote: 04-06-21 8:39 amMy guess is that MIDI -> MusicXML will be too slow to do it on the fly though.
This is a personal challenge I've taken on. The current state of affairs (my computer is 100x faster than it was in 1998 on every axis but Photoshop takes 5-10x longer to open a file today than it did back then) leaves me openly disgusted. Each of us carries an unfathomable amount of computing power in our pockets and it is only for lack of rigor that things seem to be backsliding now.

The hardest part of sheet music is drawing it, and since the revamp in 10.4, Synthesia can draw each lovely, high-resolution, anti-aliased staff (on an iPad 2 from 2011) spread carefully across three frames (1/20th of a second) using so little of the time budget that the rest of the app doesn't stutter.

MIDI-to-MusicXML is a comparatively simple, quintessential "text-to-text" transform that Computer Science is best at doing. If I can't do the conversion for an entire song in under 100ms, I'll be disappointed in myself. :anxious: That's enough time to send a network packet to the other side of the world. That's long enough to read the average MIDI file off the built-in flash drive on a tablet six thousand times. Something like two billion clock cycles (across all cores of the average iPad) occur in that interval. It's staggering how much we waste these days.
JimNYC wrote: 04-06-21 11:00 am... but just drawing the notes together as intended would make it significantly more usable than it is now for actual note reading. So if it's actually easier to just extend the internal format to maintain note positioning/clef changes and the glyph routines can just use that, then it would be a massive improvement.
This is hopefully how things will proceed. I've long since learned that trying to sit down and do all of the work at once is non-starter for me. Instead, I've broken things down and am hoping to mete out small improvements to the sheet music across each (hopefully more regular) update. The initial "hey, you can load a MusicXML file" update might just wire the notes found in the file directly to the MIDI loading code, showing none of the benefits at first... but it'll be a big step in the right direction, leaving me ready to take the next step that you just outlined.
JimNYC wrote: 04-06-21 3:58 pmI saw this as well which deals with parsing MusicXML and then converting to Vexflow...
That looks like a much more serious effort. (I didn't have to play the "Where's the code?" game this time.) :lol: Now I'm going to sound like I'm being picky: when it's not in the same language (TypeScript instead of Synthesia's C++), there are some calculations that need to be made about how much effort it might take to incorporate a TS/JS interpreter or possibly do a rewrite in the target language by hand. (Coming from a strongly-typed, imperative language like TypeScript, this might actually not be so bad.) Even if I do neither of those things, at a minimum, it looks like it will serve as a good reference if/when questions come up.

Thanks for the link!
jimhenry wrote: 04-07-21 1:11 pmOr maybe Synthesia can demand that there be a MusicXML file if you want to display sheet music and provide an off-line MIDI to MusicXML converter.
Assuming I can get the speed anywhere in the same ballpark as my bluster above, this should (hopefully) be a non-issue. Even if it ended up being a 1-3 second process, that still feels "correct" inline, inside the app. Much longer than that and it might make sense to turn it into a standalone tool.
User avatar
jimhenry
Posts: 1836

Post by jimhenry »

The initial "hey, you can load a MusicXML file" update might just wire the notes found in the file directly to the MIDI loading code, showing none of the benefits at first... but it'll be a big step in the right direction, leaving me ready to take the next step that you just outlined.
Doesn't a MusicXML file basically have two halves, a this is how its played and a this is how its drawn part? If so, is the this is how its played part close to being a MIDI file?

I think wiring in some use of MusicXML that is only as capable as a MIDI file soon would have the benefit of allowing users to start building up libraries of MusicXML files. Having users with substantial MusicXML libraries will help in testing the additional features of MusicXML as they are added.
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/
Piraya
Posts: 2

Post by Piraya »

Hi Guy's

I Have a rather large library consisting of music xml files....
If i can contribute in any way just give me a wink :)
Nicholas
Posts: 12572

Post by Nicholas »

jimhenry wrote: 04-08-21 11:25 amI think wiring in some use of MusicXML that is only as capable as a MIDI file soon would have the benefit...
I actually... completely agree. Just typing the above scenarios made me wonder why my original plan had MusicXML coming in so late in the process. It wasn't going to be "perfect"/"100% complete" at that point either, so if I'm already going to be improving it on-the-go, why not get it in there much earlier?

Lacking a better answer, I just pulled the half-dozen lines covering "bare minimum MusicXML parsing/loading" from around line 730 up to line 46. That will be in the very next update.
Piraya wrote: 04-08-21 1:25 pmI Have a rather large library consisting of music xml files....
If i can contribute in any way just give me a wink :)
I might take you up on the offer, actually. A million years ago, when I was first developing Synthesia, MIDI files were (and continue to be) very easy to come by. While I'm sure I could spend some time dredging up lots of MusicXML files, saving some of that effort sounds good!

They'd be used strictly for internal development purposes only and would never be redistributed to anyone else. Send whatever you'd like to support@synthesiagame.com and hopefully I'll be able to use them during testing. (I'd prefer a link to some zipped up archive on some cloud service or file sharing site; although directly attached files would work in a pinch, too.)

Thanks!
User avatar
jimhenry
Posts: 1836

Post by jimhenry »

Where do you stand with regard to MusicXML versions of the included files from the DeBenedetti G Major site?
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/
JimNYC
Posts: 8

Post by JimNYC »

I just sent some files in MusicXML, PDF and midi formats. There is a large repository of MusicXML files at MuseScore.com. Searching for "PianoXML" will come back with 100+ piano files available in MusicXML/PDF/midi format, and these are all accessible with a free account.

However, looking at the files and how Synthesia renders the sheet music from the midi, I think two things that would improve things in the current rendering (without anything to do with MusicXML) has to do with the hands and what clef Synthesia draws them on. Most time, the left and right hand parts are separate hands that can be adjusted in the Hands, Colors and Instruments section. However, when both hands are drawn as sheet music, the left hand will cross into the treble clef, rather than drawing them as higher/lower notes on the bass clef (and vice versa). The two things that would improve sheet music readability from midi files is:

1) keep left hand and right hand separated on each respective clef (or make this an option) rather than drawing the left hand part into the treble clef and the right hand part into the bass clef

2) when #1 is enabled, detect some interval whereby it would draw a clef change into the music when the notes would become ridiculous to draw above the clef lines.

See the Schumann Op 68, No 7 I sent for examples of these in measures 4 and 9. Those 2 things alone, without MusicXML itself, would improve sheet music readability tremendously.
User avatar
jimhenry
Posts: 1836

Post by jimhenry »

The two things that would improve sheet music readability from midi files...
I think this idea has considerable merit. Even after MusicXML is fully supported, there will still be considerable interest in a sheet music display from MIDI files. I think JimNYC proposal in pseudocode is this:

Code: Select all

if MIDI_file {
    if Hand_info {
        if Right_hand {
            //Display on upper clef with logic to shift clef range as needed
        } else {    // Left_hand
            //Display on lower clef with logic to shift clef range as needed
        }
    } else {    //No Hand_info
        Current MIDI file sheet music display processing
    }
} else {    //MusicXML_file
    MusicXML processing
}
Aside from the clef range shifting logic, this should be pretty simple to implement. While I am pretty sure the resulting sheet music display from a MIDI file and hand splitting info won't be best practice music engraving, the result is probably what many players would want. I have seen more than a few pencil markings added to scores where right hand notes are on the lower clef or left hand notes are on the upper clef.

Clef shifting would probably be useful for display of MIDI files without hand information as well although it would limited to things going off the top of the treble clef or the bottom of the bass clef, since I don't imagine you'd want to bother with trying to deduce situations that could use two treble or two bass clefs rather than the grand staff.
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/
Nicholas
Posts: 12572

Post by Nicholas »

jimhenry wrote: 04-10-21 9:24 am Where do you stand with regard to MusicXML versions of the included files from the DeBenedetti G Major site?
I don't have any of them yet. This very first "MusicXML loads as not much better than MIDI" pass probably wouldn't be the right time to switch them all over. There is a small (but non-zero) chance that, ironically, Synthesia might be able to interpret things from MIDI more effectively than from MusicXML.

Another reason to drag my feet on the switch-over is because it will effective reset everyone's song practice progress on all the built-in songs (because they're not the exact same file). I intend to keep shipping the "old" MIDIs forever, as a disabled-by-default song entry. Something like "Legacy Built-in Songs". That way if someone has years of statistics and progress, it won't all be lost.

So while I'll need to convert them eventually, there isn't any time pressure for this release.
JimNYC wrote: 04-10-21 11:27 amI just sent some files in MusicXML, PDF and midi formats.
Thanks, I received them!
JimNYC wrote: 04-10-21 11:27 amThe two things that would improve sheet music readability from midi files is...
Separate staffs for each hand (and inline clef changes + 8va/8vb that will be required for them) were part of the plan for MIDI-generated sheet music. Now they'll just be after the introduction of MusicXML loading instead of before.
revilo2
Posts: 124

Post by revilo2 »

Too bad...

Separate staffs for each hand and BEAMS (as the original sheet) are the most important elements (for me) to have a readable sheet....
User avatar
jimhenry
Posts: 1836

Post by jimhenry »

So while I'll need to convert [the DeBenedetti files] eventually, there isn't any time pressure for this release.
To sort of loop back to the original thought that JimNYC had, do YOU need to convert these files? Or is this something that can be done as a community project, like the finger hints were? Regardless of who does it, it is going to be time consuming. Seems like something that should be started sooner rather than later.

Maybe you should leave the current MIDI DeBenedetti files as the default and make using the MusicXML files an option for those who want good sheet music rather than score history.
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/
Nicholas
Posts: 12572

Post by Nicholas »

revilo2 wrote: 04-13-21 4:08 am Too bad...

Separate staffs for each hand and BEAMS (as the original sheet) are the most important elements (for me) to have a readable sheet....
You'll get much better beaming (and, uh... "staffing"? :lol: ) from MusicXML files much sooner than you would have from MIDI. And the conversion in most of the apps that can handle both formats (MuseScore, etc.) should be just a couple clicks. (And that way you'll even get a chance to manually fix things, which you'd never get with Synthesia.)
jimhenry wrote: 04-13-21 8:45 amTo sort of loop back to the original thought that JimNYC had, do YOU need to convert these files?
Technically, no, I don't have to do them myself. Although in the intervening many years I've started to feel a little strange about unpaid work from volunteers. The project is... rather healthy, so accepting free labor has begun to feel a little exploitative. For language translations, it's less awkward because it's a task I simply cannot do myself. But when I'm "buying" my MIDI-to-MusicXML conversion time back for $0, well, that's the definition of getting something for nothing.

Who's got a PayPal account that wants to convert songs?! :lol: I could put a bounty on each one or something. As long as it stays under the reporting threshold for my side (which is what, $600?) that should keep the arrangement super simple.
revilo2
Posts: 124

Post by revilo2 »

Nicholas,

I have converted all G Majors midi files in Musicxml files

I want to send you the rar file by mail but i don't know how to do ...

EDIT: Done.
Nicholas
Posts: 12572

Post by Nicholas »

Automatic machine translation will be a good first step (though, I'd start with the MIDI versions that ship with Synthesia, as they've already been quantized and had the left/right hands split into separate tracks; that should lead to much better initial conversion results).

Though after that, the next step is a meticulous comparison with the original sheet music PDFs to make sure the final conversion contains as much of the original musical intent as possible.
revilo2
Posts: 124

Post by revilo2 »

Can you send me the MIDI versions that ship with Synthesia, i could do the first translation and, later the comparision with the pdf.
Post Reply