MusicXML Coding/Research
Search the forum before posting your idea.
No explicit, hateful, or hurtful language. Nothing illegal.
No explicit, hateful, or hurtful language. Nothing illegal.
I know MusicXML support in on the horizon for some future release. There may be resources on this board who also have some coding skills, so is there any initial research or background framework setup that would help you ultimately in the implementation? It could be approached in a modular way, such that it is separate from the actual app source code. I think many people would be willing to help if they could!
This is an interesting idea, although, I worry that I wouldn't really know how to get an effort like that started.
It reminds me of this discussion, which I was also enthusiastic about. An interesting lesson I've learned each of the three or four times that I've let someone else help with the project is that it always seems to take just about as long to spin someone up on the technical details (and how they related to Synthesia's particularities) as it would to sit down and do the work myself. After a short while, they seem to lose interest and move onto other things. (This is despite it being a reasonably well-paid position, too!)
That said, for larger projects the "just as long to spin up as doing it myself" isn't necessarily the case. (I'm probably overstating my own speed, too; all of this probably reads like someone that has a problem giving up any control at all. Because I do.)
The real question would be how the MusicXML work could be sliced up to maximize this sort of opportunity (assuming I could find someone ready, able, and willing to help). There's still some rendering work (drawing arbitrary, closed Bezier shapes for things like slurs and ties). That stuff is probably too deeply integrated with the rest of the drawing code to pull it out as a modular task. Loading a MusicXML file and (through one or more steps) turning it into a list of drawing commands is a nice, big modular chunk, I suppose.
There's a question that goes with it, though: today Synthesia loads MIDI files into it's own custom internal musical representation. The existing sheet music feature takes that internal representation and turns it into a list of glyphs that need to be drawn. With MusicXML, we now have two paths to sheet music. So should MusicXML get boiled down to that same internal representation (which would need to be substantially extended to be able to express all the new things that come with MusicXML)? Or should it go the other way: MusicXML directly into glyphs, and then insert an intermediate step in the old MIDI-handling infrastructure where the internal representation is converted to basic MusicXML?
Today
MIDI → [internal] → glyphs
Possibility A
MIDI → [expanded internal] → glyphs
MusicXML → [expanded internal] → glyphs
Possibility B
MusicXML → glyphs
MIDI → [internal] → simple, interpreted MusicXML → glyphs
Possibility B seems to be the more modular (and frankly, useful) of the two. For only the price of a simple MIDI-to-MusicXML converter, everything goes through the same path, the internal representation (for things like falling notes and keeping track of the user's performance) doesn't need to change at all, and I'd get to keep the fuzzy "MIDI to sheet music" guestimate stuff as far away from the "pure" sheet music drawing code as possible.
It reminds me of this discussion, which I was also enthusiastic about. An interesting lesson I've learned each of the three or four times that I've let someone else help with the project is that it always seems to take just about as long to spin someone up on the technical details (and how they related to Synthesia's particularities) as it would to sit down and do the work myself. After a short while, they seem to lose interest and move onto other things. (This is despite it being a reasonably well-paid position, too!)
That said, for larger projects the "just as long to spin up as doing it myself" isn't necessarily the case. (I'm probably overstating my own speed, too; all of this probably reads like someone that has a problem giving up any control at all. Because I do.)
The real question would be how the MusicXML work could be sliced up to maximize this sort of opportunity (assuming I could find someone ready, able, and willing to help). There's still some rendering work (drawing arbitrary, closed Bezier shapes for things like slurs and ties). That stuff is probably too deeply integrated with the rest of the drawing code to pull it out as a modular task. Loading a MusicXML file and (through one or more steps) turning it into a list of drawing commands is a nice, big modular chunk, I suppose.
There's a question that goes with it, though: today Synthesia loads MIDI files into it's own custom internal musical representation. The existing sheet music feature takes that internal representation and turns it into a list of glyphs that need to be drawn. With MusicXML, we now have two paths to sheet music. So should MusicXML get boiled down to that same internal representation (which would need to be substantially extended to be able to express all the new things that come with MusicXML)? Or should it go the other way: MusicXML directly into glyphs, and then insert an intermediate step in the old MIDI-handling infrastructure where the internal representation is converted to basic MusicXML?
Today
MIDI → [internal] → glyphs
Possibility A
MIDI → [expanded internal] → glyphs
MusicXML → [expanded internal] → glyphs
Possibility B
MusicXML → glyphs
MIDI → [internal] → simple, interpreted MusicXML → glyphs
Possibility B seems to be the more modular (and frankly, useful) of the two. For only the price of a simple MIDI-to-MusicXML converter, everything goes through the same path, the internal representation (for things like falling notes and keeping track of the user's performance) doesn't need to change at all, and I'd get to keep the fuzzy "MIDI to sheet music" guestimate stuff as far away from the "pure" sheet music drawing code as possible.
Perhaps this MIT licensed project on GitHub would be of interest for Possibility B:
My guess is that MIDI -> MusicXML will be too slow to do it on the fly though.
My guess is that MIDI -> MusicXML will be too slow to do it on the fly though.
I would think it would be preferable to render from MusicXML directly to sheet music, since that is it's core construct. Plus, there is lots of documentation on that and open-source examples of rendering (like MuseScore). So Option B seems like it would be the better long-term implementation.
However, having said that, it boils down also to what may be a quicker way to achieve the end goal and what benefit MusicXML provides. I can't speak for others, but one of the main benefits that MusicXML support would bring is a more accurate display of basic note position. Right now, Synthesia is great to use to practice classical files that I can find in midi format. But, the sheet music display in Synthesia is often too "jumbled" with notes on different clefs to learn or play from directly, so that I end up using both a printed copy of the original sheet music along with Synthesia next to it to track my playing. MusicXML would take some of the guess work out by allowing you to know which clef notes really belonged on or clef changes within each hand's part. I know there is a lot more that MusicXML can express, but just drawing the notes together as intended would make it significantly more usable than it is now for actual note reading. So if it's actually easier to just extend the internal format to maintain note positioning/clef changes and the glyph routines can just use that, then it would be a massive improvement.
However, having said that, it boils down also to what may be a quicker way to achieve the end goal and what benefit MusicXML provides. I can't speak for others, but one of the main benefits that MusicXML support would bring is a more accurate display of basic note position. Right now, Synthesia is great to use to practice classical files that I can find in midi format. But, the sheet music display in Synthesia is often too "jumbled" with notes on different clefs to learn or play from directly, so that I end up using both a printed copy of the original sheet music along with Synthesia next to it to track my playing. MusicXML would take some of the guess work out by allowing you to know which clef notes really belonged on or clef changes within each hand's part. I know there is a lot more that MusicXML can express, but just drawing the notes together as intended would make it significantly more usable than it is now for actual note reading. So if it's actually easier to just extend the internal format to maintain note positioning/clef changes and the glyph routines can just use that, then it would be a massive improvement.
I saw this as well which deals with parsing MusicXML and then converting to Vexflow for rendering in a browser:
https://opensheetmusicdisplay.org/about/
https://opensheetmusicdisplay.org/about/
Agreed that MusicXML rendered directly to sheet music is the way to go for the best sheet music display. That is the centerpiece of the Synthesia v11 release.
The question is what to do about sheet music display from MIDI files, which are going to be the vast majority of files used in Synthesia for the foreseeable future.
If MIDI can be converted to MusicXML quickly, then Synthesia only needs to have a way to display MusicXML as sheet music. Obviously, MIDI to MusicXML is not going to yield perfect sheet music because there isn't enough information in a MIDI file to do that. It just needs to be at least as good the sheet music Synthesia now generates from MIDI files.
Or maybe Synthesia can demand that there be a MusicXML file if you want to display sheet music and provide an off-line MIDI to MusicXML converter.
The question is what to do about sheet music display from MIDI files, which are going to be the vast majority of files used in Synthesia for the foreseeable future.
If MIDI can be converted to MusicXML quickly, then Synthesia only needs to have a way to display MusicXML as sheet music. Obviously, MIDI to MusicXML is not going to yield perfect sheet music because there isn't enough information in a MIDI file to do that. It just needs to be at least as good the sheet music Synthesia now generates from MIDI files.
Or maybe Synthesia can demand that there be a MusicXML file if you want to display sheet music and provide an off-line MIDI to MusicXML converter.
Poking around in the repository, I started wondering where all the "real" code was--most of the files in the meat of it only have a dozen or two lines--but once I understood the description on the project page home, it sounds like their converter exists because they wanted to run MIDI files through NEUTRINO (a "Neural Singing Synthesizer") but it only accepts MusicXML files.
Their MIDI parser only understands Note On, Note Off, and Tempo messages (discarding the rest). So the output format is essentially MusicXML in name only, with no regard for how it might look on a printed page. It only carries along enough information to (barely) be played through a synth, not unlike MIDI itself.
(Nice find either way. I'm sure there are other efforts out there like this one that care more about the particulars.)
This is a personal challenge I've taken on. The current state of affairs (my computer is 100x faster than it was in 1998 on every axis but Photoshop takes 5-10x longer to open a file today than it did back then) leaves me openly disgusted. Each of us carries an unfathomable amount of computing power in our pockets and it is only for lack of rigor that things seem to be backsliding now.
The hardest part of sheet music is drawing it, and since the revamp in 10.4, Synthesia can draw each lovely, high-resolution, anti-aliased staff (on an iPad 2 from 2011) spread carefully across three frames (1/20th of a second) using so little of the time budget that the rest of the app doesn't stutter.
MIDI-to-MusicXML is a comparatively simple, quintessential "text-to-text" transform that Computer Science is best at doing. If I can't do the conversion for an entire song in under 100ms, I'll be disappointed in myself. That's enough time to send a network packet to the other side of the world. That's long enough to read the average MIDI file off the built-in flash drive on a tablet six thousand times. Something like two billion clock cycles (across all cores of the average iPad) occur in that interval. It's staggering how much we waste these days.
This is hopefully how things will proceed. I've long since learned that trying to sit down and do all of the work at once is non-starter for me. Instead, I've broken things down and am hoping to mete out small improvements to the sheet music across each (hopefully more regular) update. The initial "hey, you can load a MusicXML file" update might just wire the notes found in the file directly to the MIDI loading code, showing none of the benefits at first... but it'll be a big step in the right direction, leaving me ready to take the next step that you just outlined.JimNYC wrote: ↑04-06-21 11:00 am... but just drawing the notes together as intended would make it significantly more usable than it is now for actual note reading. So if it's actually easier to just extend the internal format to maintain note positioning/clef changes and the glyph routines can just use that, then it would be a massive improvement.
That looks like a much more serious effort. (I didn't have to play the "Where's the code?" game this time.) Now I'm going to sound like I'm being picky: when it's not in the same language (TypeScript instead of Synthesia's C++), there are some calculations that need to be made about how much effort it might take to incorporate a TS/JS interpreter or possibly do a rewrite in the target language by hand. (Coming from a strongly-typed, imperative language like TypeScript, this might actually not be so bad.) Even if I do neither of those things, at a minimum, it looks like it will serve as a good reference if/when questions come up.
Thanks for the link!
Assuming I can get the speed anywhere in the same ballpark as my bluster above, this should (hopefully) be a non-issue. Even if it ended up being a 1-3 second process, that still feels "correct" inline, inside the app. Much longer than that and it might make sense to turn it into a standalone tool.
Doesn't a MusicXML file basically have two halves, a this is how its played and a this is how its drawn part? If so, is the this is how its played part close to being a MIDI file?The initial "hey, you can load a MusicXML file" update might just wire the notes found in the file directly to the MIDI loading code, showing none of the benefits at first... but it'll be a big step in the right direction, leaving me ready to take the next step that you just outlined.
I think wiring in some use of MusicXML that is only as capable as a MIDI file soon would have the benefit of allowing users to start building up libraries of MusicXML files. Having users with substantial MusicXML libraries will help in testing the additional features of MusicXML as they are added.
I actually... completely agree. Just typing the above scenarios made me wonder why my original plan had MusicXML coming in so late in the process. It wasn't going to be "perfect"/"100% complete" at that point either, so if I'm already going to be improving it on-the-go, why not get it in there much earlier?
Lacking a better answer, I just pulled the half-dozen lines covering "bare minimum MusicXML parsing/loading" from around line 730 up to line 46. That will be in the very next update.
I might take you up on the offer, actually. A million years ago, when I was first developing Synthesia, MIDI files were (and continue to be) very easy to come by. While I'm sure I could spend some time dredging up lots of MusicXML files, saving some of that effort sounds good!
They'd be used strictly for internal development purposes only and would never be redistributed to anyone else. Send whatever you'd like to support@synthesiagame.com and hopefully I'll be able to use them during testing. (I'd prefer a link to some zipped up archive on some cloud service or file sharing site; although directly attached files would work in a pinch, too.)
Thanks!
Where do you stand with regard to MusicXML versions of the included files from the DeBenedetti G Major site?
I just sent some files in MusicXML, PDF and midi formats. There is a large repository of MusicXML files at MuseScore.com. Searching for "PianoXML" will come back with 100+ piano files available in MusicXML/PDF/midi format, and these are all accessible with a free account.
However, looking at the files and how Synthesia renders the sheet music from the midi, I think two things that would improve things in the current rendering (without anything to do with MusicXML) has to do with the hands and what clef Synthesia draws them on. Most time, the left and right hand parts are separate hands that can be adjusted in the Hands, Colors and Instruments section. However, when both hands are drawn as sheet music, the left hand will cross into the treble clef, rather than drawing them as higher/lower notes on the bass clef (and vice versa). The two things that would improve sheet music readability from midi files is:
1) keep left hand and right hand separated on each respective clef (or make this an option) rather than drawing the left hand part into the treble clef and the right hand part into the bass clef
2) when #1 is enabled, detect some interval whereby it would draw a clef change into the music when the notes would become ridiculous to draw above the clef lines.
See the Schumann Op 68, No 7 I sent for examples of these in measures 4 and 9. Those 2 things alone, without MusicXML itself, would improve sheet music readability tremendously.
However, looking at the files and how Synthesia renders the sheet music from the midi, I think two things that would improve things in the current rendering (without anything to do with MusicXML) has to do with the hands and what clef Synthesia draws them on. Most time, the left and right hand parts are separate hands that can be adjusted in the Hands, Colors and Instruments section. However, when both hands are drawn as sheet music, the left hand will cross into the treble clef, rather than drawing them as higher/lower notes on the bass clef (and vice versa). The two things that would improve sheet music readability from midi files is:
1) keep left hand and right hand separated on each respective clef (or make this an option) rather than drawing the left hand part into the treble clef and the right hand part into the bass clef
2) when #1 is enabled, detect some interval whereby it would draw a clef change into the music when the notes would become ridiculous to draw above the clef lines.
See the Schumann Op 68, No 7 I sent for examples of these in measures 4 and 9. Those 2 things alone, without MusicXML itself, would improve sheet music readability tremendously.
I think this idea has considerable merit. Even after MusicXML is fully supported, there will still be considerable interest in a sheet music display from MIDI files. I think JimNYC proposal in pseudocode is this:The two things that would improve sheet music readability from midi files...
Code: Select all
if MIDI_file {
if Hand_info {
if Right_hand {
//Display on upper clef with logic to shift clef range as needed
} else { // Left_hand
//Display on lower clef with logic to shift clef range as needed
}
} else { //No Hand_info
Current MIDI file sheet music display processing
}
} else { //MusicXML_file
MusicXML processing
}
Clef shifting would probably be useful for display of MIDI files without hand information as well although it would limited to things going off the top of the treble clef or the bottom of the bass clef, since I don't imagine you'd want to bother with trying to deduce situations that could use two treble or two bass clefs rather than the grand staff.
I don't have any of them yet. This very first "MusicXML loads as not much better than MIDI" pass probably wouldn't be the right time to switch them all over. There is a small (but non-zero) chance that, ironically, Synthesia might be able to interpret things from MIDI more effectively than from MusicXML.
Another reason to drag my feet on the switch-over is because it will effective reset everyone's song practice progress on all the built-in songs (because they're not the exact same file). I intend to keep shipping the "old" MIDIs forever, as a disabled-by-default song entry. Something like "Legacy Built-in Songs". That way if someone has years of statistics and progress, it won't all be lost.
So while I'll need to convert them eventually, there isn't any time pressure for this release.
Thanks, I received them!
Separate staffs for each hand (and inline clef changes + 8va/8vb that will be required for them) were part of the plan for MIDI-generated sheet music. Now they'll just be after the introduction of MusicXML loading instead of before.
To sort of loop back to the original thought that JimNYC had, do YOU need to convert these files? Or is this something that can be done as a community project, like the finger hints were? Regardless of who does it, it is going to be time consuming. Seems like something that should be started sooner rather than later.So while I'll need to convert [the DeBenedetti files] eventually, there isn't any time pressure for this release.
Maybe you should leave the current MIDI DeBenedetti files as the default and make using the MusicXML files an option for those who want good sheet music rather than score history.
You'll get much better beaming (and, uh... "staffing"? ) from MusicXML files much sooner than you would have from MIDI. And the conversion in most of the apps that can handle both formats (MuseScore, etc.) should be just a couple clicks. (And that way you'll even get a chance to manually fix things, which you'd never get with Synthesia.)
Technically, no, I don't have to do them myself. Although in the intervening many years I've started to feel a little strange about unpaid work from volunteers. The project is... rather healthy, so accepting free labor has begun to feel a little exploitative. For language translations, it's less awkward because it's a task I simply cannot do myself. But when I'm "buying" my MIDI-to-MusicXML conversion time back for $0, well, that's the definition of getting something for nothing.
Who's got a PayPal account that wants to convert songs?! I could put a bounty on each one or something. As long as it stays under the reporting threshold for my side (which is what, $600?) that should keep the arrangement super simple.
Automatic machine translation will be a good first step (though, I'd start with the MIDI versions that ship with Synthesia, as they've already been quantized and had the left/right hands split into separate tracks; that should lead to much better initial conversion results).
Though after that, the next step is a meticulous comparison with the original sheet music PDFs to make sure the final conversion contains as much of the original musical intent as possible.
Though after that, the next step is a meticulous comparison with the original sheet music PDFs to make sure the final conversion contains as much of the original musical intent as possible.