Posted: 06-02-20 4:23 pm
Background
Currently, there are 3 guided play modes in Synthesia: melody, rhythm and song recital. The last one is actually just a restricted rhythm practice, so for further discussion we can reduce it to just the first two.
Both modes impose certain restrictions on the performance (i.e. conditions such that even slight deviations from them is punished extremely heavily).
In melody practice, there are 2 restrictions: all notes should be played exactly in the order of the midi, and the speed cannot exceed the midi's tempo at any time. (There's a hack for the last one, though - you can play, e.g. at 200% speed.)
In the rhythm practice, the restriction of playing every note correctly is relaxed significantly to the point that it stops being problematic. However, a new restriction is added: the speed has to be constant all the way through.
In my experience, when I learn a piece, there usually comes a point when these restrictions make playing the piece with the app very painful. On one hand, I am proficient enough to skip over some minor mistakes (e.g. a skipped note or a wrong note played). The problem is this kind of thing is heavily punished in the melody practice mode: every such case turnes into 20 next notes considered to be wrong before you notice the problem, and then you have to go 20 notes back and lose the momentum. On the other hand, I am not proficient enough for the rhythm practice mode: there still are quite a few small pauses and slight tempo fluctuations, both of which are heavily punished there. Usually, in such situations I just give up using the app and play without tracking altogether.
Feature description
Since at this stage you are already mostly familiar with the piece, the main benefit of Synthesia would be (somewhat objective) progress tracking. After some thinking, I came to the idea of a "guided freeplay" mode as a possible solution: basically, you play the piece the same way as you would in freeplay mode, but get the scoring against the midi and progress tracking based on that score like in othe guided modes.
The scoring is probably the hard part of the feature. Obviously, the algorithms used in other modes aren't immediately applicable. My thinking is something along these lines: first, use some kind of LCS algorithm (https://en.wikipedia.org/wiki/Longest_c ... ce_problem) to match the midi with performance, then calculate the score based on how much we were able to match (which is somewhat equivalent to the number of mistakes), and the difference between the attributes of matched parts such as note duration or loudness. The exact details are probably something that's better to experiment with.
The UI could probably be exactly the same as in freeplay. It'd be good to have cues as to what notes are expected to be played next, but I don't think it's essential since you are mostly familiar with the piece anyway, and also it would probably be hard to implement real-time matching with the midi: it would likely be jumpy and have a small percentage of incorrect matches.
Currently, there are 3 guided play modes in Synthesia: melody, rhythm and song recital. The last one is actually just a restricted rhythm practice, so for further discussion we can reduce it to just the first two.
Both modes impose certain restrictions on the performance (i.e. conditions such that even slight deviations from them is punished extremely heavily).
In melody practice, there are 2 restrictions: all notes should be played exactly in the order of the midi, and the speed cannot exceed the midi's tempo at any time. (There's a hack for the last one, though - you can play, e.g. at 200% speed.)
In the rhythm practice, the restriction of playing every note correctly is relaxed significantly to the point that it stops being problematic. However, a new restriction is added: the speed has to be constant all the way through.
In my experience, when I learn a piece, there usually comes a point when these restrictions make playing the piece with the app very painful. On one hand, I am proficient enough to skip over some minor mistakes (e.g. a skipped note or a wrong note played). The problem is this kind of thing is heavily punished in the melody practice mode: every such case turnes into 20 next notes considered to be wrong before you notice the problem, and then you have to go 20 notes back and lose the momentum. On the other hand, I am not proficient enough for the rhythm practice mode: there still are quite a few small pauses and slight tempo fluctuations, both of which are heavily punished there. Usually, in such situations I just give up using the app and play without tracking altogether.
Feature description
Since at this stage you are already mostly familiar with the piece, the main benefit of Synthesia would be (somewhat objective) progress tracking. After some thinking, I came to the idea of a "guided freeplay" mode as a possible solution: basically, you play the piece the same way as you would in freeplay mode, but get the scoring against the midi and progress tracking based on that score like in othe guided modes.
The scoring is probably the hard part of the feature. Obviously, the algorithms used in other modes aren't immediately applicable. My thinking is something along these lines: first, use some kind of LCS algorithm (https://en.wikipedia.org/wiki/Longest_c ... ce_problem) to match the midi with performance, then calculate the score based on how much we were able to match (which is somewhat equivalent to the number of mistakes), and the difference between the attributes of matched parts such as note duration or loudness. The exact details are probably something that's better to experiment with.
The UI could probably be exactly the same as in freeplay. It'd be good to have cues as to what notes are expected to be played next, but I don't think it's essential since you are mostly familiar with the piece anyway, and also it would probably be hard to implement real-time matching with the midi: it would likely be jumpy and have a small percentage of incorrect matches.