Guided freeplay mode

Synthesia is a living project. You can help by sharing your ideas.
Search the forum before posting your idea. :D

No explicit, hateful, or hurtful language. Nothing illegal.
Post Reply
Posts: 1

Post by urusant » 06-02-20 4:23 pm


Currently, there are 3 guided play modes in Synthesia: melody, rhythm and song recital. The last one is actually just a restricted rhythm practice, so for further discussion we can reduce it to just the first two.

Both modes impose certain restrictions on the performance (i.e. conditions such that even slight deviations from them is punished extremely heavily).

In melody practice, there are 2 restrictions: all notes should be played exactly in the order of the midi, and the speed cannot exceed the midi's tempo at any time. (There's a hack for the last one, though - you can play, e.g. at 200% speed.)

In the rhythm practice, the restriction of playing every note correctly is relaxed significantly to the point that it stops being problematic. However, a new restriction is added: the speed has to be constant all the way through.

In my experience, when I learn a piece, there usually comes a point when these restrictions make playing the piece with the app very painful. On one hand, I am proficient enough to skip over some minor mistakes (e.g. a skipped note or a wrong note played). The problem is this kind of thing is heavily punished in the melody practice mode: every such case turnes into 20 next notes considered to be wrong before you notice the problem, and then you have to go 20 notes back and lose the momentum. On the other hand, I am not proficient enough for the rhythm practice mode: there still are quite a few small pauses and slight tempo fluctuations, both of which are heavily punished there. Usually, in such situations I just give up using the app and play without tracking altogether.

Feature description

Since at this stage you are already mostly familiar with the piece, the main benefit of Synthesia would be (somewhat objective) progress tracking. After some thinking, I came to the idea of a "guided freeplay" mode as a possible solution: basically, you play the piece the same way as you would in freeplay mode, but get the scoring against the midi and progress tracking based on that score like in othe guided modes.

The scoring is probably the hard part of the feature. Obviously, the algorithms used in other modes aren't immediately applicable. My thinking is something along these lines: first, use some kind of LCS algorithm ( ... ce_problem) to match the midi with performance, then calculate the score based on how much we were able to match (which is somewhat equivalent to the number of mistakes), and the difference between the attributes of matched parts such as note duration or loudness. The exact details are probably something that's better to experiment with.

The UI could probably be exactly the same as in freeplay. It'd be good to have cues as to what notes are expected to be played next, but I don't think it's essential since you are mostly familiar with the piece anyway, and also it would probably be hard to implement real-time matching with the midi: it would likely be jumpy and have a small percentage of incorrect matches.

Posts: 35

Post by miek » 06-03-20 2:07 pm

This describes my experience too. Please may I add to your idea a possibility of scoring yourself against your own performances of the past? A bit like what you get on loop mode (dropped notes and time taken) and kind of a "uniformity" i.e. the difference between the slowest speed and the fastest speed you have played in any one performance - this would encourage you to learn the difficult bits a bit better.

Posts: 37

Post by thiagolr » 06-09-20 12:25 pm

I agree that there is a huge gap between melody and rhythm modes!

Posts: 12430

Post by Nicholas » 06-11-20 5:49 am

So, score following is a super cool technology whose patents finally expired a few years ago.

This video (starting at 6:13) shows how well this can work to follow your tempo automatically and ignore mistakes (within reason) in a robust way.

Would a feature like that in Synthesia solve this problem sufficiently well?

Posts: 37

Post by thiagolr » 06-13-20 11:23 pm

Yes, I think it would be a very good solution!

Post Reply