What does it mean to "learn" a song?

Archived development update discussion from past versions
Archived development updates.
Nicholas
Posts: 12430

Post by Nicholas » 02-09-14 8:01 pm

Nicholas wrote:The Advanced / Simple Song View paradigm is changing. I think I've come up with a neat way to combine a lot of ideas that never really worked on their own into something much simpler. The notion of "Played by Me/Synthesia" and of "Left/Right/Both Hands" will be merging. And the new splitting interface will give you incredibly fine-grained control over all of it.
I've mentioned this a few times now. Even though it's still just some sketches on paper, they just went through something like their third evolution and the changes are even more radical than ever before. (Seriously, the Song setup screen is going to be an order of magnitude simpler... without losing any capability, and only gaining more flexibility. Bold claims!)

The scope of the changes is starting to get big though. Like... the current notion of "scoring" along with all the "song progress" stuff will now also be uprooted by this process. The reason I'm taking the time to mention it here is because I'd like to solicit feedback. It's come up before and I'm not sure there is a perfect answer, but I'm wondering:

How would you measure/define what it means to do well (or to "learn") a song?

Is it playing the entire thing at "full" speed without mistakes using a consistent set of fingerings that doesn't change between plays? How many times does that need to be accomplished until you've proven a particular play wasn't just a fluke? What role does melody (correct pitches) vs. rhythm (correct timing) play? (In a perfect world, Synthesia could allow rubato. Certainly a master pianist doesn't stick to the tempo like a metronome. Should Synthesia require it then?)

Here is a scenario involving two songs, each with 50 measures:
  • You play Song A one hundred times, each time making a single mistake in a random measure.
  • You play Song B one hundred times, each time making a single mistake at the same point in measure 47.
Which song do you know "better"? If you had to assign a metric to both, would they be 98% and 98%? Would they be 98% and ~70%? Why?

No promises, but there is a chance picking which hand you want to play might be getting pushed right to the play screen(!), meaning you could change it in real-time during practice. This assumes a more reasonable world than Synthesia currently does where it's normal to loop on sections until you get things right, trying it hands-together and hands-apart during that process.

I'm not sure the video-game notion of rhythm practice "points" makes any sense anymore. The direction I am taking Synthesia (my mission statement, if you will) is "the best piano practice tool available". I don't think assigning points to a single beginning-to-end playthrough of a song serves that goal. In fact, I'd like each iteration of those loops to count toward this "progress" metric... for that particular section of the song.

That is, I'm considering tracking this elusive "progress" on a per-measure basis(!) stored across ALL of your plays, including every loop in each of those playthroughs(!!). That seems like more valuable data to me. This change means Synthesia can start to make a distinction between things like "you know the first section of this song really well (and the rest not at all)" vs. "you know all the sections of this song fairly well". (Or, broken down one more level: "you've got the left hand part down-pat in the first section, but the right hand could still use a little work".)

The goal here is to try and get Synthesia to reflect reality a little better, strip out the superfluous metrics, but keep (and strengthen!) the motivational/gamification elements that make playing songs in Synthesia so satisfying today.

This is a great upheaval. I am interested in radical high-level ideas. I'm not so worried about implementation details at the moment. More of a "if we started from scratch, how could we do it better"? What questions do you have that Synthesia could help you answer with data? And which feedback could Synthesia offer that would help guide you the best? Let me hear what you think. Thanks!

Raymond
Posts: 528
Location: United States, Washington

Post by Raymond » 02-10-14 1:35 pm

I think it's when it sounds about* like it should, in both melody and rerical modes. *By 'about', I mean like 95%-99%, 100% is not needed and can remove some 'feeling'.

The scenario:
Song A is 70% Song B is 98%, because Song B is a consistent mess up.(You Know where you are messing up, and it can easily be fixed.) Where Song A it's unknown where you are messing up. (How well do you know the song, are you just going through it?)


Although I hate to lose all my current scores, I fully under stand the need to reset them. (not enough data)

Try not to forget non Piano paces. ex: Base on left hand, Flute on Right, Strings, Piano, etc. in background, and sometimes I would want to play Piano, or maybe the Strings.
Learning songs like that is different per song, some it's playing all at once, others it's playing a certain group(s) well. (Playing the Bass+Flute and Piano but ignoring drums/strings. Note that it's different for different songs.)

jitsua
Synthesia Donor
Posts: 12

Post by jitsua » 02-10-14 5:09 pm

Hmm. Ultimately all I want to know is did I play better/worst than previous playback history. Ideally the 'how' this was measured should be obvious/transparent.
So, if you have this performance data then how about an option to show the player some additional feedback. i.e. revisit the piano roll with added annotation

* Block out sections that need improvement due to missed notes/wrong notes/bad timing (w/option to rehearse each section)
* Ghost the players notes/with actual expected notes
* Indicate notes that were hit too hard/soft (red for too hard, blue for too soft?).
* Suggest pedal usage/mistake areas.
* etc...

also maybe add a game mode where increasing number of notes are removed from the piano roll (the better you play) and you have to remember what the next notes are :evil:

User avatar
jimhenry
Posts: 1812
Location: Illinois

Post by jimhenry » 02-10-14 6:27 pm

I think a concept of the ability to perform a song consistently over a period of time is needed to demonstrate that a song is learned. Perhaps only the best score within a period of 24 hours should be kept for long term tracking. Maybe 3 consistent performances with some period of time between should be required to confirm that a piece is learned.

Whatever, you do, I think Synthesia is currently too lenient in scoring. The distance between F and A is too small and A is too easy to achieve. I consider an A grade to mean barely acceptable, and usually that is a generous assessment. I understand that you don't want to be so harsh that it is discouraging. But there needs to be room to reach, grow, and feel a sense of accomplishment for hard work at the top end. It's OK for C to mean you have a basic grasp of the song. B can be achievable with modest effort. But A should require real mastery. A+ and beyond should only be given for exceptional performances.
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/

TonE
Synthesia Donor
Posts: 1180

Post by TonE » 02-12-14 3:34 am

Nicholas wrote:Here is a scenario involving two songs, each with 50 measures:
  • You play Song A one hundred times, each time making a single mistake in a random measure.
  • You play Song B one hundred times, each time making a single mistake at the same point in measure 47.
Which song do you know "better"? If you had to assign a metric to both, would they be 98% and 98%? Would they be 98% and ~70%? Why?

That is, I'm considering tracking this elusive "progress" on a per-measure basis(!) stored across ALL of your plays, including every loop in each of those playthroughs(!!). That seems like more valuable data to me. This change means Synthesia can start to make a distinction between things like "you know the first section of this song really well (and the rest not at all)" vs. "you know all the sections of this song fairly well". (Or, broken down one more level: "you've got the left hand part down-pat in the first section, but the right hand could still use a little work".)

The goal here is to try and get Synthesia to reflect reality a little better, strip out the superfluous metrics, but keep (and strengthen!) the motivational/gamification elements that make playing songs in Synthesia so satisfying today.

I am interested in radical high-level ideas.
Allowing rubato playing is great. How? Measuring rhythm NOT absolutely, as it is now, but relatively. So we do not care any more, when you play exactly, but the relative difference between inter-onset-intervals = time differences between successive notes. This would allow rubato style playing, too. You could give even some kind of 'swing score'. :-) Measuring the average swing value, per measure, then plotting it as another 'swing curve' or rubato curve.

Same for velocity. Now we do not measure this at all. Having this feature, you could give extra points for correct relative velocity changes. No matter, which absolute values they have. If current measures velocity is v0, next measures velocity v1, you can expect sometimes v1 > v0, or the opposite, depending on the score. Then it is possible, to check for every time point, against the next time point, if you made the correct 'velocity transition'. Here some 'velocity threshold' of some value 3..10 might help. In short this is for measuring relative loudness playing. With rubato we had relative timing playing.

What else? I like jitsua's idea, with disappearing more and more notes from piano roll, the better you play. So you have to remember more notes yourself. Great idea.

The other point you/Nicholas mentioned, I agree, measure based scores I like more, this is why currently the solution for bigger songs is simply, splitting them into multiple midi files, then practicing and scoring them separately. Like loops, but you define some part of the song as a new loop, but in a song. :) Having hierarchic loops. The entire song is the first loop, the loop inside the song, is the second loop. Loop in the loop. If you can do something 2 times, you can do it n times, too. Where to stop? When the loop length reached the shortness of 1 measure/bar. This is why measure based jumps are important, as we know and do not have yet. :shock:

What else? Realtime error output into the clipboard, I asked for a long time. Why? We could play around with various ideas, possibly coming up with something new, unexpected, similar to current Free Play, which is unexpectedly powerful. Realtime rubato error. Realtime velocity error. Realtime missing notes error. Realtime wrong notes error.

Answering your above question, of course you know song B better. Why? The errors are in same space/place/measure. For song A there is no 'space quality guarantee' as for song B. Number of correct played measures per song, would be a useful quality measure. Hamming distance could be used here. Consider the entire songs measure sequence as bit sequence, when there is an error in a measure make this bit 1, no matter how many errors. The 'hamming distance' would tell immediately which song is played better after P playings. Of course you have to use 'integrated hamming distance'.

Examples, from above:

song A: 50 playings, integrated hamming distance is more than 1, as the errors sit in different measures. If they would sit in 10 different measures, your integrated hamming distance would be 10. This would tell us: You are making errors in 10 different places/measures.

song B: 50 playings, integrated hamming distance is 1.

Easy to measure, easy to calculate, easy to analyze, easy to save, bits are fun.
Last edited by TonE on 02-12-14 3:54 am, edited 1 time in total.

TonE
Synthesia Donor
Posts: 1180

Post by TonE » 02-12-14 3:35 am

Raymond wrote:Although I hate to lose all my current scores, I fully under stand the need to reset them. (not enough data)
Save your current .xml files somewhere.

TonE
Synthesia Donor
Posts: 1180

Post by TonE » 02-12-14 4:08 am

TonE wrote:What else? Realtime error output into the clipboard, I asked for a long time. Why? We could play around with various ideas, possibly coming up with something new, unexpected, similar to current Free Play, which is unexpectedly powerful. Realtime rubato error. Realtime velocity error. Realtime missing notes error. Realtime wrong notes error.
Yes, but how to implement it? One suggestion, as follows: Write in real-time into clipboard error codes, E0..En.

Realtime rubato error = E0
Realtime velocity error = E1
Realtime missing notes error = E2
Realtime wrong notes error = E3
Realtime too short note duration = E4
Realtime too long note duration = E5

That's it. Simple and powerful. After playing a song through, every second of the song can have either no E0..E5, or one or multiple E0..E5.

deed02392
Posts: 38

Post by deed02392 » 02-14-14 6:32 am

For me after using Synthesia to learn instinctively the original tempo and volume, after providing an easy way for me to see where the notes are on the keyboard (much quicker than analysing sheet music), I'd love the ability to start memorising notes.

Could we get a feature where Synthesia randomly hides notes as you're playing, encouraging you to remember them or something?

Raymond
Posts: 528
Location: United States, Washington

Post by Raymond » 02-20-14 5:29 pm

TonE wrote:
Raymond wrote:Although I hate to lose all my current scores, I fully under stand the need to reset them. (not enough data)
Save your current .xml files somewhere.
What I mean by lose is have Synthesia no longer use them. Ex when the part that generates the Hashes changes, the scores are all still 'there' but they are no longer linked to any songs.
And even if just the scoring system changes, the scores still appear, but they are no longer relevant because they where calculated differently.

Nicholas
Posts: 12430

Post by Nicholas » 02-20-14 7:02 pm

I agree it will be unfortunate to lose all the legacy data... I just don't see a way to get from here to there while maintaining it.

I haven't decided on any specific implementation yet, but I'm thinking that at least per-measure information will be kept for every play session. At the most I'll go the PianoCheetah route of saving every note of every performance. Though for my purposes, that might be an overwhelming amount of data to try and distill into useful metrics. Per-measure data feels like a nice in-between metric.

At any rate, I'll get to remove that annoying "progress will be lost" prompt because I'm leaning toward counting any practice -- even an incomplete song -- will work toward "knowing" that song. So quitting a song early doesn't do anything bad. All progress is always saved, period.

TonE
Synthesia Donor
Posts: 1180

Post by TonE » 02-21-14 2:56 am

Error categories E0..E5 from above together with 'hamming distance method per measure per error category' could work nicely.

Example: We have error categories E0..Ex. For each error category a string of zeroes will be written, one zero per measure, then the zeroes will be counted up if there is an error in that error category in that measure. Result for 3 measure song, for example: 004. Depending if you are interested in the absolute value of '4' or not you can treat it later differently. Absolutely as exact 4, or relative information like non-zero, treating it as '1'. Both treatments can be useful in different scenarios.

Hash construction could depend on other criteria, too. For example not exact note onsets, but instead note order. So a little timeshift to any direction, but still keeping the note order would not change the hash. Similarly for pitch order. One needs to think about possible negative side effects when using these approaches.

TonE
Synthesia Donor
Posts: 1180

Post by TonE » 02-21-14 3:08 am

Saving every performance in the same song.mid directory is an interesting idea. I would use per day time tagged zip files per song. Example: 2014-02-21-song.mid.zip

Later, no matter how the score system changes, new scores can be easily recalculated, one can experiment with different ways of scoring. Even the error categories can be adjusted, we have the complete performance results anyway. If some directories get a few kb bigger in size, who cares in 2014?

For midi analysis, as a side comment, Keykit by Tim Thompson has a great feature of being able of writing 'text information' per note! All error categories can be written there, for example. Later using those information and calculating some score.

See in Keykit, Language Reference Manual, Note and Phrase Attributes
attrib
This string-valued attribute of a note can be used to store arbitrary user-defined information. To save memory, this feature may not be enabled on all versions of KeyKit.
You could save here any key=value chain, separated by commas. Something like E0=0,E1=3,E2=0,...,Ex=0

Or better saving only the non-zero cases.

Pianotehead
Posts: 321

Post by Pianotehead » 02-21-14 5:18 am

Lately I've been trying out other notation programs, MIDI Sheet MUsic, MIDI Illustrator and eMedia, but have always turned back to Synthesia, so it must be good. Though, the sheet music display needs some plastic surgery. ;)

Rubato? Is it possible to code it, without letting mistakes in? How can you distinguish between bad timing or playing, and rubato? The scoring has never been important to me, I usually know whether my playing sucks or not. Just by noticing the stops, hesitations and when not hitting the right notes. When I record myself, the mistakes become obvious, of course.

Nicholas
Posts: 12430

Post by Nicholas » 02-21-14 9:56 pm

Pianotehead wrote:Though, the sheet music display needs some plastic surgery. ;)
Yeah, yeah... :lol:
Pianotehead wrote:Rubato? Is it possible to code it, without letting mistakes in?
By using a technique called score following you can allow rubato.

Unfortunately the Home Concert Extreme folks still have a patent on the method for another 2 or 3 years. Then there are the dozens of subsequent patents citing theirs, like this incredibly groundbreaking one that doesn't expire until 2030. Even Apple cited the original score following patent for their Garageband lessons patent.
TonE wrote:Later, no matter how the score system changes, new scores can be easily recalculated, one can experiment with different ways of scoring. Even the error categories can be adjusted, we have the complete performance results anyway.
This is actually a pretty good point. Having a complete set of data means I never have to obsolete scores again. Even if the full performances are a little unwieldy to work with, there is no reason I can't also keep the boiled-down per-measure versions at the same time.

And you're right: MIDI compresses really nicely (something like 1/6 size), so that'd be a good way to go.

... although, it might be a little tricky correlating the recording back to the song. The new system I'm envisioning doesn't require today's assumption of exactly one, linear play-through. Ideally if you spend 20 minutes looping smaller sections and practicing very hard, you should get much more credit than a single play-through. I guess each time you seek to a different place in the song, the recording could have a little text event added stating the corresponding position in the original song.

TonE
Synthesia Donor
Posts: 1180

Post by TonE » 02-22-14 4:13 am

Nicholas wrote:I guess each time you seek to a different place in the song, the recording could have a little text event added stating the corresponding position in the original song.
Yes, m1..mX, like my proposal for synchronizing score pdf pages to Synthesia. Btw. there is a related technical term, in the DAW world or audio editing world, called '4 point editing'. Check youtube for a few videos if you want. Those jumped and measure based cutted and appended, recorded results, would be like '4 point editing results'. Or it is called 'source destination editing'. You select some destination time region which you want to replace, find some source area which you like, press a button, finished. Finally, this is called 'classical editing', too. For video film editing or real classical music recording editing. When you have longer chunks of compositions which you do NOT micro-cut, but cut and paste in bigger sections, using a 30ms crossfading, 15ms to both directions from the cut point.

TonE
Synthesia Donor
Posts: 1180

Post by TonE » 02-22-14 4:31 am

Another 'crazy idea' would be recording the practice performances as follows, generating let us say, 1 hour compositions:

- Assumption is our song.mid consisting of 100 measures, m1..m100.
- You practice m1..m10 on day1. Will be time and measure tagged appended into recording.mid
- You practice m1..m10 on day2. Will be time and measure tagged appended into recording.mid, later day2 follows older day1 recording. When you listen to recording.mid you will hear in one go, how you played the first time, second time, third time... possibly hearing your improvements over time.
- As those appended segments are day and measure based tagged, it is possible to quickly re-cut or '4 point edit' to generate a very specific cutted version. Like the best peformances from all performances. The score system will help in deciding which is the best. Another fun dimension. Who knows? And all would happen in the background as a side product of having fun in Synthesia. You want to patent this idea? Auto-cut-editing-of best performance results-into a new result. Then you can kick Apple with this patent, hohoho. :lol:

bluesdance
Posts: 16

Post by bluesdance » 02-25-14 11:34 am

I use the current scoring paradigm in my music class where I bring up 15 kids who each have a few turns playing. I keep track of each kid's score in a separate cell on a spreadsheet that I keep open in a separate window on my desktop. As you go into the brave future of scoring, please continue to think about those of us (maybe it's just me?) who use this software "Dance Dance Revolution" style with lots of different people in turn who need to be scored separately, in a situation where it's impractical to create separate user accounts for each player.

Nicholas
Posts: 12430

Post by Nicholas » 02-26-14 1:02 am

I am always very interested in hearing about Synthesia in the classroom. Could you tell me why you feel it's impractical to create separate user accounts for each student? Is it account creation that's too onerous? Or is it too cumbersome to switch between accounts? Too many students make the account list unwieldy? (All of the above? :D )

Assuming all of those problems are intractable, would an alternative "Clear All Progress" type button be enough to allow you a sort of quick-reset for each student?

TonE
Synthesia Donor
Posts: 1180

Post by TonE » 02-26-14 2:59 am

If I would be a teacher of a class, using Synthesia as a group, I would use an external tournament management software, like Sevilla, and start a tournament among the students. Start fighting. That is all about in life, fighting for money. :lol: In this case, fighting for points, thus the first position at the end of the tournament.

Sevilla, was originally created for managing chess tournaments, but has all kind of tournament settings so it can be used for any sports, soccer, basketball, volleyball, and so on. Synthesia can copy a score into its clipboard, from where you can transfer each score, automatically, too, to Sevilla. The tournament would develop in steps of round robin matches among the players, like in soccer. Very simple and most importantly fun.

What Synthesia can do, in my opinion, adding a simplistic 'Sevilla' into Synthesia itself, so this kind of organization gets easier, mainly for non-programmers. Now Synthesia has even a form of displaying score curves. What is missing is only, list of players, generation of round robin matches, 3 points for win, 1 point for equal, 0 points for loss. Go. After each match Synthesia would jump to 'league result view' showing all players positions, points, goals (score counts as goal, opponent-score in match counts as counter-goal). Another smaller list on the side, showing the next match players. One directory of 'tournament midis' from which they will be pulled randomly.

bluesdance
Posts: 16

Post by bluesdance » 03-04-14 12:59 pm

Nicholas wrote:I am always very interested in hearing about Synthesia in the classroom. Could you tell me why you feel it's impractical to create separate user accounts for each student? Is it account creation that's too onerous? Or is it too cumbersome to switch between accounts? Too many students make the account list unwieldy? (All of the above? :D )
Yes, all of the above. I'm not about to create 200 separate accounts and switch between them every time a kid sits down and another one comes up. There's just no benefit to me as the user/teacher.
Assuming all of those problems are intractable, would an alternative "Clear All Progress" type button be enough to allow you a sort of quick-reset for each student?
That would be INCREDIBLE and LOVELY. I would actually use it per class.

The main thing I want is a big flashy popup at the end of each round with the score information.

Also -- I really want to disable the letter grades.

Thank you for letting me know about Sevilla. It doesn't seem to be available on Mac.

In my case I manage scores manually in a spreadsheet. I like to tone down the individual competition a little bit by showing a class average score at the bottom, this helps keep the kids from killing each other and creates a shared goal of getting the class average up. Also I have 2 rounds and I show the % change which allows me to reward the kids who aren't naturally good at this the first time they come up.
Attachments
Screenshot 2014-03-04 13.02.24.png
Scoreboard
Screenshot 2014-03-04 13.02.24.png (49.17 KiB) Viewed 21272 times

Locked