Automatic Fingering Prediction 0.1ish

Synthesia is a living project. You can help by sharing your ideas.
Search the forum before posting your idea. :D

No explicit, hateful, or hurtful language. Nothing illegal.

Postby Frost » 07-23-09 8:42 am

Hey everyone,
As I mentioned before, the first working version of the finger prediction system is done. This is NOT a usable version, it will be quite long before it can be used independently or in synthesia. This is for testing and parameter optimization; I need your feedback to correct and improve the suggested fingerings. Do not expect miracles, it's pretty basic right now.
That being said, fingerings found by the method were much better than I expected. As long fragments I only tried bach's minuet in g and mozart's alla turca, they were mostly spot on with very slight differences than the one I use/shown in the books. With slight optimization of parameters/methods, it will find the exact fingering. Depending on the piece, your mileage may vary.

Download at the bottom of this post. Software requires .NET framework 2+, but if you can run synthesia there should be no problem.
Attachments
AFP_0.1.rar
AFP 0.1
(24.69 KiB) Downloaded 1206 times
Frost
 
Posts: 51

Postby Frost » 07-23-09 8:43 am

So, let's run the program. If you can bear with the horrible GUI, this is what you should see.

The notes (numeric) and notes (alphabetic) parts are the main inputs (1 & 2).
Numeric input takes numbers, 1 being A1, 2 being A1#, 3 being B1,.. and 88 being the rightmost key in piano. (actually it's not limited, if you ever want to play the 21427th key :) ) Apart from those, zero means "Rest". I will talk about rests in a minute.
Alphabetical input takes notes such as A, B, A1, B1#, C3. "R" means rest.

As a test, enter "C1 D1 E1 F1 G1 A2 B2 C2" for the C scale. Press calculate. Ta-da!

I need your help for further improvements. What I expect is that, you should enter a short (8 to 30? note) melody, check the output, and play with all of the numbers around the GUI to improve the suggested fingering into a better one. Then, if it gives good results post your parameters, or suggest how it can be made better, or if there are obviously faulty suggestions etc. Please make your suggestions on the basis that whether the automatic fingering is usable or not; the fingering may be different than what you expect, however it can still be a valid, easy fingering. Please read the helps (plus signs, label 7) in the program to see which parameters do what. Also check whether the notes, octaves are correct before dismissing the prediction as faulty. Sometimes a wrongly placed "G4" after "C4 B4 A4" instead of "G3" will be the cause of problems, or it may just be me :)

RESTS! if you enter a slightly longer fragment, rests *may* help the calculation of the best fingering: during rests you would have time to readjust your hands/fingerings, thus the algorithm would not penalize such behaviours. One rest symbol (0 or R) is a half rest, two consecutive rests would be full rests (half/full are arbitrary names). However during my tests, the software gave the same result with or without rests, thus it may not affect it much.

If you want, you can enter your expected fingering to the 3rd box, the algorithm will score your fingering as well. It can be useful while playing with the parameters to see how it affects the scoring.

If you go into the sub-optimal fingerings tab above (8), you can enter a short fragment up to 8 notes: the program will calculate ALL of the possible fingerings and show the top 100. If the good fingerings are near the top, then most likely playing with the parameters slightly would help it. Also, sometimes it finds the optimal fingering, however since many fingerings have the same score, it shows a different one at the first tab, so it's a good check.



For the curious, the program uses the methods presented in:
Parncutt et al., An Ergonomic Model of Keyboard Fingering for Melodic Fragments, 1997
Jacobs, Refinements to the Ergonomic Model for Keyboard Fingering, 2001
Kasimi et al., A Simple Algorithm for Automatic Generation of Polyphonic Piano Fingerings, 2007
plus some custom rules & modifications by me.
Attachments
afp.jpg
afp.jpg (225.17 KiB) Viewed 35728 times
Frost
 
Posts: 51

Postby Frost » 07-23-09 8:45 am

LIMITATIONS:

-No mute finger changes!
Mute/silent fingering changes are when you press down a key with a finger, then change the finger pressing that key without lifting the key itself. We don't allow such Indiana Jonesiness.

-No multiple keys with a finger!
While playing chords, some difficult pieces may require pressing 2 keys with the thumb due to large stretching of the hands. We only allow 1 finger --> 1 key

-Max of 5 keys played at once!
Corollary to the previous rule, yes, you can only play 5 keys with your hand at a given time.

-No cross-fingered playing
In a chord, leftmost finger would play the leftmost key and so on. Thus, no "2nd finger plays C1 3rd finger plays B1". Also see midi input part.

-Hand separations
This is the most limiting one. The software requires the input to be separated into hands, so no automatic hand separation as Nicholas predicted. The algorithm *can* be improved to calculate the optimal "which hand plays which part" in the future, however it would make the method much much more complex, would take a lot of time and its benefits would be not that great, given we will have the editor which will feature easy hand separation along with fingering prediction.


CURRENT LIMITATIONS that would not be present in further versions:

-Only monophonic input!
As you noticed, you cannot enter chords. However the method fully supports polyphonic (i.e. multiple notes at a given time) input, it's just the GUI thats limited to monophonic input, and for a reason. Polyphonic fingerings mostly depend on the same rules as the monophonic ones, thus I want to improve the current method to acceptable levels before going into polyphony.

-Only right hand!
Obviously it would support left hand fragments as well.

-No Midi input
Midi input would come with the editor and near the phase where the software gives good results, I assume. The hand channels in midi would be separate. Also, an important part is that the midi input should be more or less quantized. Separate consecutive notes can be sliiightly overlapping in the midi file. If not quantized properly, the algorithm assumes it's a chord, adam sandler look-a-like tries to play it with the given fingering and hilarity ensues. Especially during thumb passes (see cross-fingered playing), since the thumb may play a higher note than the passed finger.

-Constant hand size
Hand size would be variable. By changing the hand reach, different fingerings for kids or people with very small/large hands may be found.

-No manual control
I assume this prediction method is for the novices, more advanced players can deduct their own fingerings better. As it is, for use in synthesia, you can enter all of the notes manually, or make the software fill it automatically. However, since pieces will be shared and the weight of the metadata would be carried by the community, not the software, semi-automatic fingerings may be implemented. Instead of filling all of the data, the song curator would analyze the automatic prediction, see which parts can be improved, manually impose limitations (i.e. "the 3rd note in second bar should be played with finger 3"), and the method would find the best fingering with the given limitations. Process is repeated until you get satisfactory results.


FUTURE, most likely around the era of flying cars

-Parameter optimization
I'm thinking of a method to extract the parameters from pieces with known fingering. All of the papers refer it to as "future work" and noone actually tried it, it would be much harder compared to the fingering method (expectation minimization methods would not work with such high number of parameters), but still, maybe in the future.

-Neurological basis
This and all of the other fingering methods rely on the mechanical basis of playing, whether the hand can stretch from this to that and so on. However, pianists would use fingerings that they can do properly, this does not necessarily mean that it's the physically most comfortable. If you are extremely accustomed to a fingering that is similar to the given piece, you would want to reuse that fingering if you can, it would save from working on hand dexterity and coordination for that fingering. From a pedagogical viewpoint, a software that can analyze your previously learned songs and extract motifs that can be used in new songs where applicable would be much better (in theory). computationally it's possible, although practically it may be too hard to use to be of value.

-Pedaling suggestion
Automatic suggestion of pedaling where the piece gets too hard to play

-Song Difficulty Analysis
The fingering difficulty given by the method may be used in difficulty analysis of the song, although normalization would be required since difficulty score is linearly increasing with the song length.



so, that's all as far as I remember. Hopefully, some of you will review and point out bugs, problems, possible improvements etc.
Frost
 
Posts: 51

Postby Choul » 07-23-09 9:21 am

Question: Why do you start the counting on A? I mean you suppose the first A in the C-scale gets the number 2, why is that?

When I enter the C-scale one octave higher I get different fingerings, like this:

Input:
C2 D2 E2 F2 G2 A3 B3 C3

Predicted fingerings:
1 2 3 3 4 4 5 5

And it has to be the same as one octave lower:
1 2 3 1 2 3 4 5

Another test:
When my input is C2 C2 C2 (so, three times the same single note) I get as fingerings 1 1 1,
but when you need to play the same single note one after another, you use different fingers, like 3 2 1 for example.
Choul
 
Posts: 489

Postby Frost » 07-23-09 9:49 am

wow, that was fast.

Why do you start the counting on A? I mean you suppose the first A in the C-scale gets the number 2, why is that?

well that's due to technicality. was, actually. Now I can make the A0 A0# and B0 as separate and start the first octave from C, just haven't fixed it yet.

When I enter the C-scale one octave higher I get different fingerings, like this:

good thing you noticed! it seems there is a bug in hand position method 2, I will fix it asap. Meanwhile you can use method1.

When my input is C2 C2 C2 (so, three times the same single note) I get as fingerings 1 1 1,
but when you need to play the same single note one after another, you use different fingers, like 3 2 1 for example.

that's due to design. Pressing the same note with the same finger would minimize the hand movement (if that's all that you would play, I mean exception is if the piece continues, then pressing the same key with different fingers would shift the hand slightly in fluid motion, may be better), I don't think different fingers are required unless extreme speed is sought and 1 finger is just not enough. However, if you increase the same finger/same note penalty (right now it's 0), it would try to use other fingers as well. Try to play with the parameters, please submit your preferred parameters when you are more or less happy with it.
Frost
 
Posts: 51

Postby Choul » 07-23-09 10:21 am

Thanks for your quick answers, now I used Method 1:

Input: C2 E2 G2 E2 C2 B1 A1

Predicted fingering: 1 3 5 4 3 1 2
I have to make a stretch here from my weak pinky to my weak 4th finger (from G2 to E2)

Instead I would say: 1 3 5 3 1 2 1

But much of the fingerings depend on the direction the following notes will go, and that's hardly to predict with just a few notes for testing.
With the predicted fingering you end on B1 with your thumb, with my fingering you end on A1 with your thumb.
Choul
 
Posts: 489

Postby Choul » 07-23-09 10:45 am

Method 1 again, this is a broken chords exercise from ABRSM grade 1 (only going up):

Input: C2 E2 G2 E2 G2 C3 G2 C3 E3 C3

Predicted fingerings: 1 2 3 1 2 4 1 3 4 1

Instead they say (and I would use, because it's more comfortable for your fingers): 1 3 5 1 2 5 1 3 5 3

In stead of stretching only your fingers you move your hand to the left and right by folding and stretching your fingers (I hope you understand what I mean here, my English isn't very good, I'm sorry). In the predicted fingerings you make a stretch from your third finger to your fourth finger (from C3 to E3). I think it's better to prevent such stretches.
Choul
 
Posts: 489

Postby Choul » 07-23-09 11:00 am

Frost wrote:
When my input is C2 C2 C2 (so, three times the same single note) I get as fingerings 1 1 1,
but when you need to play the same single note one after another, you use different fingers, like 3 2 1 for example.

that's due to design. Pressing the same note with the same finger would minimize the hand movement (if that's all that you would play, I mean exception is if the piece continues, then pressing the same key with different fingers would shift the hand slightly in fluid motion, may be better), I don't think different fingers are required unless extreme speed is sought and 1 finger is just not enough. However, if you increase the same finger/same note penalty (right now it's 0), it would try to use other fingers as well. Try to play with the parameters, please submit your preferred parameters when you are more or less happy with it.


Input: C2 C2 C2 D2
Settings for C3: 7,00
Fingerings: 1 2 1 2

That's better than without the settings for C3 imo.
Choul
 
Posts: 489

Postby Choul » 07-23-09 11:44 am

Still testing:

Input: G2 E2 E2 F2 D2 D2 C2
Predicted fingerings: 3 2 2 3 2 2 1
Rule: Same finger/same note back to 0,00

I would use: 5 3 3 4 2 2 1

It's the fourth option from tab Sub-optimal fingerings, but I have no idea which rules I have to change to get those fingerings in the first screen?
Choul
 
Posts: 489

Postby Nicholas » 07-23-09 12:59 pm

Stickied.

I'll try this when I get home tonight. Very exciting.

Frost wrote:... we don't allow such Indiana Jonesiness.

Having had no formal piano instruction (ever), I didn't know this was frowned on. I also didn't know what it was called. That's also an excellent expression. ;)

I would love to help support this effort, by the way. If it means making a mini-app where you could drop a MIDI file on it and it would output notes in your format to a text file (or the terminal), I'd be happy to.

(Unfortunately, the same lack of formal piano training is also a major limiting factor to how much I can contribute...)
Nicholas
 
Posts: 11546

Postby Frost » 07-23-09 1:04 pm

The different fingerings with different octaves bug is fixed. During the bug hunt, I also noticed a very simple but serious bug, though I'm not sure how much it affects the fingering itself. I don't want to take any chances, the fixed version is below.

Also method2 seems to be defective right from the birth, so don't use that, (or use, maybe it will give better results, who knows?). I'm currently looking for ways to deduce the wrist position from the pressed keys. In a chord, it's easier and the current methods would hold well with their approximations, but in 1 note-1 finger scenarios (as we have right now), I'm at loss. The hand and fingers are very flexible, 2nd finger may press C2 and the wrist may easily be anywhere along a 5-6 key radius without any hand stress at all. How to approximate the wrist position from just one pressed key? Or, equivalently, how to find how much the hand must have moved between (finger F1 on N1) to (finger F2 on N2)? Ideas are welcome.
Attachments
AFP_0.1.01.rar
(25.87 KiB) Downloaded 820 times
Last edited by Frost on 07-23-09 1:21 pm, edited 1 time in total.
Frost
 
Posts: 51

Postby Frost » 07-23-09 1:20 pm

Having had no formal piano instruction (ever), I didn't know this was frowned on. I also didn't know what it was called. That's also an excellent expression.

:) well, I don't know about the frowning upon it, I'm pretty sure there are complex pieces that are impossible to play correctly and fluidly without employing that method. But as you advance to "those" pieces, I'm sure you'll find esoteric pieces where punching the keys are not only allowed but encouraged :) my reason was that it would complicate matters (both for the method and the user) for a very rare condition that would not be required >99.9% of the cases.

I would love to help support this effort, by the way. If it means making a mini-app where you could drop a MIDI file on it and it would output notes in your format to a text file (or the terminal), I'd be happy to.

wow, thanks. that would really help, actually. I also thought of this, but due to other complications (note quantization, hand separation, multiple track/instruments and so on..) I dropped the idea. If you can begin the editor, at least the main framework, you can first add the direct midi input and I can take the input assuming it's "good", then you/me/others? can add the midi quantization, track selection etc on the middle steps without much changes to the basic infrastructure.
By the way, I'm also quite the novice, so the feedback, opinion and ideas of professional/formally trained players are most welcome. so are the others'.
Frost
 
Posts: 51

Postby Nicholas » 07-23-09 1:24 pm

(Again, taken with a grain of salt because I don't have any idea what I'm talking about...)

Couldn't wrist position be "learned" over a few notes? You start with a 5-6 key radius like you mentioned, but then the next note informs it and narrows maybe to 3-4 keys. Eventually you should have a pretty good idea.

I suppose it depends what you're using wrist position for. It seems like it could be used as an input to get better fingerings. Maybe iteratively. The first time you run the algorithm, it gives you rough wrist positions. The next time you run it, you give it the wrist positions that came back last time, which help inform better fingerings (and new, tweaked wrist positions), etc.
Nicholas
 
Posts: 11546

Postby TonE » 07-23-09 2:24 pm

What is C1 for a midi pitch number in the range of 0..127 or is this unimportant?
TonE
Synthesia Donor
 
Posts: 1180

Postby BenJackson » 07-23-09 4:00 pm

You should look at lilypond both for input and output. If you supported their input format (which isn't THAT far from yours) there would be a lot of available source. Also, there are tools to convert midi to .ly and other formats (eg MuseScore can do that with a gui). Also, lilypond supports fingering annotations in the output so you could visualize your results better.

Also, if you want a simple piece with two notes to a finger, look at the end of Chopin Prelude Op 28 #7.

I'll try the program out tonight...
BenJackson
 
Posts: 38

Postby vicentefer31 » 07-23-09 4:46 pm

Hi, I hope this file can help you to test this software. It's Hanon Excercises (from 1 to 5) with the Notes and the Finger for Right Hand.
Attachments
Hanon Fingers Right Hand 1 to 5.rar
(30.57 KiB) Downloaded 829 times
Last edited by vicentefer31 on 07-24-09 4:50 pm, edited 1 time in total.
Picasso: I am always doing that which I cannot do, in order that I may learn how to do it.
vicentefer31
 
Posts: 899
Location: Madrid, Spain

Postby Nicholas » 07-23-09 5:19 pm

BenJackson wrote:... which isn't THAT far from yours...

I was wondering why it looked familiar. Still, the "THAT" is the key word here. The devil is in the details. ;)

I would much rather have Frost work on the fingering algorithm than a more advanced parser.

Evaluating the difficulty a little more fine-grained, you have four (very rough) cases:
  1. Parsing "simple" Lilypond input
  2. Parsing "difficult" Lilypond input
  3. Generating "simple" Lilypond output (with fingering annotations)
  4. Generating "difficult" Lilypond output (with fingering annotations)
Depending on the scope of what you were suggesting, Frost would have to do 1-3. #1 is almost what he has. #3 wouldn't take any effort at all and it'd be kind of neat to see.

#2 is the real sticking point.

The Lilypond scripting language can express *way* more than we're interested in (so far), so considerable effort would be spent pruning out things that weren't necessary and transforming the good stuff into usable stuff.

And, if you're only going to accept a special-case subset of Lilypond as input, it probably isn't worth making it match at all. You'll be hand-creating these files in either case.

I expect as Frost gets closer to finishing this, rather than taking text input, he might link it with some library (that I may or may not be the one to write) that would provide a richer representation of the underlying MIDI notes without having to do (or deal with) messy parsing.

For now, the GUI feels like a research interface. (Correct me if I'm wrong, Frost. ;) )
Nicholas
 
Posts: 11546

Postby vicentefer31 » 07-23-09 7:03 pm

Nicholas wrote: rather than taking text input, he might link it with some library (that I may or may not be the one to write) that would provide a richer representation of the underlying MIDI notes without having to do (or deal with) messy parsing.

Until someone mades this library, maybe we can use a "midi to txt" like you can find in http://www.midiox.com/
Attachments
02hanon.txt
(11.45 KiB) Downloaded 800 times
Picasso: I am always doing that which I cannot do, in order that I may learn how to do it.
vicentefer31
 
Posts: 899
Location: Madrid, Spain

Postby Nicholas » 07-23-09 7:12 pm

That's actually a pretty clean format.

A parser for that would boil down to:
1. Look for lines starting with a number followed by "On"
2. Skip the "ch=" part.
3. Read the "n=" part and convert it to a pitch.
4. Only keep the input if the "v=" part is >0 (some MIDI files have note-ons with zero velocity that actually means note-off).

Or, if you wanted to do it in a single regular expression:
Code: Select all
[0-9]+ On ch=[1-9][0-9]? n=([1-9][0-9]*) v=[1-9][0-9]*

And group 1 would contain your note number on a match.

I've seen some MIDI to text converters that weren't nearly that friendly. ;)

EDIT: I guess the MTrk stuff makes it a *little* less friendly. Especially because each one counts time from zero again. But, on the other hand, it does make separating out the left/right hands easier if the MIDI contains more than one track.
Nicholas
 
Posts: 11546

Postby TonE » 07-23-09 8:04 pm

Nicholas wrote:And group 1 would contain your note number on a match.
Then I do not understand why B2 equals to 15, as seen in the screenshot above.
TonE
Synthesia Donor
 
Posts: 1180

Next

Return to Feature Requests

Who is online

Users browsing this forum: No registered users

cron