How to get the duration of a song from scores.xml ?

Have questions? Just saying hello? This is the place.
No explicit, hateful, or hurtful language. Nothing illegal.
Post Reply
Andre-Louis
Posts: 2

Post by Andre-Louis »

Hello.

I wrote a program that takes the information from my scores.xml file and tallies up my total number of plays, average errors per song, average speed, etc... and I would like to add to this the total number of hours I’ve played. But, I haven’t figured out the relation between the duration of the song and any of the other variables in scores.xml. Does anyone know how I can calculate this?
Bavi_H
Posts: 116

Post by Bavi_H »

Hello, I took a peek in scores.xml, but so far I can't detect anything that matches the song duration exactly.

(Note: When I refer to the "song duration", I mean the duration from the beginning of the first note to the end of the last note, which is what Synthesia uses as the duration of the song. Synthesia doesn't include any silent durations -- durations that contain no messages or only contain non-note messages -- from the beginning or end of the MIDI file as part of the song duration.)


Here's what I know so far:

All of the amount names ending in -time appear to be in microseconds.

The practice-overage-time amount appears to indicate how much longer than the song duration you took to play through the song. If you add the song duration and the practice-overage-time, you get the "Time Spent" shown on the "Points Earned" table. The "Time Spent" is then used with the song duration to calculate the "Actual Speed" percentage (song duration / "Time Spent").

However, I can't find the song duration in scores.xml, so maybe it's calculated from the MIDI file every time, or stored somewhere else. I peeked in the other xml files but didn't see it. In particular, in scores.xml, the possible-held-time appears to be the union of the durations of the notes for the particular part you selected, for example, the Right Hand part. (By "union", I mean notes that overlap have their overlapping durations only counted once.) But if there are any rests in that part, then that time is less than the song duration.

The summed-held-time appears to the sum of the durations of the all notes for the part, with all overlapping note durations included. Any rests in the part may initially reduce the time compared to the the song duration, but any chords or overlapping notes could increase the time back over the song duration.

In scores.xml, each Config item appears to represent the particular mode you played (which of the 9 buttons you pressed: any of the 3 buttons "Left Hand", "Right Hand", or "Both Hands" under any of the 3 headings "Practice the Melody", "Practice the Rhythm", or "Song Recital") for a particular song and a particular setting of the hands in the "Hands, Colors, and Instruments" screen. I don't know how the hash value is calculated to identify this particular combination. (I know that song items in other xml files use an MD5 hash of the MIDI file to identify the song. But I don't know what the input to the hash function is that the scores.xml Config items use to identify a specific play mode and hand setting of a particular song.) Currently, I only have a small number of Config items in my scores.xml so I can look for some specific values I see on the "Points Earned" table to confirm I'm looking at the matching Config item.

The speed-integral appears to be the number of notes (notes-user-could-have-played) times 100, but I don't know what this value does.


By the way, you said you tallied up the average speed, how did you do that? I believe you need the song duration to calculate the exact "Actual Speed" shown on the "Points Earned" table, but I couldn't find the song duration in scores.xml. (If I used the possible-held-time as a potential substitute for the song duration, I got close, but not an exact match to the Actual Speed percentage or Time Spent shown, because a part can include rests. I thought perhaps identifying the longest possible-held-time for a song might get a closer match, but I don't know how the Config item hash values work to try to get all of the Config items that are for the same song.)
Nicholas
Posts: 13135

Post by Nicholas »

This is amazing work. You are correct on every account. :D

The song's overall duration is an invariant, so it doesn't get stored with performances. Like you suspected: because Synthesia already has that value on-hand (because the only list of performances you ever see in the app are in the context of an already-loaded song), it never needs to store it. (Sorry for the inconvenience!) This is why the overall stats shown when you choose a profile from the Settings screen also don't include a global'ish "time spent playing".

speed-integral is the sum of the current song speed for each note. If you keep the song at 100% speed, it will look like note-count * 100. But if you adjust the speed at any point during the song, additional notes will only be multiplied by that "percent" speed instead. This is the way to calculate the average speed that is robust against mid-song speed changes.

ConfigHashes are a pain, sorry. It's a weird, proprietary algorithm that would not be easy to reproduce (i.e., it's super brittle and has broken a few times over the years over things like loading a file on a separate thread instead of the main thread. Really. The FPU's state flags became unintentionally involved and the new thread context cleared them. It was a mess!) The answer is to have Synthesia migrate away from ConfigHashes, which should be happening relatively soon as each update migrates one more of the old .xml files over to the new SQLite database.

(This last bit is to combat the rare, 15-year-old bug that files don't always write correctly on other people's machines, but always on mine. SQLite in WAL mode includes a full ACID guarantee, which sounds better than something I'd like to try writing myself.)

Instead of unreproducible, proprietary ConfigHashes, I'm actually thinking of just using a [song MD5 + handParts string + selected hand/mode] key in the database. handParts is usually very short (6 characters?) unless you've needed to use the hand-splitting tool. Even then, it can be reasonably compact. Broken out as a separate table (so the handParts aren't repeated in the database for each performance), it shouldn't be too onerous. And having the song's MD5 in there as part of the database key means there is finally an easy way to connect the dots.
Andre-Louis
Posts: 2

Post by Andre-Louis »

Thank you both for your replies.

Like Nick said, the song speed is equal to the speed integral divided by the total notes.

Thank you for pointing out that the durations are in microseconds. I made a list of all of the actual names corresponding to hashes in my scores file (not without tedious work) so I'm able to compare the actual song duration with the possible held time quickly. If I divide the possible held time by 10^6 it's pretty close to the song duration in seconds at 100% speed for most of my songs (the biggest error was 5 seconds). If I just scale that value by my play through speed I get very close to what must be the actual value: my program puts out 533 hours, or about 34.5 minutes of daily practice for 2.5 years, which sounds about right.

https://replit.com/@AndreLouis/SynthesiaScores#main.py my program, in case anyone is interested. Just copy-paste your scores into the scores file and click run to see your stats (make sure filename on line 2 is "scores", not "scores_test")

Thank you both again!
Nicholas
Posts: 13135

Post by Nicholas »

Andre-Louis wrote: 05-03-21 9:08 am... my program puts out 533 hours, or about 34.5 minutes of daily practice for 2.5 years, which sounds about right.
That's... impressive. :D

(That easily rivals how much time I've personally had the app running during testing/debugging.) :lol:

Thanks for sharing your Python code.
Post Reply