Midi list is shuffled and bugged out and files are not showing up

Trouble with Synthesia, your keyboard, or adapter? Think you found a bug?
When describing problems, always mention your OS and game version (shown at the bottom of the title screen).

If your keyboard has USB or MIDI ports, there is a tremendously high chance (>99%) it will work with Synthesia. See what you'll need on the keyboards page.
Post Reply
Piotr Barcz
Posts: 5

Post by Piotr Barcz »

A few months ago, my song list got completely screwed up where folders would appear halfway down the list of midis instead of staying at the top, whole sections of the list also get duplicated and when you scroll down and then back up the whole thing reshuffles. I've also had midis that were perfectly fine in Synthesia just stop showing up in the lists. I don't know what is going on but reinstalling the program didn't help and neither did deleting the saved data. Is this because I have nearly 3000 midi files? Or is it because my computer has been a real piece of crap? I never had this problem with my other weaker computer running 10.8 or previous versions. I might switch back to 10.4 or 10.0 which I guess I'll have to run from a folder instead of the typical install for the time being.
I'd post an image of what the chaos looks like in my folders but I don't think I can upload one.
Nicholas
Posts: 13135

Post by Nicholas »

That is... very strange sounding.

I haven't heard any reports of anything like that happening to anyone else and 10.8 has been out for a week shy of a full year now. This sounds like one of those voodoo problems where I want to ask "how long has it been since the last time you restarted your computer?" :?

Does the same problem happen in the latest Synthesia 10.9 preview?
Bavi_H
Posts: 116

Post by Bavi_H »

I have also noticed this symptom, I can't remember when I first noticed it. I assumed the glitchiness I saw in the file list was related to my having multiple files with the same contents. But maybe that is a false lead, I don't know for sure if duplicate files are causing the glitchy file list.
  • In my case, I have folders where I keep incremental versions of MIDI files I am working on (usually with names like "1.mid", "2.mid", and "3.mid"). When I have a final version of the MIDI file ready, I make a copy of the final MIDI file and put it in a parent folder (with a name like "Song Title.mid").

    Synthesia uses a hash-based system to indentify MIDI files by their contents so it recognizes, for example, that my "3.mid" and "Song Title.mid" files are exactly the same contents. It somehow picks only one of them to show in its list of MIDI files.

    I assumed the glitchiness I saw in the file list was releated to my having multiple files with the same contents. I have seen past comments from Nicholas that he wants to eventually change the system to allow files with identical contents to all appear in Synthesia's song list. (See the second item discussed in the thread Scan MIDI files manually and allow duplicate files.)

    I didn't bother reporting the issue, since the obvious work around would be to try to put my duplicate "in progress" files somewhere else where Synthesia can't see them. But I haven't yet tested this to confirm if duplicate files are causing the glitchy file list.

I want to keep investigating to see if I can find a reproducible example that causes the glitchiness. But to start with, here is an example of the glitchiness I see.

I connected my second monitor, which I use in a portrait orientation, to try to get fewer screenshots with more files per screen. To my amazement, when I moved Synthesia to the second monitor, the glitchiness in the file list went away! And when I moved Synthesia back to the main monitor, the file list was still correct!
  • More specifically, the [removed] screenshots show what Synthesia's song list looks like when I'm looking at my main "midi" folder.

    As I scroll down, there is a strange repeat of 7 previous entries ([item 17] to [item 23]). Clicking on any of these 7 items has no effect. After that, the last 3 items ([item 38], [item 39], [item 40]) are more MIDI files that behave normally when clicked on.

    When I pressed Windows+Shift+Right to move Synthesia to my second monitor, the list is now correct. The 7 items that were repeats have now been replaced with normally working items ([item 31] to [item 37]).

    When I pressed Windows+Shift+Left to move Synthesia back to my main monitor, the list remained correct.
Last edited by Bavi_H on 11-07-22 4:50 am, edited 2 times in total.
Nicholas
Posts: 13135

Post by Nicholas »

Bavi_H wrote: 10-26-22 6:27 pmTo my amazement, when I moved Synthesia to the second monitor, the glitchiness in the file list went away!
This is the kind of sentence that makes programmers want to go live in the woods, far away from civilization. :lol:

(And, yes, the plan is still to keep duplicates in the list. I have a pretty big revamp coming to the "song scanner" that runs in a background thread to fix one of the embarrassments and the duplicates-are-removed behavior will be going away at the same time.)

I notice you have your sorting set to Title (which involves talking to SQLite's "collation" function, which I've had to mess with in the past). Have you ever witnessed the same list corruption when sorting by Points (which, admittedly is how I debug nearly 100% of the time).

@Piotr Barcz: are you also experiencing this when sorting by Title?

(It often feels like there is a deluge of new bug reports moments before I want to make an official release.) :anxious:
Bavi_H
Posts: 116

Post by Bavi_H »

Nicholas wrote: 10-26-22 6:51 pmI notice you have your sorting set to Title [...]. Have you ever witnessed the same list corruption when sorting by Points
Clicking the different sort options seems to result in different instances of glitchiness, see the [removed] sequence of screenshots. If I leave the "Points" sort option clicked and restart Synthesia, the initial list looks the same as the first screenshot (the original "Title" sort), but now with the "Points" button clicked.

However, I mainly use Synthesia's "Watch and Listen" mode, and only occasionally test its game modes, so almost all of my files have no points data.
Last edited by Bavi_H on 11-07-22 4:48 am, edited 1 time in total.
Bavi_H
Posts: 116

Post by Bavi_H »

The action of moving Synthesia to my second monitor causing the file list to become correct got me thinking about other display settings.

I normally use the Control Panel setting to make the size of all items 125% of normal. (See the exact setting in the [removed] screenshots.)

If I change the setting to 100% (and accept the prompt to log out and back in for the change to take effect), Synthesia's file list doesn't have the 7 repeated items in it.

By the way, I'm using Windows 8.1. And I noticed the problem in both Synthesia 10.8.5686 and Synthesia preview-10.9.5850. (See examples in the [removed] screenshots.)
Last edited by Bavi_H on 11-07-22 4:49 am, edited 1 time in total.
Bavi_H
Posts: 116

Post by Bavi_H »

Here's a hopefully reproducible example.
  1. In the attached 7z file, extract the "test" folder somewhere.
  2. In Synthesia, go to the Settings → Songs page and add the "test" folder to the locations searched for songs.
  3. In Synthesia, go to the "Play a Song" screen → "All Songs (grouped by folder)" → "test".
  • If the display is set to 125% sizing, the list will show items 01 to 30, then non-functional duplicates of items 17 to 23, then items 38 to 40.
  • If the display is set to 100% sizing, then all items 01 to 40 appear correctly.
Description:

In the test folder, items 01 to 22 are folders (each containing a single unique MIDI file, so the folder will show up in the list), then items 23 to 40 are unique MIDI files. This matches the pattern of my main "midi" folder.

The individual MIDI files are just dummy files I made. They each contain one quarter note with a MIDI note number that matches the number in the file name.
Attachments
test.7z
(891 Bytes) Downloaded 2 times
Nicholas
Posts: 13135

Post by Nicholas »

That test set of files was very handy, thanks. (I love that the whole archive compresses down to less than a thousand bytes because LZMA found the commonalities between all the nearly-identical files. The bulk of that file is probably the list of file paths!) :lol:

I think I tracked it down. At least... I tracked down one bug and can no longer get it to reproduce. Although, it was a little inconsistent to begin with.

I did spot it happening while changing DPI settings, but once I (vaguely) understood the catalyst, I didn't need to do that part anymore. First: it requires some number of blue ("group") entries in the list. Second, it seems to happen most regularly when you can get the library to auto-scroll past the top of the list. One way to do this is to choose an entry near the bottom of the list (so the song loads), change the folder (or even screen) and then return again. Synthesia tries to show the currently loaded song as the default selection, so it'll pop you down to that part of the list on the first frame.

Causing that automatic jump to happen seemed to be the most consistent way to get the bad entries in the middle of the list, but it still didn't happen 100% of the time. It also seems to be important that the part of the list you're viewing is within one screen's-worth of entries of the blue groups.

There were even warning lines being written to the log about song list "cache misses". (The code for that screen tries to talk to the song database as little as possible, so it saves some of the results in a tiny cache.) I'd seen a few of those warnings over the years but never correlated it to any behavior on the screen. I must simply have never noticed the song entries mixed in with the blue group entries. Hrm.

The bug itself isn't very interesting. It was just incorrect math since the very first day the smoothly-scrolling song list code was written. I was adding two numbers when I should have been taking their max. That line was written in Nov-2012 though, which... means this problem has been around for a very long time! :?
Post Reply