Metronome delays on Android

Trouble with Synthesia, your keyboard, or adapter? Think you found a bug?
When describing problems, always mention your OS and game version (shown at the bottom of the title screen).

If your keyboard has USB or MIDI ports, there is a tremendously high chance (>99%) it will work with Synthesia. See what you'll need on the keyboards page.
Post Reply
jturcotte
Posts: 4

Post by jturcotte »

Hi,

I'm seeing delays of around half a beat between the metronome and the timing to get a "Perfect" note when running at 100% speed, but only on Android.

At first I thought it was a MIDI input issues, but I can also reproduce the issue with the on-screen keyboard. I tried playing with all the settings and I can't seem to find a way to get it to work better. I also feel like it's worse when using the uniform beat metronome, but that could only be my perception.
I also tried with the macOS version but it's working perfectly fine.

It would be nice to find a way to make it more reliable since attaching the laptop to the keyboard isn't nearly as convenient as just placing the tablet there :D

I'm using a Nexus 7 (2013) tablet with Android 6.0.1 and Synthesia 10.3.
User avatar
jimhenry
Posts: 1899

Post by jimhenry »

If your keyboard can produce the metronome sound, you might be able to eliminate the delay by sending MIDI to your keyboard for the metronome sound, which, if I remember correctly, is bundled with the background sounds.
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/
jturcotte
Posts: 4

Post by jturcotte »

I initially didn't enable this since I have a Casio CDP-130 and I can only hear the first beat of the metronome and I hear nothing in uniform beat mode (I guess that instrument isn't supported). It has a metronome function itself so it would also be a good solution if I was able to select different instruments to be used for the metronome.

Thank you for the suggestion, I tried enabling the metronome on both output devices and I can clearly hear the timing difference for that first beat. The one output on the keyboard has the right timing to get "perfect" notes.
It seems to be a general delay in the sound output since playing a song also shows the same issue, the song's notes are delayed on the tablet's speakers!
Tested the same on macOS and both devices perceivably play the song with the same timing.

I just tried with my phone as well, a Google Pixel running Android 7.1.1 and I hear the same delay, and given that the keyboard is the one that allows me to get the right timing, could this be a bug general to all android devices? Is there an official way to report bugs to the developers other than this forum?

I'll try to play with only the first beat, but I think I'm not good enough to figure out the rest of the timing by myself yet.
User avatar
jimhenry
Posts: 1899

Post by jimhenry »

The issue is that Synthesia produces MIDI output, which is a stream of instructions for producing sound. The MIDI has to be converted to sound for you to hear anything. The default is to have the device that is running Synthesia, the Android in your case, convert the MIDI to sound. Converting MIDI to sound always takes some time. It is imperceptible when done by your keyboard because it has specialized hardware for doing the job quickly. An Android can be quite slow, as you have heard.

I'll now have to defer to Nicholas or someone with hardware similar to yours for suggestions on how to deal with this in your case.
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/
Nicholas
Posts: 13135

Post by Nicholas »

jturcotte wrote:... could this be a bug general to all android devices?
Yes, a bug in Android. Google left audio as a kind of afterthought when they first designed Android. (Compare to iOS where it was a first class citizen from day one.) Until very recently they'd continued to ignore the audio path latency. (They didn't start to talk about it until Android 6 or so.) We've seen upwards of a quarter second depending on the age of your device.

That said, I'm actually kind of surprised the Pixel is still that bad. First party hardware running Android 7 should make that a nearly solved problem. Hmm.
jturcotte
Posts: 4

Post by jturcotte »

I've recorded both devices using my laptop while playing the keyboard's and software MIDI outputs at the same time in Synthesia, and then measured the distance between the start of the two sound waves in Audacity (basically how much slower is the software vs hardware MIDI playback):
  • Nexus7: 122ms
  • Pixel: 109ms
I tested the audio latency using the test app at http://superpowered.com/latency on the same devices and I got the following results (which match the times in their list of devices):
  • Nexus7: 77ms
  • Pixel: 28ms
Those last times are round-trip times, so to compare I guess we should divide them by two (assuming that input takes as much time as output). There is no synthesizing to do in that case, but it's not the most CPU-intensive task to do, so I'd hope this isn't the sole reason for those added 50-90ms delay.
I didn't know that the sound latency could be so bad on Android, maybe it's not worth going that extra mile when many people will rather use their keyboard's speakers.
User avatar
jimhenry
Posts: 1899

Post by jimhenry »

I am not sure if we can subtract the second number from the first number to get the latency due to MIDI to audio conversion. But if we can, it is interesting that the Pixel seems much worse in this area than the Nexus7. In any event, all these numbers are very bad.

Maybe prospective Android users should be warned that they will need to use their keyboard's sound generation for acceptable performance. Those with controller keyboards should be strongly cautioned that using an Android is likely to be unacceptable. Those without keyboard generated metronome sounds should be warned that the metronome will be unavailable.

Nicholas, should the Android version have a default configuration of using the on-board keyboard sounds if keyboard input is configured?

As full disclosure, I use local sound and I think that should always be done if possible. Nicholas, I forget why you seem to favor generating the sound with the computing device.

In any event Nicholas, based on the number of questions about configuring Music Devices, I think you should give Music Device configuration some serious attention in the v11 release cycle. Maybe a configuration wizard?
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/
Nicholas
Posts: 13135

Post by Nicholas »

109ms is actually pretty fast for some of the devices we've seen. I've done that same test (a couple years ago, with a multi-track recorder so I could isolate the sound of the key striking the keyboard bed from the onset of the notes produced by each) with a couple Android devices we've got here. The ASUS Transformer Pad was 232ms from key strike to note onset! A quarter second is unusable by just about any standard I could imagine.

Regarding Superpowered's latency tester, it's not exactly a fair fight. Their app is designed for best-case scenario conditions to check the absolute bare minimum the hardware allows. (Even so, looking down the list on that page you linked is scary! Even in these best-case conditions, there are a remarkable number of devices above 300ms. Some above a full second. That's insane.)

Synthesia has a bit more of an uphill fight:
  • Audio synthesizing doesn't take zero time, like you suggested.
  • The USB subsystem going into Android doesn't necessarily have 0ms latency. (Nor your keyboard's MIDI output for that matter!) Your keyboard gets to start producing audio as soon as the key strike is detected. That's only the beginning of the journey for Synthesia: it doesn't hear about it until the keyboard sends it out, your device receives it, and Android hands it off to the app. That said, this one is probably less than 1-2ms, so including it in this list is a little unfair. In a pie chart, you'd barely be able to see this slice.
  • Synthesia only checks for MIDI messages and updates audio once per video frame. If a MIDI message arrives 1ms before a video frame, it will be handled in 1ms. If a MIDI message arrives 1ms after a video frame, it will be handled in 15.6ms. So the range here is a uniform distribution between 0 and 16.6ms, so on average it adds around 8ms. (Synthesia could do better if it processed MIDI/audio in another thread.)
  • The Android synth -- made primarily for early cellphones with MIDI ringtones in the early 2000's -- synthesizes a 22.05 kHz audio stream (instead of today's more standard 44.1 or 48 kHz), which means that the logcat whenever you start Synthesia shows a warning along the lines of "requested sample rate doesn't match hardware rate; audio fast-path denied". Not getting to use the "fast path" is probably the bulk of the difference between the two apps. In a pie chart, this would be nearly the entire pie.
There are a few solutions for that last point. It would be nice if there were a better synth available that we could just tell to synthesize at the native hardware sampling rate. Alternatively, we could implement our own upsampling (instead of using Android's). At first it just sounds like that wouldn't make a difference; only trading off who does the upsampling, Android vs. Synthesia. But there is more on the "slow path" than just upsampling so it should make a measurable impact.

Doing the MIDI processing on a separate thread and checking to see if doing our own upsampling on Android improves the situation are both tasks on our list.

All of that said, for exactly the reason you ended with (and because the upsampling task is Android-only when we have to choose our battles across all four platforms we support), these aren't quite at the top of the list. I wish I had better news, sorry!
User avatar
jimhenry
Posts: 1899

Post by jimhenry »

Nicholas wrote:109ms is actually pretty fast for some of the devices we've seen.
Latency is an issue for pipe organs because of the distance from the pipes to the organist. The speed of sound is roughly 1mSec/foot. Pipes 40 feet from the organist is not unusual and that's a 40mSec delay plus any delays between pressing a key and the pipe generating sound.

The rule of thumb in the organ world is that a delay over 20mSec will be perceptible by most organists. Organists differ in how well they cope with longer delays. Some manage to play well even with very long delays. But that is a skill that is pretty unique.

Pianists and players of electronic keyboards generally don't do well with any perceptible delay.

Which brings us back to the question of whether local sound generation by the keyboard should be the preferred configuration for Synthesia Music Output?
Jim Henry
Author of the Miditzer, a free virtual theatre pipe organ
http://www.Miditzer.org/
Nicholas
Posts: 13135

Post by Nicholas »

jimhenry wrote:Which brings us back to the question of whether local sound generation by the keyboard should be the preferred configuration for Synthesia Music Output?
Absolutely, at least as far as Synthesia is concerned in the near future.

Maybe I should have been more explicit: "109ms is actually pretty fast [relative to the rest of the completely abysmal performance we've seen across every other Android device]..." :lol:

At last test, the built-in Windows synth (circa Windows 8.1) was around 125ms, which also isn't even close to acceptable.

What would need to change to lift the "prefer your keyboard" guideline would be an actually-high-quality synth (and sound set) rendered directly by Synthesia and output using each platform's fastest native API. That would be WASAPI (or ASIO) in Windows and maybe each of the major Android vendor's proprietary API (e.g., Samsung's Professional Audio).

If you look at any of the major DAWs or the two primary commercial competitors in the sheet music editing space, they all include amazing sound engines with upwards of gigabytes of samples to make the output sound incredible. They've spent the time (and sound set licensing fees) to tackle this problem the right way. Their latency and quality are generally comparable to (or better than) a keyboard's.

... until then, for Synthesia, keyboard synths are the way to go.
jturcotte
Posts: 4

Post by jturcotte »

Thanks a lot for the explanation. To be honest I don't really need to use the tablet's synth, the speakers themselves are already much worse than what's on the keyboard.

The source of the issue is that I could only hear the first beat of the metronome on that keyboard, and ideally there would be a way for me to tweak Synthesia in order to fix this, but it's also something I can live without since I know now that I can rely on the graphics for timing (I wasn't sure), or use my laptop if I really need that metronome.

The main purpose was to report the issue in case this was a bug, but since this is rather a limitation and you're already very aware of it, I'd like to thank you for taking time to look at it, and for the great software :)
Post Reply