Different MIDI command for left and right hand

Synthesia is a living project. You can help by sharing your ideas.
Search the forum before posting your idea. :D

No explicit, hateful, or hurtful language. Nothing illegal.
anykey
Posts: 9

Post by anykey »

Hi,
I am using the Komplete Kontrol keyboard which has full color Guide Lights. I wrote a simple program to control the lights using Synthesia but ideally I would like to upgrade it, so that I can use 2 different colors for the left and right hand notes.

I guess there are 2 "easy" ways to do this.
- Use 2 seperate MIDI channels for left and right hand, but this might break compatibility with other keyboards
- Use the intensity/velocity value. Right now, all notes are send with intensity/velocity of 1. If you can make notes for left hand with intensity 1 and the right hand with intensity 2, I could easily use that data to control the right color lights.

Here my code to control the Komplete Kontrol (S61) lights using synthesia.
https://github.com/AnykeyNL/SynthesiaKomplete

Regards,
Richard
Nicholas
Posts: 13135

Post by Nicholas »

First: nice work! How did you manage to reverse the protocol? Did you have to use a hardware analyzer? It's a shame that it takes HID messages and isn't just shoehorned into MIDI messages. That would have been more convenient. Whenever NI adds "official" support, it sounds like it's going to be more work for us (especially on iOS)! :lol:

Also, am I reading this right: do you get to supply a raw RGB-triple for the color?! The documentation we've seen from NI (that doesn't work with stand-alone apps like Synthesia) showed a fixed set of about half a dozen colors. Full RGB support would be awesome.

Now to answer your question:

The next preview of Synthesia 11 is about to support the proprietary key lights from "The ONE Smart Piano". It supports a left/right hand distinction (using two different color lights), which we're hoping to support. I vaguely remember that information already being pulled through so it's trivially accessible to Synthesia's key light code. So it should be easy to add. We'll see if that's actually the case.

In any event, their "proprietary" scheme really just boils down to using After Touch messages (0xA0) instead of Note messages (0x90) with the lowest two bits of data byte 2 indicating hands. (Bit 1: "red" light, bit 2: "blue" light; I forget which hand those correspond to at the moment.) The only other difference I've seen is that they decided not to use MIDI note numbering -- where middle C is 60 -- and instead start counting from A0 at the far left edge of an 88 key keyboard. You could adapt your Python code to listen for that scheme instead and you'd get left/right hand broken out. And because they use completely different messages, you could just have it listen for both schemes simultaneously and get forward/backward compatibility with Synthesia's "traditional" key lights in older versions and the new proprietary scheme at the same time.

It's not ideal (mostly because it's a one-off implementation by some other manufacturer; making you adopt Synthesia's quirks which were necessarily adopted from their quirks), but it would solve the problem. :D
anykey
Posts: 9

Post by anykey »

Hi,

Yes I can provide full RGB codes and that also really seems to work. Here a short example: https://www.instagram.com/p/BZTdar0DLU6/

I also seem to be able to refresh the data very fast, without impacting the performance of the keyboard in anyway.

Thanks for the info about Version 11, when released I will modify my python code for it. Sounds like I can easily modify my code for that :-)

Will Synthesia 11 be a free upgrade? any idea on release date?
Nicholas
Posts: 13135

Post by Nicholas »

That RGB video is awesome (and promising)!

11 -- like all past and future releases -- will be a free update. :)

The next preview of 11 is coming along nicely. I've learned not to give release dates though, because they're always wrong. The best I can offer is to follow along with the news in this section of the forum.
Nicholas
Posts: 13135

Post by Nicholas »

... actually, I can do even better!

I've had a chance to check and I remembered correctly. There is an old, never-enabled feature lurking in the key light code for a device prototype we were working with a manufacturer on (years ago) that never made it to market as far as I know. It was going to have RGB lights and would use a different color per finger. Everything in Synthesia was ready to go: per-finger light messages that fell back automatically to just the hand if finger hints didn't exist. I just tested it and it appears to still be working perfectly.

This one has fewer quirks because I helped to invent the scheme! :lol:

Here's how it works:
  • Choose "Finger-based channel" from the bottom of the key light channel list.
  • Have your Python app watch for Note messages, mostly the same as before with the following difference:
  • The finger is stored in the message's channel according to the following table.
These are zero-based (despite most MIDI libraries using ones-based counting for channels).
  • Channel 0 means "we don't know who or what this note belongs to, but light something up anyway".
  • Channel 1-5 is left hand fingers, thumb through pinky.
  • Channel 6-10 are right hand fingers, thumb through pinky.
  • Channel 11 means "left hand, unknown finger".
  • Channel 12 means "right hand, unknown finger".
(Note Off messages are still channel-less and always use MIDI status 0x80.)

This system provides even more info than the upcoming ONE Smart Piano lights and also means you don't have to convert between MIDI note numbers and that weird A0-based key indexing.

The only thing that's a little wrinkly here is that the Note On channels technically won't match the Note Off channels. So if you send these out to a keyboard/synth that doesn't understand this system, you'll get a bunch of eternally-opened notes. The workaround: don't send these out to an unsuspecting keyboard/synth! :D

There is even a bonus message that gets sent when Synthesia begins playing a song. You can use it as a sort of "reset all the lights" / "get ready" type of thing. It uses a general purpose CC message (CC#16) to say which hands have been selected and whether the user is playing (vs. "Watch and Listen Only"). Here's how it works:

Status: 0xB0 (CC on channel 1)
Data1: 0x10 (CC#16: General purpose slider)
Data2: bit flags, see below

Code: Select all

enum LightHand
{
   Unknown = 0,

   Right = 0x1,
   Left = 0x2,
   Both = Right | Left,

   UserPlayed = 0x4
};
Some examples:
  • User-played right-hand: B0 10 05
  • User-played left-hand: B0 10 06
  • User-played both hands: B0 10 07
  • Watch and listen only: B0 10 00
anykey
Posts: 9

Post by anykey »

Cool! Thanks for the detailed info!!!

I can easily adjust the script to this. Will do so when availble and will make movie to show it off :-)
Nicholas
Posts: 13135

Post by Nicholas »

The new key light mode is live in the 10.4.4376 preview! You can try this out now.
smalok
Posts: 1

Post by smalok »

This sounds very interesting!

@anykey
Any progress by now?

Thanks for all your hard work!
ojacques
Posts: 8

Post by ojacques »

Greetings,

I'm working on making this work (full lighting guide on my Komplete Kontrol mk2 S61 for Synthesia). With info from another post - https://www.native-instruments.com/foru ... ts.320806/ - I can light keys as needed.
Unfortunately, I can't find the finger based channel mode on the latest 10.5 r4442. It was supposed to be there on r4376 as per viewtopic.php?f=9&t=8765.

Has it been removed or I just can't find it? If it was removed, can I still get r4376 to try this out?

Thanks :)
Nicholas
Posts: 13135

Post by Nicholas »

It should be available all the way in the official 10.4 version. Settings --> [your output device] --> Key Lights --> Finger-based Channel (all the way at the bottom of the list).

All of the discussion on that page is very promising! We already use the HID API in Synthesia's PC and Mac codebase (for Jamboxx support), so if anyone in the KK community comes up with something compatible with that (and is BSD, MIT, or similarly-licensed) that should be easy enough to incorporate into Synthesia.
ojacques
Posts: 8

Post by ojacques »

I see it now that I switched to English language!
The french translation is a bit off. It shows "Canal digital". Should be more something like "Canal basé sur les doigts" to keep it closer to original meaning.

Off to more coding... (and yes, I'll use MIT license). I use this as an exercise for programming with my son too...
ojacques
Posts: 8

Post by ojacques »

@Nicholas,

I hit an issue with the note_off event. It seems that Synthesia sends a note_off for this finger light mode, even if the note is still on.
For example, for the "Moonlight Sonatha", where A3 (57) should stay on for a little while at the beginning, it gets turned off quickly after, as you can see from my debug code.

Code: Select all

Listening to MIDI
Received note_on channel=1 note=57 velocity=1 time=0
Received note_on channel=6 note=64 velocity=1 time=0
Received note_off channel=0 note=57 velocity=0 time=0
Received note_off channel=0 note=64 velocity=0 time=0
Received note_on channel=8 note=69 velocity=1 time=0
Received note_off channel=0 note=69 velocity=0 time=0
Received note_on channel=10 note=72 velocity=1 time=0
Received note_off channel=0 note=72 velocity=0 time=0
Received note_on channel=12 note=64 velocity=1 time=0
Received note_off channel=0 note=64 velocity=0 time=0
Received note_on channel=12 note=69 velocity=1 time=0
Received note_off channel=0 note=69 velocity=0 time=0
Received note_on channel=12 note=72 velocity=1 time=0
Received note_off channel=0 note=72 velocity=0 time=0
Received note_on channel=12 note=64 velocity=1 time=0
Received note_off channel=0 note=64 velocity=0 time=0
Received note_on channel=12 note=69 velocity=1 time=0
Received note_off channel=0 note=69 velocity=0 time=0
Received note_on channel=12 note=72 velocity=1 time=0
Received note_off channel=0 note=72 velocity=0 time=0
Received note_on channel=12 note=64 velocity=1 time=0
Received note_off channel=0 note=64 velocity=0 time=0
Received note_on channel=12 note=69 velocity=1 time=0
Received note_off channel=0 note=69 velocity=0 time=0
Received note_on channel=12 note=72 velocity=1 time=0
Received note_off channel=0 note=72 velocity=0 time=0
Received note_on channel=2 note=55 velocity=1 time=0
Received note_on channel=12 note=64 velocity=1 time=0
Received note_off channel=0 note=55 velocity=0 time=0
Received note_off channel=0 note=64 velocity=0 time=0
Received note_on channel=12 note=69 velocity=1 time=0
Received note_off channel=0 note=69 velocity=0 time=0
It may be because of an issue with my code, or maybe some library in between, but I can't figure it out. Could you please check?
ojacques
Posts: 8

Post by ojacques »

Never mind, there is something else (a double output maybe?) interfering which sends this note off. I need to dig deeper to find the right combination.
ojacques
Posts: 8

Post by ojacques »

Code and video available here: viewtopic.php?f=16&t=9220
User avatar
srimantha@gmail.com
Posts: 14

Post by srimantha@gmail.com »

Hi Nicholas,

I am New to this forum.
First of all, I'd like to thank you and your team for creating great software 'Synthesia'.

I am currently doing an RGB LED project to assist Piano using Synthesia for my kids.

From the above section, I found how to SPLIT two-channels (LEFT-HAND/RIGHT-HAND) information to display just two colors in my LED Strip.

Bu,t I would like my LED's to change colors Automatically along with Synthesia LEFT-HAND/RIGHT-HAND INSTRUMENT COLOR settings, when we nce change colors in the Synthesia software.

Are you already sending LEFT-HAND/RIGHT-HAND INSTRUMENT COLOR information along with notes ? Are there any SETTINGS for that ?
is there any forum discussion about it ? Sorry, I searched but couldn't find such.

Thank you.
iPiano.us | MIDIVisualizer.com
Nicholas
Posts: 13135

Post by Nicholas »

The post just above is still the current state-of-the-art.

For many years Synthesia couldn't do much more than take advantage of whatever (almost nonexistent) features were exposed through commercial digital pianos. Now, with the advent of custom-made (often Raspberry Pi-based) LED strips along with the increasing number of people that seem to be using them, the doors have been flung wide open on what is possible.

I'm excited to start expanding the (sort of ad hoc) light message "protocol" to include just about anything that could be useful.

It looks like you found some of the related discussion about an hour later. That (and more) are in the plan now.
User avatar
srimantha@gmail.com
Posts: 14

Post by srimantha@gmail.com »

Exited to see the new updates. Hope it will happen soon. :D

I've already updated my RGB strip for current Finger Based (two) channels, and getting two channel information.

https://youtu.be/IPsIHKBg1JQ

Then I tried changing all synthesia settings to see for any possible signal to get multicolor as instruments. (at Hand,Color,Instrucments menu). It didnt worked. May be you can send 5 finger information (5 colors for 5 instrucments x 2 hands) as another profile. Just a thought. :D

Its a great idea as per your other post about sending multi-color information along with each key play,

At the same time make another color profile, just to send currently available limited colors for multiple instruments (at Hand,Color,Instrucments menu)
iPiano.us | MIDIVisualizer.com
Nicholas
Posts: 13135

Post by Nicholas »

I wonder if instead of going with some generic "here are 7 bits to express the hue" kind of scheme, the longer-term approach to deliver even more information isn't some table-based method? Maybe at the beginning of the song (along with the CC#16 to describe that the user has picked left, right, both, or no hands), Synthesia could dump out some list of information. It could be something like "index 3: uses MIDI instrument 47 and should appear using this RGB triple"?

Then, the notes could use all 7 available bits to refer to (up to 127 of) the indices in that table. The best part is that the protocol could expand to as much extra information as we wanted in the future. Like, it could say (for each of the entries in the table) which hand (or none) it was assigned. That would already be a lot more information than the single CC#16 message.

Maybe I should register Synthesia LLC with the MMA to get an official manufacturer ID that we could use in SysEx messages? :lol:
User avatar
srimantha@gmail.com
Posts: 14

Post by srimantha@gmail.com »

Great idea Nicholas. Since Synthesia has considerable market dominance I believe you should register it with the MMA ;) .
Hope you are enjoyed the weekend. :D

Untill your future updates with Color-information for RGB strips, I am still searching for ways to work around Synthesia somehow to get Instrument color Automatically to my RGB-strip.

(1) when enabling Key Lights and under Finger-Base Channel :
I could get 2 hand information. Channel 1,2 Instrument colors are always Blue or Green. So I can code the RGB-LED strip for those 2 colors.
Unfortunately, Channel1 and Channel 2 colors are not fixed. :(

(2) once I disabled Key Lights:
I can get 16x channel information. Again I couldn't find consistant Instrument colors, song to song :( .
Do you have any algorithm or any method to define Instrument colors? Please give me some idea :idea: until your next update.

If I can find a way how your channels get colors, i can match my RGB-LED strip to display those exact colors. :)
iPiano.us | MIDIVisualizer.com
Nicholas
Posts: 13135

Post by Nicholas »

This post describes the process Synthesia uses to assign colors to the various instruments. (I'm afraid it's not easy to work with in this context.)

Being able to see the 16 channels after disabling the key lights is normal: MIDI notes normally come through with the channel encoded in the status byte. (You could enable key lights and normal output to get both, although then you'd need to do more work to sort of fuse together the two messages you got for every note.)
Post Reply