Embarrassment Driven Design

Try new versions before anyone else!
Always the latest dev version: WindowsmacOSiPadAndroid (or .apk)
Please report comments and bugs!

Your data hasn't disappeared: PC/Mac development previews store their data in a different place. Details here.
Post Reply
Nicholas
Posts: 13132

Post by Nicholas »

Synthesia has always been a little... warty. From the very first design decision to build it on top of a completely custom game engine-style framework, things have been a little "weird".

Some features that are normally "free" when you use the usual tools provided by each platform (like user-interface controls---as simple as a button or as "complex" as a scroll bar---or detecting a pinch gesture on a touch screen) can take considerable effort to implement from scratch. The benefit is that I can count on all those things working identically across all Synthesia's platforms, but there is no arguing that it isn't a lot more work than had I take a more sensible approach.

That extra work has a tendency to make me put off fit-and-finish tasks that aren't "absolutely necessary", where historically I've defined "absolutely necessary" pretty laxly as "so long as people can use it and it's not too confusing". But, I've learned via (thousands!) of emails and forum posts that sometimes my "not too confusing" isn't quite the same as yours. Once identified, it's still been easy to point to some workaround and keep pushing the fit-and-finish a little further down the task list.

I remember an outraged message from a user a long time ago that went along the lines of "it's the year 2012, what do you mean I can't resize the window?!" Of course, reworking Synthesia's UI framework to add support for control "anchoring" to support automatic resizing was a tremendous effort that was the bulk of the work required for the initial iPad release back in Synthesia 8.4. It's already there when you use OS widgets (like everyone else) so the expectation led to a lot of frustration. I agreed it was absurd that it had taken so long and felt a species of guilt that these baseline assumptions weren't being met, so I made sure window resizing was added shortly thereafter.

The trouble is that window resizing isn't the only baseline expectation not being met. There are lots of them. And now it's the year 2019.

As this project continues into its second decade of development, I've learned a lot about myself and how my health is impacted by my work on Synthesia. One example: sometimes life gets in the way a little and support email reply times can begin to drift out into the 2-3 week range. Beginning each of those messages with a heartfelt apology for the inconvenience of having to wait so long for an answer actually takes a lot out of me... so I've learned that it's very important to avoid email getting backed up.

I've also learned that answering one of these "it's year X, what do you mean you can't do Y?!" with "I'm so sorry, that's been on the list forever now and we're hoping to get to it sometime in the next 5 years" when Y is something perfectly reasonable in every other app, it also takes a lot out of me.

I don't like it.

So, for my health and my pride in my work and the improved user experience and ten other good reasons, I've decided those things are more important than I've been giving them credit and I should add them sooner rather than later. I suppose I've already been doing this a little: the sheet music display was/is one of the worst, most embarrassing areas, prompting the multi-year Synthesia 11 effort. That's a huge project though and the other problems have had to sit and wait far too long (leading to more apology emails and general wear-and-tear on my person).

So, the new goal is to work (at least) one of these into each dev preview and to have more frequent previews than we've ended up with these past few years while we're at it.

In no particular order, here are the outrages, most embarrassing bugs, and outright confusing (lack of?) features that I've been able to identify so far:

Sorry, that's been on the list forever and we hope to add it soon...
  • The sheet music is terrible.
  • Why aren't there more than six colors to choose from?
  • The song list doesn't show new files until restarting the app.
  • Added in 10.8, partially: The song list completely buckles under the weight of ~500 or more files.
  • Added in 10.7: There is no way to create a simple song / there is no way to save your output in Free Play.
  • Added in 10.6: "What do you mean I can't use Ctrl-V to paste my short code into the box?"
Buggy, broken things that shouldn't be allowed to persist longer than one preview, let alone years:
  • The video on the front page of the website is a joke: ten years out of date, showing features that haven't existed in six.
  • Fixed in 10.9, partially: On some computers, very rarely, data is written incorrectly when the app is closed, leading to data loss and error messages.
  • Fixed in 10.9, partially: Data is only written when the app is closed, so a crash at any other time means all new data from that session is lost.
  • Fixed in 10.6: Dragging the app between a Retina and non-Retina monitor leads to a useless, double-size view of one quarter of the screen.
Sloppy programming that leads to a worse experience:
  • There is never an excuse for a single dropped frame during play; Synthesia knows everything about a song in advance.
  • High mobile power usage because we're wasting 95% of our effort. Synthesia shouldn't build and send the list of all the falling notes (and everything else) to the GPU every frame.
  • Consuming MIDI input on the main rendering thread adds ~10ms of extra, needless latency.
Unclear or confusing user-interface:
  • Bookmark and loop placement interactions are amateur hour.
  • A surprisingly large percentage of the 1-star mobile reviews that aren't about the app's price: "The keyboard keys are way too small!"
  • The song list "Search" box is actually just an in-place filter. A real search (that checks every folder and shows up on the address bar) would be more useful and less confusing.
  • The guides cover only a tiny fraction of the app and there is no other source of official documentation.
Every few days leading up to this post I've thought of another one and I'm sure I've missed a lot more, so please reply with things you think might belong on this list! I'd like to hear about anything that feels weird or inconsistent or bad.
Fr3dY
Posts: 12

Post by Fr3dY »

Hi Nicholas,

Have you thought about switching to some kind of standard (even multi-platform) framework? It could make you forget about all these UI issues and focus on the app itself (you'll lose "power" over it but in the long term could be a good option).
Keep up the good work and thanks for caring about the application and users... don't let it affect your health more than it should though :)
Nicholas
Posts: 13132

Post by Nicholas »

I may be succumbing to the sunk cost fallacy. That is, I've spent so much time working on the custom framework, that it would be too big a waste to switch to something else now. (The fallacy there is that the time spent---the "sunk cost"---actually doesn't matter and I might be more productive much faster with something else, like you've suggested.)

A lot of my influence comes from reading this article almost twenty years ago now:
Joel Spolsky wrote:[Netscape gave up their lead] by making the single worst strategic mistake that any software company can make: They decided to rewrite the code from scratch.
I know better than that. Although, it might not come to that.

Every once in a while I'll daydream about trying to integrate the top half of Synthesia's code base with the bottom half of the Unreal Engine. That's also written in C++ and would mean the bulk of the base, platform stuff (Vulkan support, tracking down all of Android's idiosyncrasies, etc.) would be done for me and stay(!) done with each update of their engine.

Things have been pretty solid on that front lately (and adding a contract graphics programmer hasn't hurt; Synthesia just got an Apple Metal rendering back-end for "free" as far as time is concerned, which feels pretty amazing). And there is a large cost just in the switching. So it's a tricky decision.

Worse, migrating to a different engine would only fix one of the items under the "sloppy programming" category and the (already complete) "paste" item above. Even glancing through the first three hundred lines of the internal task list (through to the release of 10.6), I only see two tiny engine tasks along the lines of ("double check that X is working the way we expect"). So maybe all the work over the years has finally "paid off" and the lower level stuff is essentially a solved problem (for now, until OS updates start breaking things again).

It's an interesting problem to consider. If I were going to add an "Engine that breaks every time Apple, Google, or Microsoft releases an update" item to the list above, I suspect it would be one of the lower-priority ones. It's more a quality of life thing just for me (which, granted, has health implications) rather than one of the baseline assumption in the user experience that simply aren't being met.
miek
Posts: 37

Post by miek »

That's a really interseting article and, for me, it applies to non-programming things as well. I use Synthesia on the PC and I find it really good - better than those web-based applications. I don't know how long I have been subscribed, but I have been learning the piano all that time. Synthesia for me is the best product. As a PC user I see that there are some tasks that you are doing for the other two platforms which I don't know wheather it affects development time of the software over-all, but maybe it is for the best. The PC I am typing this on is Windows 7 and will be obsolete in a few months - but at least I will have the opportunity to use the other two platforms because of the way it has been developed.
Keep up the good work and the main thing is not to let it damage your health :)
spstarr
Posts: 15

Post by spstarr »

Maybe this a question that shouldn't be asked or, reopened. While we pay for Synthesia and your hard work, which is really great, I know this was once an open source project at the time, I'm assuming it's just you working on the code so there's no others helping out. Does it make sense to go back to some sort of open project so as long as you can make $ from it still?

So more people could help you with the backlog of items or is that not possible with the use of other proprietary DLLs etc in the source now?

Curious,
Shawn.
Nicholas
Posts: 13132

Post by Nicholas »

I was actually just thinking about this the other day, too, although not quite the way you just described it. I like your way more.

I couldn't imagine any way to open (all) of the source while still making sure the whole project wouldn't "escape", so to speak. Like you mentioned, my (about to be one member larger in a couple months) family depends on Synthesia's revenue to survive.

But, I remember something that Tarn was able to do with Dwarf Fortress many years ago where he released the lower-level platform source code (with a silly little non-game on top of it) so that the community could help port it over to Linux. I haven't kept up with that project in a few years now, but my understanding was that the community kept any new features up to date on the Linux side so he could keep working on the actual game code on top of it. It would be a tiny bit more to keep track of (two code repositories instead of one), but that also seems surmountable.

Internally Synthesia is actually (coincidentally) already split out like that a little bit. It might take a tiny bit of work to finish pulling the last few pieces apart, but that doesn't feel very scary to think about. And philosophically, it would be nice to have Synthesia's battle-hardened, tested platform code out there again in the wild to help others. That doesn't seem like a threat to Synthesia sales when none of the music-related code is available. (I suppose it would depend on where I decided to draw the line. Should the UI code be made public, too? That would mean splitting the UI textures out from the keyboard and other Synthesia-specific textures. That's a little more work.)

I could kind of cheat and keep the two repositories completely separate. Maybe once per development preview I could just copy the changes back over to the public repository. And between them I could just accept and integrate GitHub pull requests coming from users in the other direction. That scheme would take very little work, which makes it especially attractive.

I actually like the sound of this. I wonder how much participation we would see. Low-level platform stuff isn't super exciting, but I could see people knock out their pet-peeve bugs, which would be very helpful! And additional (mostly unofficial) platforms like Linux or Raspberry Pi could be completely up to the community to supply, which would also be cool.
spstarr
Posts: 15

Post by spstarr »

Regardless of your decision Nicholas on if or what you might open, this app is amazing and worth every penny when I bought it.
miek
Posts: 37

Post by miek »

I agree, this is a great app, and if you charged to upgrade to synthesia 11 I would probably pay again. :lol:
st5555
Posts: 38

Post by st5555 »

miek wrote:I agree, this is a great app, and if you charged to upgrade to synthesia 11 I would probably pay again. :lol:
I agree, if it would help keep the Synthesia development team solvent to keep getting upgrades it would be a worthwhile investment.
Nicholas
Posts: 13132

Post by Nicholas »

Thanks everyone for the encouragement and positive feedback!

Although, the good news that I am happy to report is that unlock key sales are rather healthy. I am very lucky to be in a position where I can reasonably count on Synthesia being able to support myself and my family's needs. That's also good news for another reason: there is less pressure to make... unsavory business decisions. I don't have to do any ambulance chasing or try to squeeze more money out of people. I mostly let the business side of things coast along (say, by not updating the front page of the site in almost 8 years) so that I get to spend my energy on the part that I like the most: digging into the code and adding cool features.

Really, the only scarce resource (as always) is time. That's why doing something like re-opening the source on the lower level platform stuff is so attractive: there is a chance that work contributed by others could save time. That sounds amazing.
monkel
Posts: 199

Post by monkel »

Thanks for sharing some behind-the-scenes and under-the-carpet glimpses and great to hear that you're doing ok sales wise.
RobertM
Posts: 17

Post by RobertM »

Have just read this article. Really sorry to hear about the issues you are having / facing

I use Synthesia on Windows 10 64bit, it plays perfectly, yes I had an issue with the midi but that was down to the Microsoft Synth engine, since you told me about the built in midi player I use that. We all have to start somewhere. Playing the piano is a huge undertaking, learning to press the keys with different fingers to reading sheet music and listening to the notes. Learning to use this software is also another undertaking, I'm still finding things

It's the same as programming, I'm a software developer writing apps for the web and its hard keeping on top of it, it changes all the time, programming for the different environments and keeping customers / users happy. I used to spend all my time playing pc games when I got home to chill out. Over the years this has had an effect on my eyesight, one of the reasons why I took up piano

If it wasn't for Synthesia I wouldn't be where I am today, it's an amazing tool, I've got 690 songs loaded into it, have no issues at all, it loads perfectly everytime I just wish I'd have found this tool when I was in my 20's/30's, am 50 in 2 months time. it's slow going but I enjoy it

I'm sorry to hear that you are experiencing these issues, your family and health should always come first, as for the forum, I can understand or to be honest don't expect a reply as you have other commitments. If this software cost hundreds of pounds I would be pretty miffed. It doesn't and for the money it's a cracking piece of software. I take this program as it comes

I don't know how others in the forum feel, if you need help with coding please feel free to ask, if I can help I certainly will, we can all help each other with answers to questions. Try and take some of the burden from you. To be honest this forum is better than many professional companies forums, at least you have the decency to answer your customers

I said it in one of my other forums and it's mentioned below, I wouldn't object in paying again or if you don't feel like charging, why don't you ask for donations. I'm not going to name any. Many other big software companies charge for different versions. I must admit much of it is absolute crap and is expensive for what you get. We've only paid once and I'll say it again. For the money it's a cracking piece of software

We're here to help, you are doing excellent work.

Kind regards
Rob
User avatar
zenoxe
Posts: 2

Post by zenoxe »

Very interesting read! I'm a C++ big studio game dev myself, and we've got the opposite problem.
We're a team of 50+ programmers working in unreal, cursing it and sometimes wishing we'd written something ourselves, lol.

Regarding the super long todo list...
I know Synthesia started out open source, and later turned closed-source.
But have you considered making it semi-closed source, letting some select people work on smaller features and bugs?
Might help a lot if they could just send you some files to diff and integrate.
Nicholas
Posts: 13132

Post by Nicholas »

(Sorry for the wait on this.)

A similar idea came up just a few posts above. That's a "half-the-code, public repository" scheme.

It sounds like you're describing something closer to an "all-the-code, private repository" scheme, right? I suppose that's technically how the project is already operating. The source lives on GitHub and my contract graphics programmer has access to it. That said, I sought out a disinterested, professional third-party graphics programmer on purpose. I'm super paranoid about the source leaking. A programmer that is also an enthusiastic Synthesia user would have an order of magnitude stronger motive to "repurpose" the code for "their own" project.

Maybe it's an over-abundance of caution on my part, but this project is what puts food on my family's table. So I've been ultra-conservative in terms of letting others see the code.
icuurd12b42
Posts: 48

Post by icuurd12b42 »

Out of curiosity, what language and game engine did you write this in? And it is the same for all versions?

I wrote my first midi reader game in the guitar hero days. using FMOD and Game Maker and a lot of code :)
Nicholas
Posts: 13132

Post by Nicholas »

icuurd12b42 wrote: 11-12-21 10:24 pmI wrote my first midi reader game in the guitar hero days.
Me too. (It was Synthesia.) :lol:


Otherwise, here's the little language chart from the private GitHub repo:

languages.png
languages.png (7.46 KiB) Viewed 26155 times

I'm always working to get that Java percentage lower. The Java and Objective-C parts are the thin wrappers over Android and iPad/macOS. The rest is all C++ and the same across every platform.

Until recently, the whole thing was library-less. But when I saw how much lower latency the MIDI synth was that BASS (an FMOD competitor) had than the built-in Microsoft synth, it finally made sense to add it.

I've actually used FMOD in a few ancient projects of my from my college years.
icuurd12b42
Posts: 48

Post by icuurd12b42 »

The MS MIDI, I tried on in the options, is indeed slow to react.

I so the c++ code is pretty much the game and low level interface access on all platforms? That quite interesting of an achievement. multi platform target is hard.
Nicholas
Posts: 13132

Post by Nicholas »

icuurd12b42 wrote: 11-13-21 6:54 pmmulti platform target is hard.
I'm not sure it's worth doing it from scratch, nowadays.

I mentioned some of this in the 3rd post, but a good chunk of my time is spent patching holes in the ever-shifting ground underneath the app, anytime one of these mega corporations releases an OS update.

The reason people like the concertcreator.ai folks have been able to out-compete me (among other reasons like our decision to have two kids just a few years ago) is that I'm fiddling around with the substrate to make sure, say, my vertex format abstraction works the same way across four platforms (in the presence of four different sets of platform bugs). The concertcreator.ai folks probably didn't have to spend three hours implementing "pasting into a text box" four different ways.

It's not very forgiving and mostly all I have to show for it is a smaller app. It's only 18MB instead of like 300MB or whatever. Outside of saving a few dollars in bandwidth costs, I'm not sure that's a benefit to me or Synthesia's users worth anything remotely close to the constant effort it requires.
Post Reply