Emulator development has never been simple, but contemporary emulators face a tremendous set of challenges created by the modern era. Having witnessed some of the trials developers have gone through personally, I feel it's necessary to try and give an idea of what I've seen and my perspective on the situation. Some of the following topics are rather touchy, so please try to keep an open mind while reading.

My Short History Within Emulation

I joined the emulation development community mid-2013. Compared to many of the stalwarts of emulation I've met throughout the years, I'm a mere toddler. I'm not a coder and I'm definitely not a genius. I'm just a busy body that tests games and likes to write articles.

What invited me into emulation was Dolphin and the idea of fixing up netplay so that we could get games like Super Smash Bros. Melee working consistently on netplay. With help from my brother, who is an accomplished software engineer now, we worked on netplay issues, asked developers questions, and even released our own modified builds with the goal of eventually getting our changes merged into master.

Our branch was never fully finished, but others joined the cause and netplay saw great improvements from many developers. Nowadays, the situation is far beyond my wildest dreams, and I can play pretty much any game I want on Dolphin's netplay. There are entire communities built around Dolphin's netplay, including websites like smashladder which are far beyond the ambitions I held back in the day.

Back in 2012/2013, working netplay was little more than a pipe-dream.

During my work on Dolphin netplay, I realized I really enjoyed working on Dolphin and testing these games. There was something cathartic about digging into a game and figuring out what it was doing, how it was doing it, and how Dolphin was emulating it. The feeling of accomplishment when you finally get it and figure out what's going wrong is incredibly powerful.

Through constant testing, I began to build up my now encyclopedic knowledge of weird game behaviors and how Dolphin handles them that I have today. That knowledge would then come into play as I became one of the writers on the Dolphin Progress Reports. Thanks to the patience and kindness of developers, I know Dolphin well enough to write full length articles on hardware, how games take advantage of that hardware, and what makes things more difficult to emulate.

Learning enough about the GameCube MMU to write about Dynamic BATs was incredibly difficult.

It may not seem all that surprising nowadays, but back then I never thought it'd be possible to contribute to an emulator as a non-coder. But other skills come into play, and I've contributed to projects like Dolphin, mGBA, Citra, yuzu, RPCS3, and others. Dolphin itself has definitely influenced the community on the whole as the idea of Progress Report style monthly updates and blog teams have become fairly common.


The Core Emulator Team, Open Source and Forks

Since the Nintendo 64 and PSX era, emulation development has become about collaboration. The paradigm quickly shifted from plugin based emulators to open source emulators and that's where things have mostly settled for over a decade. After all, open source offers what one would think to be the ultimate advantage of allowing anyone with any skill set to look at the emulator and modify it.

Regardless of how many contributors a project has, there tends to be a core emulation team that manages the project. Some are more active in development than others, but they all care deeply about the project and will often weigh in on decisions and fight against or for particular changes. While everyone wants to improve the emulator, the core team can deeply affect the direction of the project.

A strong core team and good decision making is the key to keep a project maintainable and active. Considering that the master branches are the ones that most people contribute to and fork off of, it's especially important to make responsible decisions for the health of the emulator and community on the whole.

Commit history comparison between Dolphin (left) and Ishiiruka (right). Dolphin has a cleaner commit history to show exactly what every change does for people who want to modify Dolphin. Ishiiruka is developed by a few people who are familiar with their changes and thus don't need to spend as much time documenting them.

There are actually thousands of forks of Dolphin that are based off of this master branch. While the core developers are very careful about what gets merged into master, people can do whatever they want with their own fork, resulting in many specialized versions of Dolphin.

When open source works, the results can be incredible. Anyone can fix, create, or implement anything while having unparalleled access to the internals of the emulator and the software it can run. You don't even need programming experience to contribute, with many people contributing through services like transifex to help translate and increase accessibility of emulators around the world.


A New Generation of Open Source Emulation

I was introduced to Citra long before its initial public release. Since developers worked with me for the blog, I had a bit of an inside line on things they were working on. So imagine my excitement when one of the developers showed me the beginnings of a 3DS emulator as it booted its first game!

This image dates back to 2014, right around when Citra was announced and prompted much excitement.

At this point, Dolphin's Progress Reports had become extremely popular with users and this new emulator, Citra, wanted to do a similar thing. For me? It seemed like a match made in heaven. Unlike Dolphin where I joined it as a fairly mature emulator, I'd be with this one from day one. I was one of the only non-coders within their development channels before the initial announcement.

Everything at the beginning was exciting with Citra. It felt like this new project with new games to test and new bugs to discover, report, and write about. The first few months went by at a crazy pace, with performance, compatibility, and feature-set expanding at an incredible rate.

Yet, something was off about the project. I didn't know exactly what, but, I didn't have the same devotion toward it that I did Dolphin. Part of the problem actually had to do with me. I had become specialized to how Dolphin worked through years of testing and working with developers. I didn't put forth enough effort to learn about Citra and developers were too busy still discovering new things about the 3DS to waste time walking someone like me through concepts that would normally be far beyond me.

In the articles and videos that I did make, I was rather vague because I didn't quite understand how things work. I would constantly compare things to how Dolphin worked, because that was my only frame of reference.

The thumbnail to this video should say everything you need to know.

If things had proceeded normally and Citra stabilized like Dolphin did, I probably would have adjusted and eventually figured things out. Unfortunately, I didn't last quite that long with the project.

A Userbase At War with Developers

One core thing to remember is that developers care about their projects. While some of the minute details may vary, in the end most of the people working in emulation are working to preserve hardware and/or games in their own ways. It doesn't matter if that means working on the ultimate enhancements, pixel perfect accuracy, or improving performance - they all care.

Some users don't seem to understand that. They seem to think that these emulators are purposefully crashing or too slow just to frustrate them. That their Pentium IV is newer than a GameCube, and thus should be able to emulate a GameCube regardless of what these people who have worked on this emulator for decades say.

That isn't to say users are bad. Users are necessary as many of these emulators support giant libraries of games. They report bugs that developers would never know about otherwise, and some of those users fix the bugs themselves and join the project as a new developer!

I know it's wrong, but, I've lost my temper many times when working within the support forums. I understand that many of these users just want to load up their favorite game and have it work. When you deal with thousands of reports, even if only 10% of them are problematic, they start to weight on you.

Users like this are fairly rare on Dolphin's forums but became the normal for Citra for a time.

Working on Citra's social media and forums was an absolute nightmare because it seemed like everyone hated us. The desires of the userbase and developers shifted so far apart that it bred vicious malcontent and distrust. Before you could even answer a question on a thread, another user would have already replied that the developers don't care. It didn't matter if it was on reddit, facebook, chat logs, or anywhere else, it felt like users hated us.

What caused this massive divide? Because users wanted to play Pokémon now. Okay, that's a slight exaggeration, it wasn't just Pokémon, compatibility with a few popular games was slow in Citra due to some rather difficult features to emulate that they relied on. Pokémon fans only stood out because they were the most vocal of the annoyed fanbases.

Even when unrelated features were posted, users were pretty adamant about the things they wanted.

Early on in Citra it started out with just asking for the games to boot, but the longer things went, the more tense things became. Simple questions became more annoyed as they disregarded the projects efforts until this one, single, thing was supported.

I don't know if they thought that downplaying everything they didn't care about would move developers toward something they wanted, but it had the effect of making it harder to work on anything.

Third Party Builds, Licenses, and the Placebo Factor

At this point, I was already distancing myself from Citra. So much of support on the forums was just banning people for downloading games or telling them Pokémon didn't work yet. It's not exactly fun to play the bad guy, and while some users understood, others absolutely didn't. Compared to working on a project beloved by users like Dolphin, working on Citra was stress I didn't want in my life. I still stayed around the development team though, as my issues weren't coming from them.

From a distance, I witnessed as things went from bad to worse. If you've followed Citra closely enough, you probably already know about the "third party" builds. This isn't talking about every fork, but a specific kind of fork with a very specific purpose.

They were designed to prey upon an impatient userbase and violated Citra's open source license without any repercussions. By taking incomplete features posted by developers for review/testing and mixing them with egregious hacks that would make others cringe, these third party builds created a reputation among users as being "fast" builds. It started with HLE audio support but quickly spiraled out of control with any unmerged feature becoming open game.

The unofficial builds leeched incomplete features and built a sizable following of dedicated users. The "Dynarmic" was publicly revealed on 2016-09-01 for testing, BUT, was not merged into master for weeks.

This put extra emotional strain on the authors as they watched these forks get credited for changes that they sometimes spent months or even years researching and implementing. Some of these changes were incrementally put up in smaller chunks over the course of months in order to maintain a clean history, which as we mentioned above was incredibly important for the master branch.

These good development behaviors gave third party forks the opportunity to get these features much earlier than master, even if they were buggy and incomplete.

So of course developers disparaged these builds and told users not use them. Other than being hack builds, they violated Citra's GPL license by modifying the emulator and distributing binaries without releasing the adjusted source code. But here's where it gets a bit fucked up. Users trusted these builds that violated Citra's license over Citra's core development team. It was incredibly easy to see on the forums, as users would constantly recommend these builds with faster audio or the new CPU core. The weird thing is that they knew how buggy it was, and always have to give extra instructions on what games didn't work or how to bypass various bugs, but it didn't matter.

To them, these third party builds were the ones made by people who really cared about the users, unlike the core Citra team.

Many popular games, such as Pokemon X/Y, Mario Kart 7 and others took a while to get running correctly due to rather difficult behaviors to emulate that developers had to spend time reverse engineering.

It was an incredibly difficult trend to stop. Third party build developers would link to videos with obviously faked performance and resynced audio whereas the core developers obviously could only use real results and footage.

There were even experiments done, by Citra developers posting builds with no changes at all claiming they were built with similar techniques to the third party builds. Unsurprisingly, they claimed it had fixed all of their issues. To make matters even worse, some opportunists took advantage of users' willingness to download third party builds and packed their own variants with malware.

Striking Back With Bleeding Edge and Canary

The core Citra team had to go on a huge campaign to restore their image within the community and combat the third party builds. Changes began being developed in secret in order to make sure the official builds were able to get the feature before anyone else. This effort was incredibly taxing, but necessary in order to keep the main builds relevant.

In the end, the entire infrastructure behind Citra went through a radical overhaul. A complicated system was devised to automatically merge together important pull requests and automatically make them available alongside standard development builds. This started out "Citra Bleeding Edge" and eventually was replaced by "Canary" builds. By allowing users to test these builds before they were merged into nightly builds, they were effectively able to get the jump on third party builds.

This system took a tremendous amount of effort and time, but eventually allowed the project to stabilize to where it is today.

Canary Builds were a necessity caused by people leeching unfinished changes.

But what most people don't realize is that things almost went in a very terrible direction. Imagine if these third party builds had worn down developers further. Imagine if the future of 3DS emulation was suddenly stopped by these people who had no idea what they were doing, didn't care about anything else but making a single game run better.

Some users don't realize that these third party builds were 99.9999% work done by the core developers and some minuscule, broken changes done by someone who didn't have much of an idea what they were doing. Without Citra's team, these builds would also die out. Citra can exist without the third party builds, but the third party builds cannot survive without Citra remaining active.

The scars of this battle linger with many of the developers who worked on Citra. A lot of the same principles and protections exist in yuzu to prevent a similar situation from happening.

Nowadays I think back and wonder... was there a better way to handle things?


Rise of the Closed Source Emulator

October 13th, 2015, a Wii U emulator showed up that would shake the emulation community to its core. Unlike many other emulators of the era, Cemu decided to make itself as a closed source emulator while also giving out builds early over Patreon.

This threw up a bunch of red flags, and I was particularly very critical of this decision. I vowed to do what I could to help decaf, the open source Wii U emulator, while disparaging Cemu every opportunity I got at first.

I've said a lot of mean things over the years and I honestly regret them. So to the developers of Cemu and the people that enjoy the project, I apologize. I can't really justify my actions or words as I think back on them and I can't take back what I've done. All I can do is try to explain why I now believe Cemu's position as a closed source emulator may not be harmful toward our end goals of preserving gaming history.

Despite swearing to never use Cemu years ago, I finally spent some time figuring out how to set it up.

Unlike Citra, which had to constantly battle with third party builds and hack builds, the Cemu team controlled their brand extremely well. While I do think it's a bit excessive to have something like anti-piracy for week early builds, considering the amount of effort Citra had to put into combating third party builds, it doesn't seem quite as ridiculous.

When Cemu was announced, I really couldn't figure out any good reasons why you'd want to be closed source. How that could ever be better for emulation and preservation?

My conclusion was that it was purely for greed. To reinforce my opinion, Cemu turn out to be one of the most lucrative emulators ever with its Patreon reaching well into the tens of thousands of dollars per month. But is that really a bad thing? Perhaps if emulation suddenly stopped after Breath of the Wild was mostly emulated, but the emulator has continued to make strides.

At this point, Cemu has become an excellent emulator with high compatibility, performance, and even online support that are going to assist with preserving the online features of Wii U games. Patreon has become a staple of most modern emulators with yuzu even going for a similar early release model to Cemu.

Despite being Open Source, yuzu now provides preview releases to supporters for an early look at new features. In a way, Patreon releases are able to harness the same user desires that caused so many problems in Citra.

Despite being an incredibly strong emulator, Cemu still faces a ton of hatred and vitriol. A lot of this is directed toward the fact that it's a closed source emulator. Much like how third party builds derailed conversations about Citra, Cemu's release posts end up plastered with mean spirited comments about it being a useless closed source emulator.

I remember being a part of that and feeling like I was totally justified. Now I feel like I was just being a jackass. Even if you disagree with what Cemu is doing, is that how you convince someone that another way is better? If anything, it makes the people calling for them to go open source look bad.

At this point, Cemu's popularity has died down a bit as some users have moved onto newer emulators. That said, it still remains the fastest and most compatible Wii U emulator by a sizable margin over decaf. Despite the fact that people (...including myself in the past) have tried to pit the Cemu and decaf developers against each other, they don't seem to carry malice toward one another.

Going up against a titan can be frustrating, and decaf developer exjam has expressed frustration about coexisting with a closed source emulator targeting the same console. Fortunately, both teams have been willing to share information and it's fairly clear both care about emulation and the Wii U console because of their continued dedication toward their projects.

Here's one last thought on Cemu's position. Some of the currently dominant emulators like Dolphin and Project 64 started out as closed source projects before going open source. In fact, even Dolphin was closed source for longer than Cemu has been around at this time. Maybe it isn't such a bad thing for an emulator to be closed source while it establishes its identity. Then once the early development, structuring, and userbase is established, the project is more capable of fending off any malicious third parties.

Much like with other emulators, there's no saying Cemu can't go open source later on. I really do hope Cemu goes open source in the future, decaf has already proven useful to Dolphin with some of its documentation on Wii U behaviors that ended up true about the Wii as well. Even beyond that, Cemu is an incredibly popular and versatile emulator that should see use for many, many years to come. I don't think anyone wins if the emulator ends up locked to obsoleted platforms because no one can maintain it after development has ceased.

But that is purely the responsibility of the Cemu developers. Right now, we have two Wii U emulators that can boot retail games very early on in this era where games quickly update and services die out faster than ever. So, to all of the developers of Wii U emulators and those that have supported the projects, I thank you for your efforts and contributions toward emulation.


Emulators in the Modern Era

Whether or not we like it, the situations with both Cemu and Citra have greatly changed the landscape of emulation. Cemu is an incredibly successful closed source emulator that has given other projects a template to follow toward becoming successful. Citra is an incredibly successful open source emulator that has shown both the difficulties of open source emulation and the resilience of developers.

Every project is going to have its own trials, challenges, and decisions. mGBA isn't going to have people trying to leech users with third party builds, but it's also a second generation GB/GBC/GBA emulator with many alternatives. There isn't nearly as much demand from users for a new emulator.

mGBA has tons of innovative features that can make GBA games look better than ever.

However, some older emulators do run into similar problems as new demands come up. PCSX2 has its own third party, closed source build problem stemming from users wanting the emulator on a new platform. The situation is even more annoying than Citra as these developers are selling this closed source fork on the Play store with zero repercussions.

In these cases where a fork is problematic, there's really not much in the way of recourse. Some of the biggest open source projects like Firefox can actually defend their trademarks and copyright, but they're the exception. An emulator is often times already considered a legal gray area and very few people care to stand up for their rights. Even if an emulator was in position to fight, most developers I know would rather just work on emulation than deal with superfluous problems.

Going closed source does almost entirely mitigate the existence of third party builds. However, it doesn't ensure that you're able to establish yourself freely. For as long as I've known, there have been fake websites popping up to try and siphon ad-revenue from legitimate software. Even Arisotura, the developer behind many emulators, including melonDS and BlargSNES, ran into that issue frequently with their earlier projects! It's surprisingly common - when I was researching my melonDS 0.8 article, figuring out which website was the official DeSmuME website was rather annoying!

Regardless of the choice, developing and establishing an emulator isn't a trivial task.

At a Crossroads

For the longest time, I was certain that going open source was the only reasonable option. When Cemu arrived as a closed source emulator, it challenged everything I knew and I reacted with extreme negativity. It took emulation in a direction I didn't like. But all things said, Cemu has absolutely thrived and perhaps that alone justifies the developers for taking things in that direction.

Cemu has undoubtedly had a huge effect on the emulation community at a larger scale. Their use of Patreon is now commonplace among most emulators. Citra's situation has also affected other projects, as I've seen a lot more caution and secrecy within projects regarding new features.

One of the premiere Switch emulators, yuzu, has been undoubtedly influenced by both projects. As its developed by some of the same developers from Citra, they use the Canary system to combat potential third party builds from getting the jump on the newest features. In addition to that, they also allow even earlier access to specific features to patrons through Patreon builds, which seems to be inspired by Cemu's Patreon builds.

For me, this is all a very precarious position to be in as I've criticized many of these projects over the years for their decisions. In all honesty, things are much more complicated than I original gave them credit for and there oftentimes is not a completely correct way to do things. Every project and situation is unique.

Because of that, I'm done being upset with developers for their choices on where they take their projects. These people put a huge amount of time and effort into emulation and their work is worth its weight in gold. Right now, they deal with tremendous amounts of stress and many still show so much patience in spite of it all.

We might not be able to fix everything, but we can make the emulation community a better place if we're more respectful to one another and to the people that dedicate their lives toward the emulation and preservation of games.