Many years ago I set up a disused Pentium 4 desktop with Debian as an emulator console connected to our TV. The system served well enough – I played through all of Chrono Trigger on it, at least – but it has a particular problem, which is that the motherboard is saddled with the pathetic i740-based “Intel 3D Graphics” system from 2001.. At the time when 1024×768 was a high-end gaming resolution, this machine cannot even reach 60fps at 1920×1080 of an HD panel – certainly CPU scaling is a dead end, and in my testing even a “full screen quad” (one 1920×1080 textured 3d polygon, the “accelerated” way to do hardware scaling these days) doesn’t hit the full framerate.

Yet somehow, both Snes9x’s X11 mode and the DOSBox in “Overlay” render mode can manage it. The secret is the “Xv” or “XVideo extension”, where graphics chips enabled a specific route for sending YUV streams to the card, and custom hardware managed scaling to fill almost any screen size. This was, obviously, for video – especially DVD, which is already YUV format, the hardware offload from DVD directly into graphics card into full-screen presentation saved a lot of CPU cycles back in the day. But you could use it with anything producing a YUV format image, including games or applications: Snes9x supports it directly in the unix port, while DOSBox’s SDL1-based system uses functions that ultimately map to this feature.

I’d like to add more emulators to the system, but finding any that use this is a major challenge. Most don’t use X11 direct programming at all, instead depending on a framework like SDL or GTK to provide cross-platform support. And SDL 1.2 was superseded by SDL 2.0 in 2013: the former no longer receives maintenance and is heavily discouraged for new developments. SDL2 dropped support for YUV overlays, a relic of a bygone era before GPUs could easily manage the task. My options seemed to be:

• Add code myself to each emulator to support Xv functions (nightmare)
• Replace the computer with something newer (but why? it’s close to working as is…)

A third option occurred to me though: many emulators also can be built as a “RetroArch core”, which means the emulation guts are contained in a library, and the I/O is supposed to be handled by the “frontend” application instead. The interface between these is something called “Libretro”. Since the emulators communicate in a standardized way, and I supply the video output, could I write a “frontend” that uses legacy SDL1 and displays the frames with its YUV overlay support?

Libretro

The first challenge is finding Libretro. Should be easy, https://libretro.com right? But that site appears to largely contain information about RetroArch, the flagship frontend that emulator cores interface with. There’s a Libretro link at the top, which takes you to some kind of marketing fluff about how you should really use it, please.

The actual, canonical location for libretro appears to be this libretro.h file buried in the RetroArch github repository. I’m kind of surprised at this – I expected the “lib” in “libretro” to stand for “library,” and was hoping it might come with useful glue functions as well… cross-platform shader hooks, scaling algorithms, audio filtering, other reusable components for emulator / frontend devs. But it’s not that, it’s really just the one .h file.

The license for this (specific) file is MIT, which also strikes me as an odd choice. It doesn’t produce code, it’s just an interface definition. I guess a protocol can be proprietary and have legal protections, but I think I’d mostly like this to be public domain or some other “open standard”.

But I digress. The difficulty I faced next is how to use Libretro as a frontend developer. Nearly all the documentation is written on how to make your emulator Libretro compatible, which I guess makes sense, because Retroarch wants to entice more cores and doesn’t want to encourage competing frontends. The official documentation for frontend development amounts to “look at these list of other frontends”. I made the most progress looking at noarch by Rob Loach, a frontend which “does nothing”, but suffices to load a core and expose the required functions.

The general idea of libretro is to define some subset of “things all emulators do”, and then outsource them to the frontend via the interface. For example, “all emulators have a controller”, “all emulators put out video / audio”, “all emulators have savestates” and so on. At the same time, it imposes limitations on the cores so they must conform to some baseline: “all video must be 0RGB565 or XRGB8888 – no exceptions!” and the emulator dev has to twiddle the native machine format into one of these. I don’t even want to know how 3d acceleration passthrough is done – most of this seems to be set up for pushing 2d scenes to the screen.

The libretro.h file itself is complex and confusing. I get the strong sense that this document isn’t particularly the result of thoughtful design, so much as it started with a couple targets (SNES and Genesis maybe), and has since been developed via growing pains. It leads off with #define RETRO_API_VERSION 1 and has apparently never been incremented. There are a lot of deprecated functions as it turned out that some things were not good ideas – the frontend handling “overscan”, for example, which has since been kicked back to the emulator devs to manage. Or the idealized Libretro controller – resembling the Dualshock 2 maybe – which has since outgrown itself as it needs to accommodate more controllers like the Intellivision’s keypad, or wheels for driving, etc. It turns out that the pursuit of accuracy on the part of emulator devs sometimes violates the assumptions made before.

Right, so, how has libretro handled these changing requirements? There are, in fact, only six callbacks the frontend needs to implement in order to work:

void (*set_environment)(retro_environment_t) = NULL;
void (*set_video_refresh)(retro_video_refresh_t) = NULL;
void (*set_input_poll)(retro_input_poll_t) = NULL;
void (*set_input_state)(retro_input_state_t) = NULL;
void (*set_audio_sample)(retro_audio_sample_t) = NULL;
void (*set_audio_sample_batch)(retro_audio_sample_batch_t) = NULL;

That’s two ways to read input, one way to send a video frame, two ways to send audio, and one to do “environment” work. What’s happened to support all other needs (get directories, console reset, ask about microphone presence, etc) is an extension of the set_environment() function to cover everything else. The function takes a number (#define) and then a void * to the rest of the parameters for that type. So, for example, set_environment(RETRO_ENVIRONMENT_SET_MESSAGE, “hello”) tells the frontend to show “hello” on-screen for a time, while set_environment(RETRO_ENVIRONMENT_SET_NETPACKET_INTERFACE, &retro_netpacket_callback) gives the core the ability to talk to the network somehow (?) Originally meant for reading / setting options, it becomes the majority of the API.

I’m not an expert but I find this ridiculously overstuffed interface extremely inelegant. It’s hard to follow the documentation, and though it’s “stable” (old cores should continue to work with new frameworks), it’s also really hard to tell when or what new features were added. Commit history has updates every couple months, but every message is just “Resync”, and provides no details on what new features were present. I think this is a side effect of the Libretro’s Patreon-based development model: new development is actually done elsewhere and only periodically pushed to Github, and the .h extensions are done ad-hoc as needed by whatever core the devs are currently chasing. Retroarch is not only the “flagship” but, again, the team does no favors to people wanting to develop other “competing” frontends.

Result

All that said, Retroarch’s efforts have convinced a number of emulator projects to adopt the “core” system, and so a lot of folks have pushed their square pegs through this particular round hole over the years. And indeed I was able, with a few hundred lines of code, to make a frontend that opens an SDL1 context, talks to SameBoy and then forwards its screen to the output driver. YUV works, but also I can pick different output types – here’s Tetris running on my headless server, with aalib graphics 🙂

With this I should be able to bring a bunch of other systems to my emulator box, and despite its awkwardness, the entire point of this exercise was to not have to repeat the work a bunch of times. The remaining thing is that as I want to bring new cores in, they may make requests to set_environment() that I haven’t yet implemented, and I’d have to occasionally update it to support more features. But that is relatively trivial.

There is a constant tension in game emulator devs / audiences between those who just want to play the games, and those who want to understand the machine – and Libretro has planted itself pretty firmly in the former camp. I am trying to imagine e.g. building a memory + CPU debugger in libretro, which is missing dialog box primitives or other UI elements, and it seems like it’d make things much harder than just using Qt or something. But their argument would be, most users don’t care about that! They want the controller buttons to work the same across multiple systems. So maybe it’s designed how it is because of this guiding principle.

An Alternative

Astute readers may have noticed, when I mentioned aalib above: why didn’t I have to rewrite my frontend to pick a different output type? The answer is SDL’s “driver” system: you may choose, at run-time, which output method you want to use for video or audio when running your SDL application. There exist a bunch, for different subsystems (wayland vs x11, alsa vs oss) but also new ones can be written to take advantage of unique hardware (iOS screen) or oddball uses like outputting to remote desktop, terminal, etc.

Since a bunch of emulators are currently SDL2 powered, it might have made just as much sense for me to pursue writing an SDL2 driver that uses Xv under the hood – it could take a frame buffer and push it into the Xv extension, and I could effectively do whole-screen acceleration with that. In that case, I wouldn’t have to implement all the things I don’t care about, like answering the “rumble support?” request from a core. It would strictly be working with the video frame and its output.

Plus, SDL is a library that comes with a ton of glue helpers – it has fast pixel conversion, for example, and so if your system is really 8-bit palette SDL can handle the conversion to true-color for you. Maybe I will try that another day, especially for emulators that don’t have a libretro core but do run on SDL2 (the Stella Atari 2600 emulator, for example).