James Mackenzie's Blog

James' Blog

Blog   Projects   About   Sitemap

Projects

Project summaries for completed and in-progress projects below. Enjoy!


Projects


Framerate Analysis Tooling

Status In-Progress
Goal Build my own “Digital Foundry” style tooling to analyze videogame performance

Inspired by Digital Foundry, VG Tech and NX Gamer, I wanted to deep-dive on videogame technical analysis. I set myself the following goals:

  1. Write my own performance analysis tooling. Look at captured videogame footage and analyze framerate, torn frames, resolution etc
  2. Run some real performance comparisons (e.g. Xbox Series X vs PS5 performance)
  3. Technical learning! Build some proficiency in videogame technical analysis and machine vision

After some background research, I selected OpenCV as the foundational project library. As a specialised framework for computer vision and image analysis, it’s a great project fit.

First, I worked on difference detection between video frames. If two frames show zero difference, that means we have a duplicate frame (i.e. a frame drop vs the target frame rate).

With diff detection in place, I was able to do some rudimentary framerate analysis on The Legend of Zelda: Breath of the Wild and (after some more iteration) add performance monitoring and graphing.

Next, I investigated detection of torn frames. This is when the renderer doesn’t have enough time to complete an entire frame, so a partial frame is rendered to the screen instead (with the remaining row gaps filled in with data from the last frame). I enhanced diff detection and was able to reliably detect torn frames in Gran Turismo 5 (a game known for significant image tearing).

My next focus was resolution counting. Videogames often dynamically reduce rendering solution to keep frames “within budget” (i.e. rendered fast enough to hit the desired 30fps or 60fps target). This is known as Dynamic Resolution Scaling (DRS). The frame is then upscaled back to the target resolution (e.g. 1080p or 4K). It’s an effective approach, but reduces image quality vs “native resolution” rendering. Artifacts (such as jagged edges) appear in the image, and it’s possible to exploit these to identify the original render resolution of the frame.

As a first pass, I was able to find all edges in each frame. I was then able find “countable edges” - straight line edges that can be pixel counted to identify the original render resolution of the frame.

Unfortunately the automation didn’t work. Despite hours or tuning, the edge detection algorithm wasn’t able to distinguish between polygon edges (which are suitable for counting) and straight line edges found within filtered textures (which aren’t). The counts are unreliable, so I’ve parked this for now.

Next, I looked at framerate counting for compressed video. Video compression is typically lossy and introduces image artifacts e.g. pixelation, macroblocking etc. These artifacts make it very challenging to detect duplicate frames. 100% identical frames rendered by the game engine will appear different in compressed video.

In short, compressed video broke my diff detection. Every single frame is perceived as unique - no frame drops detected. To address this, I needed to adopt a more statistical approach. Instead of looking for zero difference between frames, I looked at difference trend:

  • Hypothesis: because video frames from a 3D engine are temporally connected, we would usually expect frame difference trend to be relatively consistent between frames. Significant trend divergence is aberration - and might indicates a frame drop

To test the hypothesis, I created a time series of frame-to-frame difference (Euclidean difference) for compressed video with known frame drops:

See those downspikes in the graph above? Those are the frame drops. And they’re easy to identify. This is great news!

A few updates later, I was able to complete framerate analysis for the Starfield Official Gameplay Reveal.

Finally, I looked at high quality video export from OpenCV. Unfortunately the built-in OpenCV VideoWriter is not super configurable - and video output is relatively low bitrate and blocky. Instead I chose to output results as PNG and then re-assemble into video using ffmpeg.

Challenges overcome:

  • Sampling framerate from captured gameplay video
  • Identifying duplicate frames and torn frames
  • Graphics and performance graphing in OpenCV
  • How to accommodate compression noise in captured video
  • High quality video export from OpenCV

Upcoming goals:

  • Side-by-side comparison of 2 (or 3) videos
  • Resolution counting revisited

You can also find the journey chronicled on Twitter here.


HP Thin Client DOS Gaming PC

Status Completed July 2021
Goal Build a small and light gaming PC for DOS and Windows 98 gaming

After DOS gaming on MiSTer, I was left wanting for more. The MiSTer felt underpowered, I wanted something a little faster to play Half-Life, Unreal and other Windows 98 classics.

In search of a retro PC (but short on apartment space), I purchased a HP Compaq t5710 thin client.

Despite the 2004 build, the hardware is very well suited to DOS and Windows 98 gaming. Sound Blaster and Ad-Lib audio works even in DOS mode, making this a great all-in-one unit.

After some trial and error, I was able to succesfully install and game on Windows 98. I was also able to find a working PS/2 splitter.

Next, I experimented with the PCI expansion slot. After some success with PCI sound hardware, I installed a 3dfx Voodoo2. It worked fantastically too!

I’m super happy with this build. The t5710 is now my go to retro PC – I don’t need anything more!

Posts:


DOS Gaming on MiSTer FPGA

Status Completed March 2021
Goal Play DOS and Windows 95 games on the MiSTer FPGA platform

Having recently acquired a Terasic DE10-Nano FGPA Development Kit, I was very keen to install the MiSTer software distribution and try some retro computing cores.

First on the list: the 80486 PC. I’ve many fond memories of DOS and Windows 95 gaming, messing with IRQ settings, tuning config.sys and autoexec.bat to get things working “just so”. Luckily for me, DOS gaming is very well served on MiSTer via the ao486 core.

After a lot of tinkering, I was able to setup file sharing, install a Memory Manager, configure the Sound Blaster, add CD-ROM and Mouse support, configure Roland MIDI and install Windows 95. A great result!

Posts:


Remote Play Xbox Series S/X Games on PC

Status Completed January 2021
Goal Stream / Remote Play Xbox Series S/X games from my Windows PC

With the Xbox One, I often used the Xbox Console Companion app to remote play games on PC. Frustratingly, this feature was silently dropped for Xbox Series S/X – the Console Companion app just doesn’t work for the new consoles.

Looking for a solution, I was able to install a hidden “Xbox Game Streaming (Test App)” from the Windows 10 Store, and start streaming again. Full details here. Case closed!

Posts:


Lenovo Tiny Series Emulation Station

Status Completed May 2020
Goal Repurpose a Lenovo Tiny Series business desktop as a small and quiet retrogaming PC

For many years I’ve been on quest to find the perfect emulation station. Something small and quiet, hidden under the TV, with a wireless controller and enough horsepower for high quality retrogaming.

Two years ago I built and custom-configured a Raspberry Pi with RetroPie. A great device, but not really powerful enough for 32-bit consoles like PlayStation or Saturn. I need something with a bit more grunt, reasonable price (~$100) and good Wife-Acceptance-Factor (no tower PC!)

After some investigation, I purchased a Lenovo ThinkCenter M93p Tiny Desktop from eBay for ~$100.

The form factor looks great under a TV and emulation performance is also great. Even Dreamcast and Gamecube run well, a significant step up from the Raspberry Pi. Full write up here.

Posts:


Porting Commander Keen to PlayStation Vita

Status In Progress
Goal Bring the DOS classic Commander Keen to PlayStation Vita.

I’ve wanted to explore cross-compilation (i.e. building code on one platform for execution on another) for a while. I’m also passionate about bringing classic retro games to a new audience.

With the advent of h-encore, pretty much any PlayStation Vita can now be unlocked to run homebrew code. There’s also the Vita SDK development framework available to help create working binaries. So the Vita seems like a great target platform.

I’m aiming to bring Commander Keen (in the form of CloneKeen) to the Vita. The source code is C/C++ and uses the multiplatform SDL development library, so should be relatively portable. I’m also looking for improvement opportunities over the orignal release (e.g. level editor, increased resolution).

So far I have the game engine running, but need to add controller support.

Posts:


Porting Commander Keen to WebAssembly

Status Completed November 2019
Goal Port Commander Keen to WebAssembly, making it playable in the Browser

I’ve wanted to find a creative outlet for my recently-acquired WebAssembly skills. I’m also passionate about bringing classic retro games to a new audience.

I used Emscripten to port Chocolate Keen to WebAssembly. The initial goal was to refactor the entire code base to Emscripten Loops but this proved too much effort. In the end I found a working (but slower) solution using Emterpreter.

I’m happy with the end result. Play Commander Keen in the browser right here, find instructions here and source code on GitHub.

In future, I could refactor towards Emscripten Loops to improve performance and add on-screen controls for mobile devices.

Posts:


Learning About WebAssembly

Status Completed November 2019
Goal Learn about WebAssembly. Understand how to make C/C++ code run in the Browser. Code some “Hello World” examples and document my findings

I’ve wanted to explore the WebAssembly ecosystem for a while. I learn best by sharing what I learn, so I’ve written up my findings as WebAssembly Lessons:


Raspberry Pi Retrogaming

Status Completed October 2018
Goal Build a cheap, small and light device to retrogame on the big screen

I’m a big fan of retro games, with many fond memories of the 8-bit and 16-bit years. Looking for a way to relive those memories on the big screen.

I selected the Raspberry Pi and RetroPie as the base for my build. After a week of on/off tweaking, I found optimal settings and documented the full setup here. For Phase 2, I configured Moonlight Game Streaming to allow low latency streaming of more demanding titles from my gaming PC.

Overall very pleased with the result, and pleasantly surprised by how well Moonlight Game Streaming works.

Posts:


Atari ST on the Web

Status Completed June 2018
Goal Connect my venerable Atari ST to the World Wide Web. Browse some sites!

The Atari ST was released in 1985. Is it usable on the modern Web today? How far can I push the experience?

Starting with a NetUSBee Ethernet adapter, I installed TOS drivers and the STinG TCP/IP stack. I also used an ET4000 graphics card to push the resolution to a (somewhat) modern 1024x768 in 256 colours.

With those (and a successful TCP/IP ping) in place, I downloaded and configured Crystal Atari Browser (CAB).

The Web experience is slow and doesn’t support CSS - which rules out a lot of content. However I was able to make older sites work.

In search of a better experience: it might be possible to use a proxy server to strip CSS and shrink images. Or use a text-based browser like Lynx. Browsh also looks interesting. Lots of avenues for future investigation!


Shooting and Sharing VR Photos

Status Completed November 2016
Goal Learn how to shoot, interactively view and share VR photos

Ever seen one of these photos where you can interactively zoom and pan around? They’re known as 360° or VR photos.

More formally, VR photography (Virtual Reality photography) is the art of capturing or creating a complete scene as a single image, as viewed when rotating about a single central position

I want to learn how to shoot, interactively view and share my own VR photos. Here’s what I found:

Posts:


Transferring Files Between PC and ST

Status Completed March 2016
Goal Explore the various ways to share files between PC and Atari ST

Can my Atari ST interoperate with modern PCs? What are the best ways to send files back and forth?

I explored several ways to transfer files between PC and Atari ST, including:

I also took a detour into file compression, and learned how to split large files into chunks that can be individually copied using (for example) floppy disks:


Playing Downloaded Games on Real ST Hardware

Status Completed September 2015
Goal Download an Atari ST game image from the web and play it on my real ST

My old ST disks are dead! Can I download replacement disk images from the web and play on my physical ST?

I explored several options, including floppy disk images and hard disk adaptations:


Watch Mobile Content on the Go

Status Completed December 2014
Goal Download video content to my mobile device so I can watch it on the go

Update Dec 2019: YouTube has emerged as the dominant platform for Web video. In future, it makes sense to look at the YouTube API. Can I automatically download every video tagged to the “Watch Later” playlist for offline consumption?

The work commute is a great time to catch up on news, vlogs and other Web videos. But cellphone signal is way too patchy for reliable streaming - especially in the subway.

I explored use of the Pocket API and PlexSync to download and push web videos to my phone:


Ghost Blogging

Status Completed June 2014
Goal Learn how to use the Ghost blogging platform

Update Jun 2018: I now use Jekyll and GitHub Pages as my blogging platform.

I’m interested in Ghost as a crisp, clean, lightweight blogging platform. I took some time to learn about the platform, setup my own developmemt environment and even write my own theme. Findings below.

Posts: