A Year With Unreal + Git

First, some relevant info about our project

While I can’t reveal much about Believer’s game, there are a few details that are worth sharing for the purpose of demonstrating the scale of our project.

The game is open world, and it uses Unreal’s World Partition feature set.
We have ~50 ICs contributing work.
There are ~300 commits to main per week, touching ~6000 unique files per week on average.

Lessons learned

Now, let’s go through some key takeaways. These are not meant to be positioned as “good” or “bad” - they’re just important observations we’ve made along the way, and we invite you to consider these learnings when thinking about your own source control needs.

1. Focus on delivering workflows, not technology.

We learned very early on, especially as more folks started joining the studio, that despite how important we felt it was to give Git a shot, contributors don’t really care about Git specifically (or Perforce either, for that matter). Our goal needed to be to deliver the best possible experience working on our game, and the technical specifications of our source control backend only represent a small fraction of that experience.

It’s critical to pay attention to how people work - How do they create files? When do they save their work? When do they decide it’s time to submit? What do they do when something unexpected happens? Regardless of the technology involved, it turns out that everyone works a little bit differently, and it’s important to try and enable the workflows that are most important to members of the project at hand.

What this means for Git, in our case, is that there was no way we were going to double down on teaching everyone how to use the Git CLI or any existing Git desktop UI tool. These tools expose way too much of Git’s internals. Contributors want to modify files, select those files, and then submit. We wanted to deliver that workflow to our users, and that’s what led to us starting work on Friendshipper.

2. Telemetry is invaluable for software tools.

For many months, user feedback and communication was our primary mechanism for figuring out what issues folks were experiencing using Friendshipper and its companion Unreal Editor plugin. For a while, this felt sufficient, but as we started adding features we also started seeing reports of issues with performance. To paraphrase a couple:

“Locking hundreds of files at once takes minutes”
“My Friendshipper has been stuck syncing latest for 10 minutes. Should I kill it?”

Recently, we instrumented a lot of Friendshipper’s hot code paths with OpenTelemetry tracing, which allows us to see how long operations take at almost any granularity we’d like. We ship the telemetry to Honeycomb, where we can create dashboards, run queries, and set up alerts.

This instrumentation helped us find a handful of issues that our users were simply “dealing with”. For example, one user reported that it took two minutes for Friendshipper to start up, so we added instrumentation to the startup flow. We found out that our file watching handler was taking upwards of 5 minutes to initialize on some PCs, and the telemetry showed us exactly in which function the issue was occurring. For some ridiculous reasons I won’t get into here, the issue disproportionately impacted non-engineers, so nobody who works on Friendshipper could repro. A fix took the average start time for Friendshipper from ~100 seconds to 240 milliseconds.

The takeaway here is that it’s important to instrument errors and performance metrics in tools. We’ve found that many performance issues will often go unreported, and folks will just “deal with it” as tools get slower and slower and more bloated. Adding OpenTelemetry to Friendshipper has been totally invaluable.

3. No tool was prepared for One File Per Actor (OFPA)

One File Per Actor is a feature of Unreal 5’s World Partition System that saves each actor within a level in its own file. This is really awesome for groups of people who all want to work on the same level at once because this means individual actors can be checked out of source control, whereas previously someone would check out the entire .umap and everyone else would just need to wait.

However, this also means that a user may need to check out and save hundreds of files at once. Say you want to shift a large portion of the map slightly to the right, for example. You’d need to check out and save every actor in the region. In our project, that could realistically mean 500-600 individual files. This means that source control operations on individual files need to be fast, and anything that can process files in parallel should do so.

When we first switched over to OFPA in our repo, our source control operations were unacceptably slow. Checking out hundreds of files could easily take minutes. I’m happy to say that we’ve put a lot of effort into the performance of our Unreal plugin, and now you can check out hundreds or even thousands of files in less than a second.

Another issue we ran into with OFPA was the fact that assets are saved using filenames that are randomly generated, so we can’t rely on information from Git to know which files represent which actors. We got around this by implementing OFPA name translation in Friendshipper, about which the very talented Reuben Dunnington has published a post of his own.

The work continues…

The above list of learnings isn’t meant to be comprehensive. It’s been a pretty wild ride working with Git and Unreal so far, and we’re sure there will be more challenges to come. We improve Friendshipper constantly, and we’re always looking for new ways to help make workflows better for engineers, designers, and content creators. Our tools are all open source and available here on GitHub. Feel free to check them out! We’ve also got the Believer OSS Discord if you’d like to come chat.

Thanks for reading!

rudeboy