Rudolfs, Fork, and the Project Borealis UEGitPlugin

This is part 2 of a devblog series detailing our journey using Git with Unreal. Click here for Part 1 where we discuss how we came to that decision, and join us on Discord to talk open source!

In this post we’ll take a look at our first functional end-to-end workflow using Unreal and Git. Although we’ve made a lot of changes since we got started, a lot of this workflow is still in use today. Let’s get into it!

Our first pass at a tech stack for Git used Rudolfs as an LFS server, Fork as a desktop GUI, and Project Borealis’ UEGitPlugin as an Unreal Engine source control plugin. Let’s talk about how we got there.

Rudolfs

Choosing a Git LFS server is somewhat of a daunting task. Check out this list of implementations published as part of the LFS documentation. We categorized these into three groups:

Commercial offerings as part of a larger Git platform (GitHub, Bitbucket, etc)

Generally fully-featured LFS implementations
Keeping LFS close to the actual Git repo is convenient
Unbelievably expensive – pricing tends to be based on storage + network transfer and they’re priced significantly higher than the storage + transfer of the cloud systems that probably back them (AWS S3, et. al).

Open-source offerings as part of a larger Git platform (Gitea, GitBucket, etc.)

These tend to not separate their LFS server from their Git server, meaning without some hackery we’d be hosting an entire Git platform.
We were highly confident we wanted to use GitHub for its rich feature set, so installing and managing an open-source Git platform was not interesting to us.

Open-source LFS server implementations (Rudolfs, lfs-server-go, etc.)

Some of these are missing key features like file locking support or authentication
Some of these have not been updated in 5+ years

We were confident our selection was going to come from category 3 – the commercial offerings were far too expensive and we had no interest in using, deploying, or operating an entire Git stack. So, after playing around with a handful of open-source Git LFS servers, we chose https://github.com/jasonwhite/rudolfs!

What we liked about it:

S3 backend – We were already using S3 for a handful of things, and our team’s comfort level with S3 in general is high
Written in Rust, our tools language of choice
Somewhat regular updates, mostly to dependencies
The code itself is simple and straightforward – We felt confident we’d be able to add or fix stuff ourselves as needed

However, Rudolfs was missing a few things:

File locking support – we would need to implement the file locking API and design a lock storage mechanism ourselves
Authentication / authorization support

These two missing features would definitely take some engineering work, but our team was sure we could implement them in a relatively short time.

Decision made: Rudolfs for our LFS backend!

Fork

Many folks at our studio were unfamiliar with Git and its workflows, and we knew that onboarding everyone onto the Git CLI was not going to be a sustainable option. After all, when it came to binary Unreal filetypes like .uasset , there’s only a distinct subset of Git features we needed folks to use (more on that later). We were already confident we’d be building our own tools for interacting with source, whether through a UI we built or Unreal Editor itself, but in the meantime, we wanted folks to have a GUI to use, and so we asked the team to try out Fork.

We had no expectation that this would be a long- or even medium-term solution. It exposes a lot of Git functionality, and would be very useful even for Git power users, but that’s not what we felt we needed. It worked great for helping designers add, push, and sync files from the Git repository though.

Project Borealis UEGitPlugin

As our content creators do a lot of work in Unreal Editor, we considered it a high priority to be able to interface with source control from there. Unreal Editor provides some interfaces for implementing a SourceControlProvider which in turn will allow designers to check out, modify, check in, and see source control status for various project files.

Project Borealis is a fan game project looking to create Half-Life 2: Episode 3. A lot of their Unreal tooling is open source, including their LFS-enabled Git plugin, https://github.com/ProjectBorealis/UEGitPlugin. This plugin was relatively easy to install and use in our own Unreal project, so our designers were up and running quickly.

The team identified a few issues with using the plugin, some of which had more to do with Unreal itself than the plugin’s code or functionality:

Unreal abstracts source control operations in general, but the broad adoption of Perforce has resulted in Unreal’s operations not mapping to Git operations 1:1. For example, “Check In” in Unreal terms maps well to Perforce’s p4 submit, but for Git this is a combination of git add, git commit, git pull, and git push.Syncing a large number of commits often led to Unreal Editor crashing mid-pull, resulting in the repository being stuck in a rebase. This wasn’t often enough for us to abandon the plugin, but it was enough to prioritize fixing in the near-term.The plugin will try to detect conflicts to avoid git pulling itself into an unmerged state, but its feedback instructs the user to use the command line to avoid the scenario, and that wasn’t satisfying UX to us.The plugin relies on staging and unstaging files in ways that are completely hidden to the user, complicating a few operations and making troubleshooting difficult at times.

Unreal abstracts source control operations in general, but the broad adoption of Perforce has resulted in Unreal’s operations not mapping to Git operations 1:1. For example, “Check In” in Unreal terms maps well to Perforce’s p4 submit, but for Git this is a combination of git add, git commit, git pull, and git push.
Syncing a large number of commits often led to Unreal Editor crashing mid-pull, resulting in the repository being stuck in a rebase. This wasn’t often enough for us to abandon the plugin, but it was enough to prioritize fixing in the near-term.
The plugin will try to detect conflicts to avoid git pulling itself into an unmerged state, but its feedback instructs the user to use the command line to avoid the scenario, and that wasn’t satisfying UX to us.
The plugin relies on staging and unstaging files in ways that are completely hidden to the user, complicating a few operations and making troubleshooting difficult at times.

Despite these issues, the plugin got our team up and running, syncing and pushing updates from Unreal Editor with relative ease.

Setting Everything Up

In order to use Git LFS, we needed to first deploy Rudolfs and stand up networking infrastructure in front of it that could be reached by our team. Our strategy for infrastructure management is a story for another blog post, but in short, we deployed a Rudolfs Docker container to AWS ECS and slapped an AWS Application Load Balancer in front of it.

Then, we needed to add two files to the root of our Git repository: .gitattributes for defining which files should use LFS, and .lfsconfig for defining the location of our LFS server. This configuration is basically taken directly from Project Borealis.

[attr]lfs filter=lfs diff=lfs merge=binary -text
[attr]lock filter=lfs diff=lfs merge=binary -text lockable
[attr]lockonly lockable
[attr]lfstext filter=lfs diff=lfstext merge=lfstext -text
# Unreal Engine file types.
*.uasset lfs
*.umap lfs
[lfs]
url = "<https://example.com/api/gitorg/reponame>"

Then, we had everybody on the team make GitHub accounts and download Fork. As part of this, we published an internal onboarding guide that walked folks through making the account, downloading Fork, and configuring Fork to use our repo.

Additionally, we installed UEGitPlugin and made sure folks could successfully sync and submit from within Unreal Editor. In this case, installing the plugin involved simply copying its code into our repository. Our repo architecture has changed a bit since then as we’ve moved to building our own version of the engine, but again, a story for another time.

Phase 1: Enhancing the LFS Server

At this point, the whole team was using Git to work on our game’s prototype. In general, it was working, but there was one major problem causing a lot of pain: the lack of file locking capabilities. As we mentioned earlier, Rudolfs didn’t implement the Git LFS file locking API, so even though both Fork and UEGitPlugin support file locking, our LFS backend did not. The team was still quite small at this time, so it was possible for folks to effectively communicate which files they were working on, but there were a few serious incidents involving merge conflicts. Filling this gap became our number one priority.

Implementing Auth

Before expanding the functionality of the LFS server, we needed to implement authentication and authorization. Because we were using GitHub, it seemed sensible to use a user’s GitHub identity as a source for this. We decided to make Rudolfs require all requests to pass a valid GitHub Personal Access Token. Luckily, we were able to find inspiration through a closed pull request in the Rudolfs upstream repo. Rudolfs checks this token against GitHub’s API to see if the user has proper permissions for the requested repository.

Implementing Locks

With auth implemented, it was time to start working on file locking. Our first challenge was choosing where to persist lock data, but we quickly determined that DynamoDB would make a great storage backend:

It’s highly performant for our use case and would scale well as our studio doesIt would be very affordable (as of writing our utilization still falls within the AWS Free Tier)Rudolfs was already configured for AWS access and had libraries in use for interfacing with AWS APIs

It’s highly performant for our use case and would scale well as our studio does
It would be very affordable (as of writing our utilization still falls within the AWS Free Tier)
Rudolfs was already configured for AWS access and had libraries in use for interfacing with AWS APIs

For our implementation, we intended to follow the LFS v2.0 File Locking spec except for one aspect. In the API docs, locks are branch-specific, and we specifically did not want that. Our use case calls for locks being repo-global because there is no reasonable way to merge changes between .uasset files, so we wanted to ensure a file could only be locked once repo-wide. In the future, we’ll make this a configuration option on Rudolfs

So… we did it! We implemented locks in Rudolfs and deployed a new container for the team to use. The introduction of locks improved the team’s workflow substantially, as folks could see which files were being worked on at any given time.

You can find (and use) our LFS server on GitHub here!

A Preview for Part three

While getting Rudolfs stood up and implementing locks was a huge step for us, it was only the first step of many. We wanted to create a user experience for Git and Unreal that made the team excited to work on our project. For us, we felt that meant developing some tools of our own that were tailor-made for merging Unreal and Git together, and so that’s what we started on next. We believe that’s worthy of its own chapter in the story, and so we’ll be talking about it in detail in the next part of this series.

Next time:

How we built CI/CD pipelines that make client and server builds available to the entire company
How we enable anyone at the company to spin up their own game server
How we deploy and run infrastructure for our internal playtests

Thanks so much for reading! If you’d like to continue the conversation, join us on Discord!

Using Git with Unreal Engine, Part 2: A First Pass at a Friendly Workflow