Data Model & Version Control

Shared data model and save file versioning strategies for the game engine

Shared data model

The most crucial property of any save/load system is data integrity, meaning that all important data gets saved, and no data is mutated or lost when saving or loading.

My game engine preserves data integrity using a tool called Serde, which promises data integrity by making save files share an identical data model with the running game.

flowchart LR
	save <==> engine

Sharing a data model means that whatever data lives inside the running game engine will be the exact same data that gets saved to a file, satisfying the requirement that all important data gets saved.

But what about ensuring that no data is mutated or lost? Serde currently has 732,416,894 downloads according to the crates.io index - a battle tested tool that can be reasonably trusted to behave correctly.

Coding the engine’s data model to leverage Serde is easy:

use engine::{Model, Script};
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)] // <-- Serde magic!
pub struct GameData {
	pub models:  Vec<Model>,
	pub scripts: Vec<Script>,
}

This #[derive] annotation augments our data model with Serde’s battle-tested saving and loading machinery. Data integrity is solved, my shortest blog post ever!

Multiple Versions

Things fall apart when updating the engine, though. Serde promises that new save files match the engine’s data model, but what about old save files? As the engine accumulates new features and optimizations, its data model may diverge from the data model used by older save files. In that case, Serde cannot promise data integrity between these versions.

Serde may produce an error like this one when the data models are not identical:

[ERROR] Failed to load save_v1: missing field `added_in_v2` at line 6 column 7

Picture something like this:

flowchart TB
	engine_v1 <==> save_v1
	engine_v2 <==> save_v2
	engine_v3 <==> save_v3

Serde preserves data integrity for save files using the same version as the engine, but we lose information between versions because Serde doesn’t understand the old data model!

Imagine losing a project you’ve spent months building because of an engine update. Clearly, we need to support older save files in newer engine versions somehow.

Let’s consider a couple of possible solutions.

Upgrading the Files

The idea here is to upgrade each save from one version to the next, until its data model matches the engine’s current data model so Serde can understand it.

Picture something like this:

flowchart LR
	save_v1 --> save_v2
	save_v2 --> save_v3
	save_v3 <==> engine_v3

The bold line between save_v3 and engine_v3 indicates they share an identical data model and Serde promises to preserve data integrity when saving and loading.

Migrating between data models can’t use Serde and must instead use hand-written code - code that doesn’t have 708 million users catching and reporting its bugs.

Let’s imagine a bug in the code that migrates save files from v2 to v3:

flowchart LR
	save_v1["**save_v1**"] --> save_v2
	save_v2["**save_v2**"] -.**X**.-> save_v3
	save_v3 <==> engine_v3

	style save_v1 fill:#ff0000
	style save_v2 fill:#ff0000

Because the upgrade is sequential from version to version, a bug anywhere in the chain affects all previous versions as well!

Upgrading the Engine?

Let’s take our diagram, and move the arrows around a bit. Rather than upgrading old save files sequentially, we can teach the engine to consume the old files directly:

flowchart TB
	disk_v1 --> engine_v3
	disk_v2 --> engine_v3
	disk_v3 <==> engine_v3

Remember, v3 save files are uncorruptable because Serde guarantees data integrity wherever the data model is exactly the same. Only the code that loads v1 and v2 save files may contain nasty serialization bugs.

Since we broke the serial procedure into a parallel one, have we fixed the backpropagating data corruption issue from before? Not necessarily! If two older save files both need the same upgrade logic applied to them, a cunning developer may re-use the code for each of their loading procedures - meaning that bugs may still propagate between versions!

How can we protect against this? Is hope lost?

Automated Testing > Serde

Let’s briefly revisit the value that Serde provides to my game engine.

When the engine is first released (think of a 1.0 build, or something) most save files are going to match the engine’s data model and integrity is preserved. But as time marches on into the future, the number of older save versions that I must support is going to grow, and the amount of code which benefits from Serde is going to asymptotically approach zero.

Look, only five versions, and the majority of save file versions aren’t guaranteed to be safe:

flowchart TB
	save_v1 --> engine_v5
	save_v2 --> engine_v5
	save_v3 --> engine_v5
	save_v4 --> engine_v5
	save_v5 <==> engine_v5

How do we provide some guarantees for the data integrity of these older save file formats? The professional strategy I’d use here is automated testing.

The idea is to cultivate a growing record of authentic and artificial save files for each prior version of the engine. A dedicated software environment could push all of these older save files into a running engine, and validate the engine loads them “correctly.” By passing all new versions of the engine through this testing environment, we may catch bugs that may have otherwise reached users.

As the amount of supported save file formats grow, we’ll increasingly rely on this automated testing environment to ensure data integrity, and Serde becomes less and less important.

Upgrade … no thing?

I was riding the blue line with my friend and mentor Mukesh when he suggested this approach: what if we completely break the promise that same-versioned files and engines share an identical data model? So even as the engine’s data model changes over time, the save file format remains constant!

That means there’s no 1:1 mapping from disk_v3 to engine_v3 but, since older files must be supported, we always had custom deserialization logic for those loading pathways anyways.

Picture something like this:

flowchart LR
	subgraph v3
		subgraph v2
			subgraph v1
			end
		end
	end

	v3 <--> engine

Basically, as long as we only add features to the save files then we guarantee each disk model is a superset of all previous disk models. If the engine tries loading an old save file, it can instantiate reasonable defaults for each missing field. That locks us in to whatever data model came out “first,” which may degrade load-time performance if the engine model evolves to deviate too far from the disk model, but it solves the problem of having multiple scripts to load each prior data model.