How to Add Cryptographic Integrity to an Object Store
A store that opens and deserializes correctly may still differ from the copy you approved, because deserialization says nothing about whether a value was changed. Detecting that needs a digest of the contents that can be compared with a trusted value.
An ObjectSpace in Sphere10 Framework can maintain that digest itself. This article shows how Merkleized() adds a Merkle tree to every dimension and a root over the whole store, how to read and compare the root, and what a matching root does and does not prove.
1. Enable Merkleization
Merkleization is requested on the builder, after the dimensions have been added.
var builder = new ObjectSpaceBuilder()
.UseMemoryStream(stream)
.UsingSerializerFactory(serializers)
.UsingHashFunction(CHF.SHA2_256)
.WithIndexBTreeOrder(64)
.WithEndianness(Endianness.LittleEndian);
SampleSchema.Configure(builder);
builder.Merkleized();
Three details matter here.
-
Order of calls.
Merkleized()attaches a tree only to the dimensions that already exist, so it comes afterConfigurehas added them. -
Hash function.
UsingHashFunctionselects the digest from the Framework'sCHFenumeration. The hash function and the Merkle setting are part of the stored format, and a file must be reopened with the same values. -
Serializers. A root is a hash of serialized bytes, so the serializers must be the same on both sides of a comparison. The sample passes its own factory through
UsingSerializerFactory, which must be called before any dimension is added.
2. How the Merkle structure is built
Merkleized() works at two levels. Each dimension receives a Merkle tree whose leaves are the hashes of its stored objects. The ObjectSpace then keeps a second tree whose leaves are the roots of the dimension trees, one leaf for each dimension in declaration order.
Saving or deleting an object changes one leaf, which changes that dimension's root, which updates one leaf of the store's tree. The trees are stored in the same file as the data and are brought up to date by Flush(), so read a root only after a flush or a commit.
Diagram source
%%{init: {"flowchart": {"nodeSpacing": 20, "rankSpacing": 28}, "themeVariables": {"fontSize": "16px"}}}%%
flowchart TB
R["ObjectSpace root<br/>kept in the header"] --- C["Leaf 0<br/>Customer root"]
R --- P["Leaf 1<br/>Purchase root"]
C --- C0["H(slot 0)"]
P --- P0["H(slot 0)"]
P --- P1["H(slot 1)"]
3. Read the root through a version-bound adapter
The store's tree keeps the current root in the container header. Version 3.1.3 does not expose it as a property, so the sample derives from MemoryObjectSpace and reads the value from the header.
public byte[] ReadRoot() {
using (EnterAccessScope()) {
var length = Hashers.GetDigestSizeBytes(Definition.HashFunction);
// This application-owned extension targets the pinned ObjectSpace header layout.
return (byte[])_streams.Header.MapExtensionProperty(0, sizeof(bool) + length, new ConstantSizeNullableByteArraySerializer(length)).Value.Clone();
}
}
public void VerifyStorageConsistency() {
using (EnterAccessScope())
_streams.Attachments[Sphere10FrameworkDefaults.DefaultSpatialMerkleTreeIndexName].VerifyIntegrity();
}
ReadRoot returns a copy of the root bytes. VerifyStorageConsistency calls the store tree's VerifyIntegrity(), which checks that each leaf equals the root of its dimension tree and that the stored root equals the root recomputed from those leaves.
ReadRoot depends on the header layout of the Framework version the sample pins. Check it again before moving to another version.
4. Capture a root and compare it later
A root is captured once the data has settled.
space.Flush();
capturedRoot = space.ReadRoot();
space.VerifyStorageConsistency();
The store is then written out, read back into a new stream and opened by a new space with the same definition. Its root is compared with the captured value.
using var reopened = VerifiableObjectSpace.Open(received);
SampleChecks.Require(ByteArrayEqualityComparer.Instance.Equals(capturedRoot, reopened.ReadRoot()), "The reopened store does not match the captured root.");
ByteArrayEqualityComparer is the Framework's comparer for byte arrays. Equal roots mean that the stored objects of every dimension hash to the same values as before.
5. See one edit change the root
Changing a single member of a single object produces a different root.
var purchase = reopened.Get((Purchase item) => item.Id, 101L);
purchase.Total = 45m;
reopened.Save(purchase);
reopened.Flush();
SampleChecks.Require(!ByteArrayEqualityComparer.Instance.Equals(capturedRoot, reopened.ReadRoot()), "The edit did not change the root.");
reopened.VerifyStorageConsistency();
The consistency check still passes after the edit. The trees correctly describe the new contents, and they simply no longer match the earlier root. Consistency and equality with a known root are separate questions.
6. What a matching root proves
| Check | Establishes | Does not establish |
| Root equals a trusted root | The contents are identical to the contents that produced that root | Who produced them |
VerifyIntegrity() passes |
The stored trees agree with each other | That the contents are the ones you expected |
Neither check knows anything about your application's own rules, so data can match a root and still need validating. The comparison is only as good as the root you compare against. Keep the trusted root somewhere the data's editor cannot reach, or sign it. A root stored beside the data it describes can be replaced along with the data.
A Merkle root also allows one record to be proven a member of the set without sending the rest, which record inclusion proofs covers. Hashes, signatures and proofs compares what each of those tools establishes.
7. Get the package and run the sample
Use .NET 10 and the Sphere10.Framework NuGet package.
dotnet new console --framework net10.0 --name ArticleSample
cd ArticleSample
dotnet add package Sphere10.Framework --version 3.1.3
The complete MIT-licensed sample contains the project and every file shown here. Extract the download over the generated project and run it against a new directory.
dotnet run -c Release -- ./fresh-data
Reopened store matches the captured root: passed
One edit changes the root: passed
Edited store is still internally consistent: passed
The ObjectSpace integrity guide documents the Merkle and hash settings alongside the other builder options.
The Sphere10 Framework overview describes the rest of the toolkit.
Author: Herman Schoenfeld