A file format often needs several byte streams that grow and shrink independently, such as records, indexes and attachments. Laying them out by hand means reserving fixed regions or moving data whenever one stream outgrows its space.

ClusteredStreams in Sphere10 Framework solves this once. It multiplexes any number of resizable logical streams onto one root Stream, in the way a file system places many files on one disk. This article explains how the container is laid out and how to create, edit and reopen it.

1. Create a container and add streams

Construct the container over a root stream, such as a FileStream or a MemoryStream. An empty root stream is initialized with the given cluster size.

using (var file = File.Open(containerPath, FileMode.CreateNew, FileAccess.ReadWrite)) {
	using var streams = new ClusteredStreams(file, clusterSize: 4096, autoLoad: true);
	using (var first = streams.Add())
		first.Write(Encoding.UTF8.GetBytes("first stream"));
	using (var second = streams.Add())
		second.Write(Encoding.UTF8.GetBytes("second stream"));
	using (var third = streams.Add())
		third.Write(large);
	using (var fourth = streams.Add())
		fourth.Write(Encoding.UTF8.GetBytes("temporary stream"));

	// Replace the content of stream 1 with content of a different length.
	using (var edit = streams.OpenWrite(1)) {
		edit.SetLength(0);
		edit.Write(Encoding.UTF8.GetBytes(replacement));
	}

	// Remove the last stream, then exchange the indexes of the first two.
	streams.Remove(3);
	streams.Swap(0, 1);
}

autoLoad: true loads the container in the constructor. Without it, call Load() before the first operation.

Add() appends a new logical stream and returns it open for writing. The four calls above create streams 0 to 3. The returned ClusteredStream derives from Stream, so it works with readers, writers and anything else that accepts a stream.

OpenWrite(1) reopens the second stream. SetLength(0) truncates it and the following Write stores new content of a different length. The 1 MB stream after it is untouched, and no offsets were calculated by the caller.

Remove(3) deletes the last stream and releases its clusters. Swap(0, 1) exchanges the first two indexes, so the rewritten stream is now stream 0.

2. How the container is laid out

---
config:
  packet:
    bitsPerRow: 16
    bitWidth: 22
    showBits: false
---
packet-beta
0-3: "Version"
4-9: "Policy"
10-15: "StreamCount"
16-31: "StreamDescriptorsEndCluster"
32-39: "ReservedStreams"
40-47: "ClusterSize"
48-55: "TotalClusters"
56-63: "Extension bytes"
64-79: "Cluster 0, at byte 256"
80-95: "Cluster 1"

erDiagram
    DESCRIPTOR ||--|{ CLUSTER : "StartCluster to EndCluster"
    DESCRIPTOR {
      byte Traits
      long StartCluster
      long EndCluster
      long Size
    }
    CLUSTER {
      byte Traits
      long Prev
      long Next
      bytes Data
    }

3. Stream operations

Streams are identified only by index. The container stores no names, so an application that wants named entries keeps its own directory, commonly in stream 0. Because Insert and Remove renumber later streams, such a directory has to be updated with them.

An open ClusteredStream holds the container's access scope until it is disposed. Only one handle per index can be open at a time, and Insert and Remove require that no stream is open. Dispose each handle as soon as you finish with it.

4. Reopen an existing container

A non-empty root stream is validated when the container loads. ClusteredStreams.FromStream reads the cluster size, policy and reserved-stream count from the stored header, so the caller does not have to repeat them.

using var reopenedFile = File.Open(containerPath, FileMode.Open, FileAccess.ReadWrite);
using var reopened = ClusteredStreams.FromStream(reopenedFile, autoLoad: true);

using var stream = reopened.OpenRead(index);
using var reader = new StreamReader(stream);
texts[index] = reader.ReadToEnd();

The constructor used for creation also opens an existing container. In that case the clusterSize argument must equal the value stored in the header.

5. Choose a cluster size

The cluster size is fixed when the container is created. Every cluster carries its links as well as its data, and a stream always occupies whole clusters.

• Small clusters waste little space on short streams but give long streams long chains.
• Large clusters suit long streams but leave more unused space at the end of each short one.

The default is 4,096 bytes (Sphere10FrameworkDefaults.ClusterSize), and the Framework's recommended range is 256 to 8,192 bytes. Choose from the typical size of the streams you expect to store.

6. Ownership and limits

ClusteredStreams is also the storage layer beneath the Framework's stream-mapped collections, ObjectStream and ObjectSpaces. Those types register attachments, which are indexes and other structures held in reserved streams at the start of the container.

7. Get the package and run the sample

Add the package to a .NET 10 project.

dotnet add package Sphere10.Framework --version 3.1.3

The sample uses the .NET 10 SDK and the Sphere10.Framework 3.1.3 package. Download the sample project and run it.

dotnet run -c Release -- ./fresh-example-data

Choose a new output directory because the sample creates its file with CreateNew. It writes four streams, rewrites one, removes one and swaps two, then reopens the container and checks every stream.

Streams after reopen: 3
Stream 0: second stream, rewritten with a different length
Stream 1: first stream
Stream 2 unchanged bytes: 1048576

• Clustered stream container API
• Clustered stream persistence tests

The Sphere10 Framework overview describes the rest of the toolkit.

Author: Herman Schoenfeld