LDES server implementation guide for developers

Living Document,

Previous Versions:
Editor:
Gerald Haesendonck (Ghent University - imec)

CC0 To the extent possible under law, the editors have waived all copyright and related or neighboring rights to this work. In addition, as of 21 January 2025, the editors have made this specification available under the Open Web Foundation Agreement Version 1.0, which is available at https://www.openwebfoundation.org/the-agreements/the-owf-1-0-agreements-granted-claims/owfa-1-0. Parts of this work may be from another specification document. If so, those parts are instead covered by the license of that specification document.

Abstract

This specification aims to be a practical guide for developers to create a Linked Data Event Stream (LDES) server.

1. Introduction

Creating a Linked Data Event Stream (LDES) server involves understanding two specifications, the LDES specification ([ldes-spec]) and the TREE hypermedia specification ([tree-spec]), upon which the LDES specification is built. The goal of this specification is to describe the requirements for a server that hosts a time-based fragmented LDES and provides a mechanism for adding members to an LDES, based on Linked Data Platform concepts.

Note: At this moment this document addresses the requirements of a specific use case. Existing implementations of generic LDES servers are LDES Solid Server and VSDS LDESServer4J.

2. LDES

2.1. Example and definition

This guide uses a running example to explain a Linked Data Event Stream and, in the next chapter, the operations the server supports.

The scenario is a scale that measures weights of a person. The readings are sent to an LDES server which keeps track of every reading.

A reading might look like this:

{
  "sensor": "http://example.com/sensors/weightSensor",
  "value": 78.4,
  "timestamp": "2025-01-01T08:00:00Z",
  "unit": "weightInKg"
}

Definition: A LDES is a collection representing a stream of unmodifiable objects called members.

Here is an example of a minimal LDES with three weight readings as members:

@prefix sosa: <http://www.w3.org/ns/sosa/> .
@prefix ldes: <https://w3id.org/ldes#> .
@prefix tree: <https://w3id.org/tree#> .
@prefix xsd:  <http://www.w3.org/2001/XMLSchema#> .
@prefix sensors: <http://example.com/sensors/> .
@prefix shapes: <http://example.com/shapes/> .

@base <http://example.com/weights/> .

<#EventStream> a ldes:EventStream ;
  ldes:timestampPath sosa:resultTime ;
  tree:shape shapes:memberShape ;
  tree:member <2025/01/weight1>, <2025/01/weight2>, <2025/02/weight3> .

<2025/01/weight1>
  a sosa:Observation ;
  sosa:madeBySensor sensors:weightSensor ;
  sosa:hasSimpleResult "78.4"^^xsd:decimal ;
  sosa:observedProperty: "weightInKg" ;
  sosa:resultTime "2025-01-01T08:00:00Z"^^xsd:dateTime .

<2025/01/weight2>
  a sosa:Observation ;
  sosa:madeBySensor sensors:weightSensor ;
  sosa:hasSimpleResult "78.6"^^xsd:decimal ;
  sosa:observedProperty: "weightInKg" ;
  sosa:resultTime "2025-01-15T08:00:00Z"^^xsd:dateTime .

<2025/02/weight3>
  a sosa:Observation ;
  sosa:madeBySensor sensors:weightSensor ;
  sosa:hasSimpleResult "78.1"^^xsd:decimal ;
  sosa:observedProperty: "weightInKg" ;
  sosa:resultTime "2025-02-01T07:15:00Z"^^xsd:dateTime .

Example 1 is formatted in Turtle [TURTLE], a compact human-readable serialisation of RDF [RDF-CONCEPTS], a graph-based data model for linked data. The same information could also be formatted in JSON for linking data [JSON-LD]:

{
   "@context" : {
      "@base": "http://example.com/weights/"
      "ldes" : "https://w3id.org/ldes#",
      "sensor" : {
         "@id" : "http://www.w3.org/ns/sosa/madeBySensor",
         "@type" : "@id"
      },
      "sosa" : "http://www.w3.org/ns/sosa/",
      "timestamp" : {
         "@id" : "http://www.w3.org/ns/sosa/resultTime",
         "@type" : "http://www.w3.org/2001/XMLSchema#dateTime"
      },
      "tree" : "https://w3id.org/tree#",
      "unit" : "http://www.w3.org/ns/sosa/observedProperty",
      "value" : "http://www.w3.org/ns/sosa/hasSimpleResult",
      "xsd" : "http://www.w3.org/2001/XMLSchema#",
      "sensors": "http://example.com/sensors/",
      "shapes": "http://example.com/shapes/"
   },
   "@graph" : [
      {
         "@id" : "#EventStream",
         "@type" : "ldes:EventStream",
         "ldes:timestampPath" : {"@id" : "sosa:resultTime"},
         "tree:shape": {"@id": "shapes:memberShape"},
         "tree:member" : [
           {"@id" : "2025/01/weight1"},
           {"@id" : "2025/01/weight2"},
           {"@id" : "2025/02/weight3"}
         ]
      },
      {
         "@id" : "2025/01/weight1",
         "@type" : "sosa:Observation",
         "sensor" : "sensors:weightSensor",
         "value" : 78.4,
         "timestamp" : "2025-01-01T08:00:00Z",
         "unit" : "weightInKg"
      },
      {
         "@id" : "2025/01/weight2",
         "@type" : "sosa:Observation",
         "value" : 78.6,
         "sensor" : "sensors:weightSensor",
         "timestamp" : "2025-01-15T08:00:00Z",
         "unit" : "weightInKg"
      },
      {
         "@id" : "2025/02/weight3",
         "@type" : "sosa:Observation",
         "value" : 78.1,
         "sensor" : "sensors:weightSensor",
         "timestamp" : "2025-02-01T07:15:00Z",
         "unit" : "weightInKg"
      }

   ]
}

The context (@context) maps terms to IRIs [IRI] (necessary for Linked data). The @graph contains the data itself. Thanks to the term mapping in the context, the members of the LDES almost look identical to the original readings in JSON.

Throughout this document Turtle and JSON-LD serialisations will be used.

2.2. Properties

The ldes:EventStream instance SHOULD have the following properties:

2.2.1. tree:member

This property indicates the members of the collection. Members are immutable objects in the collection forming the LDES.

2.2.2. tree:shape

This property defines the shape of the members. A shape can be seen as a schema for RDF graphs. All members of the stream have been validated by the shape. The shape MAY evolve over time, but because the members are immutable, it MUST be backwards compatible to earlier versions.

Here is a shape for the example, expressed in SHACL [SHACL]:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix schema: <http://schema.org/> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix sosa: <http://www.w3.org/ns/sosa/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix shapes: <http://example.com/shapes/> .

shapes:memberShape
  a sh:NodeShape ;
  sh:targetClass sosa:Observation;
  sh:property [
    sh:path sosa:madeBySensor ;
    sh:dataType xsd:string ;
    sh:minCount 1 ;
    sh:maxCount 1
  ],[
    sh:path sosa:hasSimpleResult ;
    sh:dataType xsd:decimal ;
    sh:minCount 1 ;
    sh:maxCount 1
  ],[
    sh:path sosa:observedProperty ;
    sh:dataType xsd:string ;
    sh:minCount 1 ;
    sh:maxCount 1
  ],[
    sh:path sosa:resultTime ;
    sh:dataType xsd:dateTime ;
    sh:minCount 1 ;
    sh:maxCount 1
  ] .

ldes:EventStream instance MAY have the following properties:

2.2.3. ldes:timestampPath

An important property is the timestamp field of the readings. Because the tree:member property simply lists the members, the timestampPath can be used to determine a time-based order. It can also be used as basis for fragmentation (split up a large LDES into smaller fragments, see later). In the weights example the JSON key containing timestamps is timestamp. Using the context this field maps to the timestampPath http://www.w3.org/ns/sosa/resultTime.

2.2.4. ldes:versionOfPath

If members represent a version of an object, this property indicates the non-version object. The followin example shows an LDES containing address changes:

ex:C2 a ldes:EventStream ;
  ldes:timestampPath dcterms:created ;
  ldes:versionOfPath dcterms:isVersionOf ;
  tree:shape ex:shape2.shacl ;
  tree:member ex:AddressRecord1-version1 .

ex:AddressRecord1-version1 dcterms:created 
  "2021-01-01T00:00:00Z"^^xsd:dateTime ;
  adms:versionNotes "First version of this address" ;
  dcterms:isVersionOf ex:AddressRecord1 ;
  dcterms:title "Streetname X, ZIP Municipality, Country" .

2.3. Fragmentation

A LDES can be split up into smaller parts or fragments, so clients don’t need to process potentially a lot of data, and in the context of time series older fragments are unlikely to change and can easily be cached when serving over HTTP. LDES relies on the TREE specification ([tree-spec]) for fragmentation supporting many options. In this document we pick time-based fragmentation because that’s the most interesting for time series data. The ldes:timestampPath property is used to determine to which fragment a reading gets added.

In our weights example, we will make a fragment per month of data. This results in a tree structure which might look like this:

Tree representtion of a possible fragmentation per year with a sub-fragmentation per month. 
weights
└── 2025
    ├── 01
    │   ├── weight1
    │   └── weight2
    └── 02
        └── weight3

The directory structure is used here because it maps to the paths in the identifiers. For example, the first reading has IRI 2025/01/weight1 (relative to the base IRI).

To get fragmentation, we replace the member listing (tree:member) in the ldes:eventStream instance with a tree:view. This view points - in this case - to the root node in the tree structure which has relations to other nodes.

Applying the fragmentation of Example 5 looks like this:

Top level fragmentation per year: 
@prefix ldes:   <https://w3id.org/ldes#> .
@prefix tree:   <https://w3id.org/tree#> .
@prefix xsd:    <http://www.w3.org/2001/XMLSchema#> .
@prefix sosa:   <http://www.w3.org/ns/sosa/> .
@prefix shapes: <http://example.com/shapes/> .

@base <http://example.com/weights/> .

<#EventStream> a ldes:EventStream ;
  ldes:timestampPath sosa:resultTime ;
  # Rather than listing the members directly, we point to a view:
  tree:view <> ;
  tree:shape shapes:memberShape .

<>
  a tree:Node ;

  tree:relation [
    # Readings for 2025 can be found in fragment /2025
    a tree:GreaterThanOrEqualToRelation ;
    tree:node <2025/> ;
    tree:path sosa:resultTime ;
    tree:value "2025-01-01T00:00:00Z"^^xsd:dateTime
  ] .

The relation points to another node (fragment of all weights of the year 2025), where we introduce a fragmentation per month:

@base <http://example.com/weights/2025/> .

<#EventStream> a ldes:EventStream ;
  ldes:timestampPath sosa:resultTime ;
  # Rather than listing the members directly, we point to a view:
  tree:view <> ;
  tree:shape shapes:memberShape .

<>
  a tree:Node ;

  tree:relation [
    # Readings for January can be found in fragment 01
    a tree:GreaterThanOrEqualToRelation ;
    tree:node <01/> ;
    tree:path sosa:resultTime ;
    tree:value "2025-01-01T00:00:00Z"^^xsd:dateTime
  ] ;
  tree:relation [
    # Readings for February can be found in fragment 02
    a tree:GreaterThanOrEqualToRelation ;
    tree:node <02/> ;
    tree:path sosa:resultTime ;
    tree:value "2025-02-01T00:00:00Z"^^xsd:dateTime
  ] .

The fragment http://example.com/weights/2025/01/ contains an event stream for January, with links to the actual weight readings:

@base <http://example.com/weights/2025/01/> .

<> a tree:Node .

<#EventStream> a <https://w3id.org/ldes#EventStream> ;
  tree:view <> ;
  ldes:timestampPath sosa:resultTime ;
  tree:shape shapes:memberShape ;
  tree:member <weight1>, <weight2> .

The same goes for February:

@base <http://example.com/weights/2025/01/> .

<> a tree:Node .

<#EventStream> a <https://w3id.org/ldes#EventStream> ;
  tree:view <> ;
  ldes:timestampPath sosa:resultTime ;
  tree:shape shapes:memberShape ;
  tree:member <weight3> .

3. Implementing a server

This section describes how a timeseries-based LDES server might be implemented.

3.1. Conceptual layout

The idea is to use some concepts from the Linked Data Notifications [LDN]: a container to store a LDES and an inbox to write data to.

Applied to the weights example, suppose the server hosts the domain example.com. Then a container can be created to host weights, called weights. It will be reachable at https://example.com/weights.

Here is a conceptual server layout:

A possible conceptual server layout for the weights example: 
weights
├── 2025
│   ├── 01
│   │   ├── root     # Event stream metadata for January fragment 
│   │   ├── weight1
│   │   └── weight2
│   ├── 02
│   │   ├── root
│   │   └── weight3
│   └── root         # Event stream metadata for the 2025 fragment
├── inbox            # Endpoint to post new readings.
├── shape            # Endpoint to post new readings.
└── root             # Contains info about the container and the
                     # fragmented event stream.

In Example 7 weight readings can be written to https://example.com/weights/inbox..

LDES metadata is available at https://example.com/weights/#EventStream.

Individual weights can be fetched at https://example.com/weights/2025/01/weight1, https://example.com/weights/2025/01/weight2, and https://example.com/weights/2025/01/weight3.

3.2. Configuring a container

The server must know some things before it can initialize a container hosting a LDES. This is called the container configuration. It contains following parameters:

How this container configuration gets to the server is up to the implementation. A server may read the configuration at start-up time and initialize the containers it finds in the configuration. Another option is to provide an endpoint on the server to post a container configuration to, in which case a container can be created on-the-fly.

The member context and fragmentation strategy parameters are explained more in detail in the next sections.

3.2.1. Member context

Readings are expected to be represented as RDF. So the server MAY accept formats like Turtle for readings. However, the JSON format is more likely to be used for readings, and MAY also be accepted by the server. Adding a context turns these messages to RDF.

Remember an example plain JSON weight reading:

Example weight reading in JSON. 
{
  "sensor": "http://example.com/sensors/weightSensor",
  "value": 78.4,
  "timestamp": "2025-01-01T06:00:00Z",
  "unit": "weightInKg"
}

The corresponding version in linked data (RDF) is:

Example weight reading in RDF (Turtle): 
<2025/01/weight1>
  a sosa:Observation ;
  sosa:madeBySensor sensors:weightSensor ;
  sosa:hasSimpleResult "78.4"^^xsd:decimal ;
  sosa:observedProperty: "weightInKg" ;
  sosa:resultTime "2025-01-01T08:00:00Z"^^xsd:dateTime .

Which can also be represented in JSON-LD as:

Example weight reading in RDF (JSON-LD): 
{
  "@context": "https://example.com/weights/context",
  "@id" : "2025/01/weight1",
  "@type" : "sosa:Observation",
  "sensor": "https://example.com/sensors/weightSensor",
  "value": 78.4,
  "timestamp": "2024-01-01T06:00:00Z",
  "unit": "weightInKg"
}

Example 10 looks just like the original reading (Example 8), except that a few things were added by the server: an @id field adding a unique IRI for this reading, a @type field telling that this is an "observation" and a @context, which maps the fields to their RDF counterparts. In this case the context is served at https://example.com/weights/context, but it can also be inlined. In this example the context looks like:

Example JSON-LD context for weight reading. 
{
  "sensor": {
    "@id": "http://www.w3.org/ns/sosa/madeBySensor",
    "@type": "@id" },
  "value": "http://www.w3.org/ns/sosa/hasSimpleResult",
  "timestamp": {
    "@id": "http://www.w3.org/ns/sosa/resultTime",
    "@type": "http://www.w3.org/2001/XMLSchema#dateTime" },
  "unit": "http://www.w3.org/ns/sosa/observedProperty"
}

3.2.2. Fragmentation strategy

TREE, and thus LDES, support many types of fragmentation. Here we focus on time-based fragmentation, where the time granularity is important.

The server MAY support following time granularities:

In our weight example, the granularity is month.

3.2.3. Example container configuration

Conformance

Conformance requirements are expressed with a combination of descriptive assertions and RFC 2119 terminology. The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in the normative parts of this document are to be interpreted as described in RFC 2119. However, for readability, these words do not appear in all uppercase letters in this specification.

All of the text of this specification is normative except sections explicitly marked as non-normative, examples, and notes. [RFC2119]

Examples in this specification are introduced with the words “for example” or are set apart from the normative text with class="example", like this:

This is an example of an informative example.

Informative notes begin with the word “Note” and are set apart from the normative text with class="note", like this:

Note, this is an informative note.

Index

Terms defined by this specification

References

Normative References

[IRI]
M. Duerst; M. Suignard. Internationalized Resource Identifiers (IRIs). January 2005. Proposed Standard. URL: https://www.rfc-editor.org/rfc/rfc3987
[JSON-LD]
Manu Sporny; Gregg Kellogg; Markus Lanthaler. JSON-LD 1.0. 3 November 2020. REC. URL: https://www.w3.org/TR/json-ld/
[LDES-SPEC]
Pieter Colpaert. Linked Data Event Streams. URL: https://w3id.org/ldes/specification
[LDN]
Sarven Capadisli; Amy Guy. Linked Data Notifications. URL: https://linkedresearch.org/ldn/
[RDF-CONCEPTS]
Graham Klyne; Jeremy Carroll. Resource Description Framework (RDF): Concepts and Abstract Syntax. URL: https://w3c.github.io/rdf-concepts/spec/
[RFC2119]
S. Bradner. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL: https://datatracker.ietf.org/doc/html/rfc2119
[SHACL]
Holger Knublauch; Dimitris Kontokostas. Shapes Constraint Language (SHACL). URL: https://w3c.github.io/data-shapes/shacl/
[TREE-SPEC]
Pieter Colpaert. The TREE hypermedia specification. URL: https://w3id.org/tree/specification
[TURTLE]
Eric Prud'hommeaux; Gavin Carothers. RDF 1.1 Turtle. URL: https://w3c.github.io/rdf-turtle/spec/