LDES server implementation guide for developers

Living Document,

Previous Versions:
Editor:
Gerald Haesendonck (Ghent University - imec)

CC0 To the extent possible under law, the editors have waived all copyright and related or neighboring rights to this work. In addition, as of 20 January 2025, the editors have made this specification available under the Open Web Foundation Agreement Version 1.0, which is available at https://www.openwebfoundation.org/the-agreements/the-owf-1-0-agreements-granted-claims/owfa-1-0. Parts of this work may be from another specification document. If so, those parts are instead covered by the license of that specification document.

Abstract

This specification aims to be a practical guide for developers to create a Linked Data Event Stream (LDES) server.

1. Introduction

Creating a Linked Data Event Stream (LDES) server involves understanding two specifications, the LDES specification ([ldes-spec]) and the TREE hypermedia specification ([tree-spec]), upon which the LDES specification is built. The goal of this specification is to describe the requirements for a server that hosts a time-based fragmented LDES and provides a mechanism for adding members to an LDES, based on Linked Data Platform concepts.

Note: At this moment this document addresses the requirements of a specific use case. Existing implementations of generic LDES servers are LDES Solid Server and VSDS LDESServer4J.

2. LDES

2.1. Example and definition

This guide uses a running example to explain a Linked Data Event Stream and, in the next chapter, the operations the server supports.

The scenario is a scale that measures weights of a person. The readings are sent to an LDES server which keeps track of every reading.

A reading might look like this:

{
  "sensor": "http://example.com/sensors/weightSensor",
  "value": 78.4,
  "timestamp": "2025-01-01T08:00:00Z",
  "unit": "weightInKg"
}

Definition: A LDES is a collection representing a stream of unmodifiable objects called members.

Here is an example of a minimal LDES with three weight readings as members:

@prefix sosa: <http://www.w3.org/ns/sosa/> .
@prefix ldes: <https://w3id.org/ldes#> .
@prefix tree: <https://w3id.org/tree#> .
@prefix xsd:  <http://www.w3.org/2001/XMLSchema#> .
@prefix sensors: <http://example.com/sensors/> .
@prefix shapes: <http://example.com/shapes/> .

@base <http://example.com/weights/> .

<#EventStream> a ldes:EventStream ;
  ldes:timestampPath sosa:resultTime ;
  tree:shape shapes:memberShape ;
  tree:member <2025/01/weight1>, <2025/01/weight2>, <2025/02/weight3> .

<2025/01/weight1>
  a sosa:Observation ;
  sosa:madeBySensor sensors:weightSensor ;
  sosa:hasSimpleResult "78.4"^^xsd:decimal ;
  sosa:observedProperty: "weightInKg" ;
  sosa:resultTime "2025-01-01T08:00:00Z"^^xsd:dateTime .

<2025/01/weight2>
  a sosa:Observation ;
  sosa:madeBySensor sensors:weightSensor ;
  sosa:hasSimpleResult "78.6"^^xsd:decimal ;
  sosa:observedProperty: "weightInKg" ;
  sosa:resultTime "2025-01-15T08:00:00Z"^^xsd:dateTime .

<2025/02/weight3>
  a sosa:Observation ;
  sosa:madeBySensor sensors:weightSensor ;
  sosa:hasSimpleResult "78.1"^^xsd:decimal ;
  sosa:observedProperty: "weightInKg" ;
  sosa:resultTime "2025-02-01T07:15:00Z"^^xsd:dateTime .

Example 1 is formatted in Turtle [TURTLE], a compact human-readable serialisation of RDF [RDF-CONCEPTS], a graph-based data model for linked data. The same information could also be formatted in JSON for linking data [JSON-LD]:

{
   "@context" : {
      "@base": "http://example.com/weights/"
      "ldes" : "https://w3id.org/ldes#",
      "sensor" : {
         "@id" : "http://www.w3.org/ns/sosa/madeBySensor",
         "@type" : "@id"
      },
      "sosa" : "http://www.w3.org/ns/sosa/",
      "timestamp" : {
         "@id" : "http://www.w3.org/ns/sosa/resultTime",
         "@type" : "http://www.w3.org/2001/XMLSchema#dateTime"
      },
      "tree" : "https://w3id.org/tree#",
      "unit" : "http://www.w3.org/ns/sosa/observedProperty",
      "value" : "http://www.w3.org/ns/sosa/hasSimpleResult",
      "xsd" : "http://www.w3.org/2001/XMLSchema#",
      "sensors": "http://example.com/sensors/",
      "shapes": "http://example.com/shapes/"
   },
   "@graph" : [
      {
         "@id" : "#EventStream",
         "@type" : "ldes:EventStream",
         "ldes:timestampPath" : {"@id" : "sosa:resultTime"},
         "tree:shape": {"@id": "shapes:memberShape"},
         "tree:member" : [
           {"@id" : "2025/01/weight1"},
           {"@id" : "2025/01/weight2"},
           {"@id" : "2025/02/weight3"}
         ]
      },
      {
         "@id" : "2025/01/weight1",
         "@type" : "sosa:Observation",
         "sensor" : "sensors:weightSensor",
         "value" : 78.4,
         "timestamp" : "2025-01-01T08:00:00Z",
         "unit" : "weightInKg"
      },
      {
         "@id" : "2025/01/weight2",
         "@type" : "sosa:Observation",
         "value" : 78.6,
         "sensor" : "sensors:weightSensor",
         "timestamp" : "2025-01-15T08:00:00Z",
         "unit" : "weightInKg"
      },
      {
         "@id" : "2025/02/weight3",
         "@type" : "sosa:Observation",
         "value" : 78.1,
         "sensor" : "sensors:weightSensor",
         "timestamp" : "2025-02-01T07:15:00Z",
         "unit" : "weightInKg"
      }

   ]
}

The context (@context) maps terms to IRIs [IRI] (necessary for Linked data). The @graph contains the data itself. Thanks to the term mapping in the context, the members of the LDES almost look identical to the original readings in JSON.

Throughout this document Turtle and JSON-LD serialisations will be used.

2.2. Properties

The ldes:EventStream instance SHOULD have the following properties:

2.2.1. tree:member

This property indicates the members of the collection. Members are immutable objects in the collection forming the LDES.

2.2.2. tree:shape

This property defines the shape of the members. A shape can be seen as a schema for RDF graphs. All members of the stream have been validated by the shape. The shape MAY evolve over time, but because the members are immutable, it MUST be backwards compatible to earlier versions.

Here is a shape for the example, expressed in SHACL [SHACL]:

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix schema: <http://schema.org/> .
@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix sosa: <http://www.w3.org/ns/sosa/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix shapes: <http://example.com/shapes/> .

shapes:memberShape
  a sh:NodeShape ;
  sh:targetClass sosa:Observation;
  sh:property [
    sh:path sosa:madeBySensor ;
    sh:dataType xsd:string ;
    sh:minCount 1 ;
    sh:maxCount 1
  ],[
    sh:path sosa:hasSimpleResult ;
    sh:dataType xsd:decimal ;
    sh:minCount 1 ;
    sh:maxCount 1
  ],[
    sh:path sosa:observedProperty ;
    sh:dataType xsd:string ;
    sh:minCount 1 ;
    sh:maxCount 1
  ],[
    sh:path sosa:resultTime ;
    sh:dataType xsd:dateTime ;
    sh:minCount 1 ;
    sh:maxCount 1
  ] .

ldes:EventStream instance MAY have the following properties:

2.2.3. ldes:timestampPath

An important property is the timestamp field of the readings. Because the tree:member property simply lists the members, the timestampPath can be used to determine a time-based order. It can also be used as basis for fragmentation (split up a large LDES into smaller fragments, see later). In the weights example the JSON key containing timestamps is timestamp. Using the context this field maps to the timestampPath http://www.w3.org/ns/sosa/resultTime.

2.2.4. ldes:versionOfPath

If members represent a version of an object, this property indicates the non-version object. The followin example shows an LDES containing address changes:

ex:C2 a ldes:EventStream ;
  ldes:timestampPath dcterms:created ;
  ldes:versionOfPath dcterms:isVersionOf ;
  tree:shape ex:shape2.shacl ;
  tree:member ex:AddressRecord1-version1 .

ex:AddressRecord1-version1 dcterms:created 
  "2021-01-01T00:00:00Z"^^xsd:dateTime ;
  adms:versionNotes "First version of this address" ;
  dcterms:isVersionOf ex:AddressRecord1 ;
  dcterms:title "Streetname X, ZIP Municipality, Country" .

2.3. Fragmentation

A LDES can be split up into smaller parts or fragments, so clients don’t need to process potentially a lot of data, and in the context of time series older fragments are unlikely to change and can easily be cached when serving over HTTP. LDES relies on the TREE specification ([tree-spec]) for fragmentation supporting many options. In this document we pick time-based fragmentation because that’s the most interesting for time series data. The ldes:timestampPath property is used to determine to which fragment a reading gets added.

Conformance

Conformance requirements are expressed with a combination of descriptive assertions and RFC 2119 terminology. The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in the normative parts of this document are to be interpreted as described in RFC 2119. However, for readability, these words do not appear in all uppercase letters in this specification.

All of the text of this specification is normative except sections explicitly marked as non-normative, examples, and notes. [RFC2119]

Examples in this specification are introduced with the words “for example” or are set apart from the normative text with class="example", like this:

This is an example of an informative example.

Informative notes begin with the word “Note” and are set apart from the normative text with class="note", like this:

Note, this is an informative note.

Index

Terms defined by this specification

References

Normative References

[IRI]
M. Duerst; M. Suignard. Internationalized Resource Identifiers (IRIs). January 2005. Proposed Standard. URL: https://www.rfc-editor.org/rfc/rfc3987
[JSON-LD]
Manu Sporny; Gregg Kellogg; Markus Lanthaler. JSON-LD 1.0. 3 November 2020. REC. URL: https://www.w3.org/TR/json-ld/
[LDES-SPEC]
Pieter Colpaert. Linked Data Event Streams. URL: https://w3id.org/ldes/specification
[RDF-CONCEPTS]
Graham Klyne; Jeremy Carroll. Resource Description Framework (RDF): Concepts and Abstract Syntax. URL: https://w3c.github.io/rdf-concepts/spec/
[RFC2119]
S. Bradner. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL: https://datatracker.ietf.org/doc/html/rfc2119
[SHACL]
Holger Knublauch; Dimitris Kontokostas. Shapes Constraint Language (SHACL). URL: https://w3c.github.io/data-shapes/shacl/
[TREE-SPEC]
Pieter Colpaert. The TREE hypermedia specification. URL: https://w3id.org/tree/specification
[TURTLE]
Eric Prud'hommeaux; Gavin Carothers. RDF 1.1 Turtle. URL: https://w3c.github.io/rdf-turtle/spec/