Working With Metadata

The catalogue has a very simple data structure:

  • On the highest level you have catalogues
  • Each catalogue can contain datasets
  • Each dataset holds distributions
  • A distribution represents the actual data - aka hold a reference to the data
You can learn more about it here: https://semiceu.github.io/DCAT-AP/releases/3.0.0/

About RDF

The catalogue is based upon the Resource Description Framework. So you need to pass your payload always in a RDF format. The catalogue supports a variety of these formats, but we will focus on Turtle and JSON-LD. It is recommend that you make yourself familiar with RDF before using the catalogue API.

You should avoid managing RDF on a string level, because the the RDF data model does not offer an unambiguous serialization. There are various libraries to handle RDF: https://github.com/semantalytics/awesome-semantic-web?#programming

Access Control

Currently the API is protected with a simple API key. You can obtain it via contacting Fraunhofer FOKUS. You pass it in the header field X-API-Key. There is no fine-grained access control yet. So be careful and only work with your own data.

Managing Catalogues

First you need to create a catalogue.

Creating or Updating a Catalogue

Turtle
PUT https://develop.pistis-market.eu/srv/repo/catalogues/<catalog-id>
Content-Type: text/turtle
X-API-Key: <api-key>


@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix dct:  <http://purl.org/dc/terms/> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .

<https://piveau.io/id/catalogue/data-company>
    a                dcat:Catalog;
    dct:creator      <https://piveau.eu/def/creator>;
    dct:description  "The Data Company specializes in curated datasets for the education and public sectors."@en;
    dct:issued       "2023-01-18T06:33:36Z"^^<http://www.w3.org/2001/XMLSchema#dateTime>;
    dct:modified     "2023-01-18T06:33:36Z"^^<http://www.w3.org/2001/XMLSchema#dateTime>;
    dct:language     <http://publications.europa.eu/resource/authority/language/ENG>;
    dct:spatial      <http://publications.europa.eu/resource/authority/country/GRC>;
    dct:title        "The Data Company"@en;
    dct:type         "dcat-ap";
    foaf:homepage    <https://www.the-data-company.eu/> .

<https://piveau.eu/def/creator>
    a          foaf:Agent;
    foaf:name  "The Data Company" .
JSON-LD
PUT https://develop.pistis-market.eu/srv/repo/catalogues/<catalog-id>
Content-Type: application/ld+json
X-API-Key: <api-key>

{
  "@context": {
    "dcat": "http://www.w3.org/ns/dcat#",
    "foaf": "http://xmlns.com/foaf/0.1/",
    "dcterms": "http://purl.org/dc/terms/",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  },
  "@graph": [
    {
      "@id": "https://piveau.eu/def/creator",
      "@type": "foaf:Agent",
      "foaf:name": "The Data Company"
    },
    {
      "@id": "https://piveau.io/id/catalogue/data-company",
      "@type": "dcat:Catalog",
      "dcterms:creator": {
        "@id": "https://piveau.eu/def/creator"
      },
      "dcterms:description": {
        "@language": "en",
        "@value": "The Data Company specializes in curated datasets for the education and public sectors."
      },
      "dcterms:issued": {
        "@type": "http://www.w3.org/2001/XMLSchema#dateTime",
        "@value": "2023-01-18T06:33:36Z"
      },
      "dcterms:language": {
        "@id": "http://publications.europa.eu/resource/authority/language/ENG"
      },
      "dcterms:modified": {
        "@type": "http://www.w3.org/2001/XMLSchema#dateTime",
        "@value": "2023-01-18T06:33:36Z"
      },
      "dcterms:spatial": {
        "@id": "http://publications.europa.eu/resource/authority/country/GRC"
      },
      "dcterms:title": {
        "@language": "en",
        "@value": "The Data Company"
      },
      "dcterms:type": "dcat-ap",
      "foaf:homepage": {
        "@id": "https://www.the-data-company.eu/"
      }
    }
  ]
}

You will find your catalogue the here: https://develop.pistis-market.eu/srv/catalog/catalogues/<catalog-id>

Deleting a Catalogue

DELETE https://develop.pistis-market.eu/srv/repo/catalogues/<catalog-id>
X-API-Key: <api-key>
Deleting a catalogue deletes all its datasets.

Managing Datasets

Creating a Dataset

If you have created a catalogue you can easily add datasets to it.

Turtle
POST https://develop.pistis-market.eu/srv/repo/catalogues/<catalog-id>/datasets
Content-Type: text/turtle
X-API-Key: <api-key>

@prefix dcat:                <http://www.w3.org/ns/dcat#> .
@prefix dct:                 <http://purl.org/dc/terms/> .
@prefix xsd:                 <http://www.w3.org/2001/XMLSchema#> .
@prefix odrl:                <http://www.w3.org/ns/odrl/2/> .
@prefix vcard:               <http://www.w3.org/2006/vcard/ns#> .
@prefix foaf:                <http://xmlns.com/foaf/0.1/> .
@prefix skos:                <http://www.w3.org/2004/02/skos/core#> .

<https://piveau.io/set/data/test-dataset>
    a                   dcat:Dataset ;
    dct:description     "Description of a Dataset"@en ;
    dct:title           "Title of a Dataset"@en ;
    dcat:keyword        "Engagement"@en, "Volunteerism"@en, "Stakeholders"@en, "Universities"@en, "Trends"@en   ;
    dct:publisher       [ a     foaf:Organization ;
                                foaf:mbox <mailto:email@publisher.eu> ;
                                foaf:name "Publisher Limited" ; ] ;
    dcat:theme          <http://publications.europa.eu/resource/authority/data-theme/EDUC> ;
    dct:language        <http://publications.europa.eu/resource/authority/language/ENG> ;
    dct:issued          "2023-05-25T00:00:00"^^xsd:dateTime ;
    dct:modified        "2023-05-25T00:00:00"^^xsd:dateTime ;
    dcat:distribution   <https://piveau.io/set/distribution/1> .

<https://piveau.io/set/distribution/1>
    a              dcat:Distribution ;
    dct:title      "Name of the actual File" ;
    dct:license    [
                        dct:identifier "SUB-LI" ;
                        dct:title "Subscription License" ;
                        skos:prefLabel "Subscription License" ;
                        skos:exactMatch <https://subscriptionlicense.com>
                   ] ;
    dct:format     <http://publications.europa.eu/resource/authority/file-type/CSV> ;
    dcat:byteSize  "1288490189"^^xsd:nonNegativeInteger ;
    dcat:accessURL <http://factory-storage/test-document_company2> .
JSON-LD
POST https://develop.pistis-market.eu/srv/repo/catalogues/<catalog-id>/datasets
Content-Type: application/ld+json
X-API-Key: <api-key>

{
  "@context": {
    "dcat": "http://www.w3.org/ns/dcat#",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "vcard": "http://www.w3.org/2006/vcard/ns#",
    "foaf": "http://xmlns.com/foaf/0.1/",
    "skos": "http://www.w3.org/2004/02/skos/core#",
    "dcterms": "http://purl.org/dc/terms/",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "odrl": "http://www.w3.org/ns/odrl/2/"
  },
  "@graph": [
    {
      "@id": "_:g0",
      "@type": "foaf:Organization",
      "foaf:mbox": {
        "@id": "mailto:email@publisher.eu"
      },
      "foaf:name": "Publisher Limited"
    },
    {
      "@id": "_:g1",
      "dcterms:identifier": "SUB-LI",
      "dcterms:title": "Subscription License",
      "skos:exactMatch": {
        "@id": "https://subscriptionlicense.com"
      },
      "skos:prefLabel": "Subscription License"
    },
    {
      "@id": "https://piveau.io/set/data/test-dataset",
      "@type": "dcat:Dataset",
      "dcterms:description": {
        "@language": "en",
        "@value": "Description of a Dataset"
      },
      "dcterms:issued": {
        "@type": "xsd:dateTime",
        "@value": "2023-05-25T00:00:00"
      },
      "dcterms:language": {
        "@id": "http://publications.europa.eu/resource/authority/language/ENG"
      },
      "dcterms:modified": {
        "@type": "xsd:dateTime",
        "@value": "2023-05-25T00:00:00"
      },
      "dcterms:publisher": {
        "@id": "_:g0"
      },
      "dcterms:title": {
        "@language": "en",
        "@value": "Title of a Dataset"
      },
      "dcat:distribution": {
        "@id": "https://piveau.io/set/distribution/1"
      },
      "dcat:keyword": [
        {
          "@language": "en",
          "@value": "Engagement"
        },
        {
          "@language": "en",
          "@value": "Volunteerism"
        },
        {
          "@language": "en",
          "@value": "Stakeholders"
        },
        {
          "@language": "en",
          "@value": "Universities"
        },
        {
          "@language": "en",
          "@value": "Trends"
        }
      ],
      "dcat:theme": {
        "@id": "http://publications.europa.eu/resource/authority/data-theme/EDUC"
      }
    },
    {
      "@id": "https://piveau.io/set/distribution/1",
      "@type": "dcat:Distribution",
      "dcterms:format": {
        "@id": "http://publications.europa.eu/resource/authority/file-type/CSV"
      },
      "dcterms:license": {
        "@id": "_:g1"
      },
      "dcterms:title": "Name of the actual File",
      "dcat:accessURL": {
        "@id": "http://factory-storage/test-document_company2"
      },
      "dcat:byteSize": {
        "@type": "xsd:nonNegativeInteger",
        "@value": "1288490189"
      }
    }
  ]
}

The request will return you the UUID of your dataset, which you need for follow-up requests.

{
    "id": "f1da9a7d-4f44-4511-8f8e-ff7b50ed0f5e"
}

Creating a Streaming Dataset

For streaming data, the access mechanism should be defined using a dcat:DataService within the distribution. Example:

Turtle
POST https://develop.pistis-market.eu/srv/repo/catalogues/<catalog-id>/datasets
Content-Type: text/turtle
X-API-Key: <api-key>

<your-prefixes>

<your-dcat:Dataset>

<https://piveau.io/set/distribution/1>
    a              dcat:Distribution ;
    dct:title      "Kafka Stream Distribution" ;
    dct:format     <http://publications.europa.eu/resource/authority/file-type/CSV> ;
    dcat:accessService  [ rdf:type                  dcat:DataService;
                          dct:title                 "Kafka Stream"@en;
                          dcat:endpointDescription  <https://example.com>;
                          dcat:endpointURL          <https://develop.pistis-market.eu/srv/data-connector/api/provider/kafka/this-is-an-id-1234>
                        ] .
JSON-LD
POST https://develop.pistis-market.eu/srv/repo/catalogues/<catalog-id>/datasets
Content-Type: application/ld+json
X-API-Key: <api-key>

{
  "@context": {
    "<your-prefixes>"
  },
  "@graph": [
    {
      "<your-dcat:Dataset>"
     
    },
    {
      "@id": "https://piveau.io/set/distribution/1",
      "@type": "dcat:Distribution",
      "dcterms:format": {
        "@id": "http://publications.europa.eu/resource/authority/file-type/CSV"
      },
      "dcterms:title": "Kafka Stream Distribution",
      "dcat:accessService": {
        "@type": "dcat:DataService",
        "dct:title": {
          "@value": "Kafka Stream",
          "@language": "en"
        },
        "dcat:endpointDescription": {
          "@id": "https://example.com"
        },
        "dcat:endpointURL": {
          "@id": "https://develop.pistis-market.eu/srv/data-connector/api/provider/kafka/this-is-an-id-1234"
        }
      }
    }
  ]
}

Note that dcat:endpointDescription is a description of the services available via the end-points, including their operations, parameters etc. and it may be expressed in a machine‑readable form. If no such description exists, you may omit this property from your metadata. See: https://www.w3.org/TR/vocab-dcat-2/#Property:data_service_endpoint_description. More informations and examples about Data Services can be found here: https://www.w3.org/TR/vocab-dcat-3/#examples-data-service

Setting the Correct Format

The property dct:format MUST be set with the concept URI of the EU File Type Vocabulary. Some examples relevant for PISTIS:

Reading a Dataset

You get read your dataset with a simple GET request.

Turtle
GET https://develop.pistis-market.eu/srv/repo/datasets/<dataset-id>
Accept: text/turtle
JSON-LD
GET https://develop.pistis-market.eu/srv/repo/datasets/<dataset-id>
Accept: application/ld+json

Updating a Dataset

You can update a dataset by performing a PUT request and adding the payload in the body.

Turtle
PUT https://develop.pistis-market.eu/srv/repo/datasets/<dataset-id>
Content-Type: text/turtle
X-API-Key: <api-key>

<your-payload>
JSON-LD
PUT https://develop.pistis-market.eu/srv/repo/datasets/<dataset-id>
Content-Type: application/ld+json
X-API-Key: <api-key>

<your-payload>

Deleting a Dataset

Turtle
DELETE https://develop.pistis-market.eu/srv/repo/datasets/<dataset-id>
X-API-Key: <api-key>
JSON-LD
DELETE https://develop.pistis-market.eu/srv/repo/datasets/<dataset-id>
X-API-Key: <api-key>

Searching

Use the following endpoint to search for datasets: https://{factoryName}.pistis-market.eu/srv/search/

And please refer to this documentation for more information on searching for datasets.

Table of Contents