Factory Data Storage
This is the developers' documentation of the Factory Data Storage component available in PISTIS factories.
Introduction
The Factory Data Storage acts as the primary database for a PISTIS factory. Factory users can upload their raw or processed datasets into this component as part of the data registration workflow. This component holds the actual datasets, while the Factory Data Catalogue contains the corresponding metadata. Each metadata record is represented as a dataset in the Factory Data Catalogue, with references to the actual datasets through distributions. Each distribution includes information about a dataset in the Factory Data Storage along with its unique ID. Following the data registration workflow, a dataset is created in the Factory Data Catalogue, featuring distributions that represent the actual dataset stored in the Factory Data Storage.
MongoDB and PostgreSQL are the main databases of this component along with an API to run CRUD operations on datasets in the database. MongoDB stores files and PostgreSQL stores datasets in the form of SQL tables. The API endpoints can only be authorized using a Bearer token from the PISTIS Keycloak.
Access
The API of the Factory Data Storage is available through: https://{FACTORYNAME}.pistis-market.eu/srv/factory-data-storage/
Usage - SQL tables
| Method | Input parameters | Output |
|---|---|---|
| POST /create_table | JSON body including the dataset schema and rows | 200 success with UUID of the newly created SQL table |
| PUT /add_rows | JSON body including the additional dataset schema and rows | 200 success with UUID of the updated SQL table |
| GET /get_table | asset_uuid , JSON_output | 200 status with Table schema and rows of the table as JSON |
| DELETE /delete_table | asset_uuid | 200 status if the table is deleted |
| GET /count_rows | asset_uuid | 200 status with the total number of rows of a table |
| GET /get_all_tables | -- | 200 status with a list of all SQL tables as JSON |
| GET /get_fields | asset_uuid , sort_column , sort_type , DISTINCT , OFFSET , LIMIT, JSON body with additional parameters | 200 status with rows of a table that satisfies the request as JSON |
Usage - Files
| Method | Input parameters | Output |
|---|---|---|
| POST /create_file | file | 200 status with UUID of the newly stored file |
| PUT /update_file | asset_uuid , file | 200 status with the UUID of the updated file |
| GET /get_file | asset_uuid | 200 status along with the file if it is available |
| GET /get_all_names | -- | 200 status with a list of all files as JSON |
| PUT /rename_file | asset_uuid , file_name | 200 status with confirmation that the file name was changed |
| DELETE /delete_file | asset_uuid | 200 status with the UUID of the deleted file |