Software
All the source code developed under this project is freely available at https://github.com/EOEPCA
Docker images are published at hub.docker.com/u/eoepca.
A selection of building blocks are available, addressing the three key domains identified in the architecture:
- Resource Management - catalogue and discovery of data and applications
- Processing and Chaining - deployment and execution of applications
- User Management - identity, security, access control and accounting
The Deployment Guide is available here.
Building Blocks
Login Service
User Management
The Login Service building block provides an OIDC and UMA compliant solution enabling authentication and authorization mechanisms within an Exploitation Platform. Other building blocks, such as the Policy Decision Point, Policy Enforcement Point and User Profile rely on this Building Block to provide several standard interfaces.
User Profile
User Management
In order to support the Billing Service, the User Profile building block allows to identify users (leaving a reference to their home IdP), to assign them Billing Identities, Service API keys, License keys and to record Terms and Conditions acceptance. It’s a persistence service with interfaces that will be queried by other building blocks (License Manager, Billing Service, Policy Decision Point) and modified by both the License Manager and the Login Service (during creation of a new user profile or assignment of new Licenses).
Policy Enforcement Point (PEP)
User Management
The main functionality of the PEP is to be able to stand between a client and the client’s desired resource. By creating this setup, where only the PEP is visible to an external request, we effectively secure whatever is behind the PEP. The PEP will enforce any policy for a resource configured in the Authorization Server, following the UMA 2.0 standard.
Policy Decision Point (PDP)
User Management
The main functionality of the PDP is to be able to perform complex Policy Decisions based on Policy Documents. In order to do so, several functionality blocks are identified:
- Policy Check Endpoint. A XACML-compliant endpoint that allows to submit JSON XACML requests and receive the corresponding responses.
- Policy Repository Management. An exposed API allowing to register and/or query policies assigned to specific resources.
Application Deployment & Execution Service (ADES)
Processing and Chaining
The Processing & Chaining domain area provides an extensible repository of processing functions, tools and applications that can be discovered by search query, invoked individually, and utilised in workflows. ADES is responsible for the execution of the processing service through both a OGC WPS 1.0 & 2.0 OWS service and an OGC Processes REST API. The processing request are executed within the target Exploitation Platform (i.e., the one that is close to the data).
Processor Development Environment (PDE)
Processing and Chaining
The Processor Development Environment provides a rich, interactive environment in which processing algorithms and services are developed, tested, debugged and ultimately packaged so that they can be deployed to the platform and published via the marketplace. It includes a JupyterLab instance and the Cloud Theia IDE.
Resource Catalogue
Resource Management
The purpose of the resource catalogue service is to register metadata from either data sources or processing results where they are indexed for discovery and search by downstream applications.
This service uses the COTS https://pycsw.org product with custom configuration.
Data Access Services
Resource Management
There are actually two kinds of deployments of the same software stack in two slightly different roles.
The first role is sort of system global instances, registering and serving upstream data products whereas the second one is used within the context of a user workspace. In this second context, only user provided data will be registered and served.
Workspace
Resource Management
The Workspace is responsible to coordinate a user’s resources within the platform, and make them available through Catalogue and Data Access services. In doing so it provides an abstraction of the underlying infrastructure. The Workspace provisions storage within the underlying infrastructure, typically buckets, on behalf of the user. Components needing access to this user storage, such as the ADES staging out processing results, interrogate the Workspace to obtain details of the storage.
Release v1.0
We are pleased to announce the release of v1.0 on 24 December 2021. This may be considered as a minimum viable product, to be followed by a series of incremental releases 1.1, 1.2, etc., extending the functionality.
Release v1.0 is a first formal release that includes versions of the following building blocks:
- Login Service
- User Profile
- Policy Enforcement Point (PEP)
- Policy Decision Point (PDP)
- Application Deployment & Execution Service (ADES)
- Processor Development Environment (PDE)
- Resource Catalogue
- Data Access Services
- Workspace
For more details see the Release Note and this description.
The release demonstrates the following capabilities:
User authentication
- Login with GitHub
- Login with ESA Commercial Operator Identity Hub (COIH)
- Login with username/password
Client Registration
- Dynamic client registration via SCIM endpoint
Authorisation
- Dynamic Resource Registration
Resource servers dynamic registration of resources - Resource protection
Enforcing a policy in which resources are owned and protected accordingly - Policy-based resource protection
Enforcing policy based upon policy rules maintained in the PDP
Processing Capabilities (ADES resource server)
- OGC WPS 2.0 and OGC API Processes interfaces
- Secure protected resource server, with access policy enforcement via PEP
- List available processes
- Deploy process (docker container with CWL application package)
- Execute process (create job)
- Get job status
- Data stage-in via STAC/OpenSearch catalogue reference
- Data stage-out to S3 bucket
- Undeploy process
- Integration of Calrissian CWL Workflow engine
Provides native Kubernetes integration and out-of-the-box support for a variety of execution patterns - such fan-in, fan-out, etc. - Dedicated user 'context' within ADES service
Processor Development Environment
- JupyterLab interface to interact with code and data
- Theia IDE to develop using an integrated development environment
- Local S3 Object Storage with MinIO to store EO data and results
- Jenkins instance to build the code with Continuous Integration
- Docker-in-Docker (with an Ubuntu host)
- Tools for application package testing
Resource Catalogue
- Implements ISO Metadata Application Profile 1.0.0
- Support for ISO-19115-1 and ISO-19115-2
- OGC CSW 3.0.0 and 2.0.2 interfaces
- Certified OGC Compliant and OGC Reference Implementation for both CSW 2.0.2 and CSW 3.0.0
- Harvesting support for WMS, WFS, WCS, WPS, WAF, CSW, SOS
- Federated catalogue distributed searching
- OGC API Records
- STAC (SpatioTemporal Asset Catalog)
- OpenSearch
- OGC OpenSearch Geo and Time Extensions
- OGC OpenSearch EO Extensions
Data Access Service
- OGC WMS 1.1 - 1.3 interfaces
- OGC WMTS 1.0 interfaces with automatic caching
- OGC WCS 2.0 interfaces with EO Application Profile
- OGC OpenSearch with EO, Geo and Time Extensions
- Workspace management API
- Dataset registration API
- Registration schemes for Sentinel-2 L1C/L2A data in Data Access Service and Ressource Catalogue
Data Harvesting
- Population of resource catalogue and data-access services from external data offerings
- Sources supported include: file-system, search service (e.g. OpenSearch), catalog file
- Filters, e.g. time, bbox, collection
- Post-processing to adjust harvested results, e.g. for completion of missing metadata
- Harvested results are transformed to STAC items for registration
End-to-end Processing Execution
- Authenticated user accessing protected ADES endpoints
- Dynamic creation of ADES user context with dynamic resource protection
- Processing inputs discovered in Resource Catalogue
- Processing inputs accessed via S3 (e.g. CREODIAS eodata)
- Processing stage-in using STAC file to describe inputs
- Processing execution on ADES
- Processing stage-in using STAC file to describe inputs
- Processing stage-out to S3 bucket
- Processing stage-out using STAC file to describe outputs
- Secure interfacing between ADES and user's protected Workspace
ADES client -> Get Workspace Details / (De-)Register Resources
User Workspace
- Secure protection of user-owned resources, with access policy enforcement via PEP
- User-specific S3 bucket for resource storage
- User-specific Resource Catalogue
- User-specific Data Access Services
- Secure protected management interface (workspace-api), with access policy enforcement via PEP
- Management functions via REST API...
- Create workspace (admin)
- Get workspace details (user)
- Delete workspace (admin)
- Patch workspace (admin)
- Redeploy workspace (admin)
- Register resources (user)
- Deregister resources (user)
Sample applications
- s-expression for EO product band math
Three application packages based-upon s-expression:- App s-expression
- App Water Mask
- App NDVI
- Normalized Hotspot Indices