The availability of free and open data, such as that provided by the Copernicus Sentinel fleet, together with the availability of affordable computing resources, create an opportunity for the wide adoption and use of Earth Observation (EO) data in all fields of our society. ESA's "EO Exploitation Platforms" initiative aims at facilitating adoption with the paradigm shift from "bring the data to the user" (i.e. user downloads data locally) to "bring the user to the data" (i.e. move user exploitation to hosted environments with collocated computing and storage). This leads to a platform-based ecosystem that provides infrastructure, data, compute and software as a service.
The resulting Exploitation Platform is where scientific and value adding activities are conducted, to generate targeted outputs for end-users. The goal of the "Common Architecture" is to define and agree the technical interfaces for the future exploitation of Earth Observation data in a distributed environment. The Common Architecture will thus provide the interfaces to facilitate the federation of different EO resources into a "Network of EO Resources". The "Common Architecture" will be defined using open interfaces that link the different resources (building blocks) so that a user can efficiently access and consume the disparate services of the "Network of EO Resources".
Telespazio VEGA UK - a Leonardo and Thales company will lead the definition of the Common Architecture through an open process of public discussion and consensus building with the EO community. It will be promoted as a Reference Architecture that will be designed to meet a broad set of use cases that cover Federated Identity Management, Processing & Chaining, and Data Access and Management. Leveraging free and open source software, a reusable Reference Implementation will be developed and deployed operationally, to act as a validation of the architecture and to provide an existing solution to third-parties.
The EO Exploitation Platform Common Architecture (EOEPCA) is an ESA funded project running from 2020 - 2023. This can be regarded as Phase 1 of the programme of work
that now continues with EOEPCA+ (Phase 2).
What is an Exploitation Platform?
A virtual work environment where expert users can access data, develop algorithms, conduct analysis and publish their value-adding outcomes. The platform facilitates
the sharing of data and applications to create collaborative communities of expertise.
Exploitation Platform as an Information Factory
We provide a perspective in which an Exploitation Platform supports the transformation of data to actionable information.
The infrastructure provides data, compute, and storage upon which the platform is built
Data must be discovered and made available for transformation
Processing transforms the data to more digestible forms – such as Analysis Ready Data, or input data for machine learning
Datacubes and Machine Learning models allow interpretation and extraction of Actionable Information
Dashboards make the information available interactively to decision-makers
EOEPCA Goals
There is an additional dimension, as there is an ecosystem of platforms with complementary offerings. Which brings us to the goal of the Common Architecture, which
is to encourage interoperation by reducing friction between platforms.
Architecture
We have defined a re-usable architecture whose interfaces are based on open standards – such as those provided by OGC, STAC, and for federating user identity across platforms.
Reference Implementation
We have implemented an open-source reference implementation of the architecture to encourage uptake.
The exploitation platform capability is broadly grouped into domain areas for Resource Management, Processing, and User Management.
With the support of our partner domain experts, we have designed and implemented a set of building blocks that implement the exploitation.
platform capabilities. The building blocks are designed to work together as a coherent system, but can equally be reused on their own.
Deployment Guide
A comprehensive guide to deploying the latest EOEPCA system, including;
The documentation is available below and can also be viewed in a normal browser window through the 'View the Guide' button above (N.B. This will open a new tab).
Software Overview
The software building blocks of our architecture are organized into three main categories. Click on the buttons below to learn more about each category.
User Management
Login Service
The Login Service building block provides an OIDC and UMA compliant solution enabling authentication and authorization mechanisms within an Exploitation Platform.
The User Profile building block allows to identify users (leaving a reference to their home IdP), to assign them Billing Identities, Service API keys, License keys and to record Terms and Conditions acceptance.
The main functionality of the PEP is to be able to stand between a client and the client's desired resource. By creating this setup, where only the PEP is visible to an external request, we effectively secure whatever is behind the PEP.
The Processing & Chaining domain area provides an extensible repository of processing functions, tools and applications that can be discovered by search query, invoked individually, and utilised in workflows.
The Processor Development Environment provides a rich, interactive environment in which processing algorithms and services are developed, tested and debugged before packaging and deployment to the platform.
The purpose of the resource catalogue service is to register metadata from either data sources or processing results where they are indexed for discovery and search by downstream applications.
The first role is sort of system global instances, registering and serving upstream data products whereas the second one is used within the context of a user workspace. In this second context, only user provided data will be registered and served.
The Workspace is responsible to coordinate a user's resources within the platform, and make them available through Catalogue and Data Access services. In doing so it provides an abstraction of the underlying infrastructure.
The Reference Implementation of our architecture can be deployed as a set of components (building-blocks) to
create an integrated exploitation platform. The Deployment tab above provides details and
supporting scripts to configure and make your own deployments - either to minikube (for testing) or into
your existing Kubernetes clusters.
System Design
The Master System Design EOEPCA.SDD.001 provides an EO Exploitation Platform architecture that meets the service needs of current and future systems, as defined by the use cases described in [EOEPCA-UC]. These use cases must be explored under 'real world' conditions by engagement with existing deployments, initiatives, user groups, stakeholders and sponsors within the user community and within overlapping communities, in order to gain a fully representative understanding of the functional requirements.
The system design takes into consideration existing precursor architectures (such as the Exploitation Platform Functional Model [EP-FM] and Thematic Exploitation Platform Open Architecture [TEP-OA]), including consideration of state-of-the-art technologies and approaches used by current related projects. The master system design describes functional blocks linked together by agreed standardised interfaces.
The importance of the OGC in these activities is recognised as a reference for the appropriate standards and in providing mechanisms to develop and evolve standards as required in the development of the architecture. In order to meet the design challenges we must apply the applicable existing OGC standards to the full set of federated use cases in order to expose deficiencies and identify needed evolution of the standards. Standards are equally important in all areas of the Exploitation Platform, including topics such as Authentication & Authorization Infrastructure (AAI), containerisation and provisioning of virtual cloud resources to ensure portability of compute between different providers of resource layer.
Data and metadata are fundamental considerations for the creation of an architecture in order to ensure full semantic interoperability between services. In this regard, data modelling and the consideration of data standards are critical activities.
The system design must go beyond the provision of a standalone EO Exploitation Platform, by intrinsically supporting federation of similar EO platforms at appropriate levels of the service stack. The Network of EO Resources seeks, ‘to unite the available - but scattered - European resources in a large federated and open environment’. In such a context, federation provides the potential to greatly enhance the utilization of data and services and provide as stimulus for research and commercial exploitation. From the end-user point of view, the federated system should present itself as a single consolidated environment in which all the federated resources are made available as an integrated system. Thus, the system design must specify federation-level interfaces that support this data and service-level interoperability in such a way that is seamless to the end users.
The goal is to create an Integrated Data Exploitation Environment. Users will apply their workflows close to the hosted data, supplemented by their own data. Processing outputs may be hosted as new products that can themselves contribute to the global catalogue. This paradigm can then be extended to encompass the federated set of Exploitation Platforms within the Network of EO Resources. The result is a Federated, Integrated Data Analysis Environment.
A Reference Implementation of the full architecture will be developed to prove the concepts and also to provide an off-the-shelf solution that can be instantiated by future projects to implement their EO Exploitation Platform, thus facilitating their ability to join the federated Network of EO Resources.
Thus, the Reference Implementation can be regarded as a set of re-usable platform services, in the context of a re-usable platform architecture.
Domain Overviews
The overall system design has been considered by taking the ‘Exploitation Platform – Functional Model’ [EP-FM] as a starting point and then evolving these ideas in the context of existing interface standards (with some emphasis on the OGC protocol suite) and the need for federated services.
Domain Areas
The system architecture is designed to meet the use cases as defined in [EOEPCA-UC] and [EP-FM]. [EOEPCA-UC] makes a high-level analysis of the use-cases to identify the main system functionalities organised into domain areas: 'User Management', 'Processing & Chaining' and 'Resource Management'. The high-level functionalities are often met by more than one domain area, and User Management (specifically Identity & Access Management) cuts across all use cases, and forms the basis of all access control restrictions that are applied to platform services and data.
Figure 1 depicts the domain areas as top level component blocks in a Platform ‘A’. The arrows may be read as “uses”, each implying one or more service interfaces.
A potential federation concept is represented by interactions between corresponding blocks in a collaborating Platform ‘B’. The architecture aims to minimise dependencies and is conducive to the principle of subcontracting the implementation to experts in the respective domains. The web portal integrates various client components to form a rich user-facing interface.
The Web Portal is depicted as it has interfaces with the other domain areas - but it is not a priority concern for the Common Architecture. Each exploitation platform would be expected to develop its own web interfaces according to its needs.
User Management
Responsible for all aspects related to user identity, authentication, authorization, accounting and billing in a federated system-of-systems environment.
It provides authentication, authorization and accounting services that are required to access other elements of the architecture, including the processing-oriented services and resource-oriented services. Individual resources may have an associated access and/or charging policy, but these have to be checked against the identity of the user. Resource consumption may also be controlled e.g. by credits and/or quotas associated with the user account. In the Network of EO Resources, a user should not need to create an account on multiple platforms. Therefore some interactions will be required between the User Management functions, whether directly or in directly via trusted third party.
Processing and Chaining
Provides access to a variety of processing functions, tools and applications, as well as execution environments in which to deploy them.
Provides a deployment and execution environment for processing tasks, analysis tools and interactive applications. Supports the chaining of processing tasks in workflows whose execution can include steps executed external to the origin exploitation platform. Handles and abstracts the low-level complexities of the different underlying compute technologies, and ensures the compute layer is scaled in accordance with current demand. Provides an integrated development environment to facilitate development of new processing algorithms and applications. Facilitating the network of EO resources by providing a federated interface to other processing services within the wider EO network.
The development and analysis environment provides a platform for the expert user to develop their own processing chains, experiments and workflows. It integrates with platform catalogue services (for data, processing services and applications) for discovery of available published datasets and processing elements. Subject to appropriate controls and permissions, the user can publish their own processing services and results. Workflows can be executed within the context of the processing facility, with the possibility to execute steps ‘remotely’ in collaborating platforms, with the results being collected for the continuation of the workflow.
Resource Management
Responsible for maintaining an inventory of platform and federated resources, and providing services for data access and visualisation.
Storage and cataloguing of all persistent resources. First and foremost, this will contain multidimensional geo-spatial datasets. In addition it may include a variety of heterogeneous data and other resources, such as documentation, Docker images, processing workflows, etc. Handles and abstracts the low-level complexities of different underlying storage technologies and strategies. Facilitating the network of EO resources by providing a federated interface to other data services within the wider EO network.
The catalogue holds corresponding metadata for every published resource item in the local platform storage, as well as entries for resources that are located on remote collaborating platforms. Catalogue search and data access is provided through a range of standard interfaces, which are used by the local Web Portal and Processing & Chaining elements and may be exposed externally as web services.
Access to services and resources is controlled according to an associated authorization policy as defined by the IAM approach. This component may interact with corresponding peer components on other platforms - for example to synchronise catalogue entries.
The user has a personal workspace in which to upload files, organise links to resources of interest (services/application/data), and receive/manage results output from processing executions. Shared workspaces for collaboration can be similarly provisioned. The ingestion of new data is controlled to ensure the quality of any published resource, including associated metadata, and to maintain the integrity of the catalogue.
Platform API
Defines standard interfaces at both service and programmatic levels.
The Service API and its associated Client Library together present a standard platform interface against which analysis and exploitation activities may be developed, and through which platform services can be federated. The Platform API encourages interoperation between platforms and provides a consistent and portable programming paradigm for expert users.
Web Portal
Presents the platform user interface for interacting with the local resources and processing facilities, as well as the wider network of EO resources.
The Web Portal provides the user interface (themed and branded according to the owning organisation) through which the user discovers the data/services available within the platform, and the analysis environment through which they can exploit these resources. It provides a rich, interactive web interface for discovering and working with all kinds of resources, including EO data, processing and documentation. It includes web service clients for smart search and data visualisations. It provides a workspace for developing and deploying processing algorithms, workflows, experiments and applications, and publishing results. It includes support and collaboration tools for the community.
Web Portal integrates together various web service clients that uses services provided by the specialist domains (Processing, Resource, User) on the local platform and collaborating platforms.
Architecture Layers
Figure 2 provides a simplified architectural view that illustrates the broad architecture layes of the Exploitation Platform, presented in the context of the infrastructure in which it is hosted and the end-users performing exploitation activities.
Resource TierThe Resource Tier represents the hosting infrastructure and provides the EO data, storage and compute upon which the exploitation platform is deployed.Platform TierThe Platform Tier represents the Exploitation Platform and the services it offers to end-users. The layers comprising the Platform Tier are further described below.Exploitation TierThe Exploitation Tier represents the end-users who exploit the services of the platform to perform analysis, or using high-level applications built-in on top of the platform’s services.
The Exploitation Platform builds upon the services provided by the hosting infrastructure - specifically accessing its data holding and using its compute resources. The components providing the EP services are deployed within the compute offering, with additional compute resources being provisioned on-demand to support end-user analysis activities.
The EP’s services access the data resources through a Data Access Gateway that provides an abstraction of the data access interface provided by the resource tier. This abstraction provides a 'standard' data access semantic that can be relied upon by other EP services - thus isolating specific data access concerns of the resource tier to a single EP component.
The Processing Framework provides the environment through which processing services and applications are executed in support of end-user analysis activities. It might be envisaged that some built-in (common) processing functions are provided, but the main focus of the processing framework is to support deployment and execution of bespoke end-user processing algorithms, and interactive analysis. Access to the underlying data from the executing processes is marshalled through the Data Access Gateway and its supporting Data Access Library.
The EP provides Catalogue services, so that end-users can discover and browse the resources available in the platform and its federated partners. Thus, end-users can discover available processing services and applications, and search for data available for inclusion in their analysis.
Data Services based upon open standards serve the clients of the Exploitation Platform for data access and data visualisation. Access to the underlying data is made via the Data Access Gateway.
The Service API represents the public service interfaces exposed by the Exploitation Platform for consumption by its clients. Covering all aspects of the EP (authentication, data/processing discovery, processing etc.), these interfaces are based upon open standards and are designed to offer a consistent EP service access semantic within the network of EO resources. Use of the network (HTTP) interfaces of the Service API is facilitated by the Client Library that provides bindings for common languages (Python, R, Javascript). The Client Library is a programmatic representation of the Service API which acts as an abstraction of the Exploitation Platform and so facilitates the development of portable client implementations.
The Exploitation Tier hosts the web clients with which the end-user interacts to conduct their analysis/exploitation activities. These clients would typically utilise the Client Library in their implementation. The Web IDE is an interactive web application that Experts use to perform interactive research and to develop algorithms. The Command-line Client builds upon the Client Library to provide a command-line tool that can be used, for example, to automate EP interactions through scripts.
The Web Portal provides the main user interface for the Exploitation Platform. It would be expected that each platform would provide its own bespoke portal implementation, and so is beyond the scope of the Common Architecture. Nevertheless, the architecture and its service interface must meet the needs anticipated by future exploitation platform implementations. Similarly, the External Application represents web applications (external to the hosting environment of the exploitation platform) that use the services of the EP via its Service API and Client Library.
All user interactions with the services of the EP are executed within the context of a given user and their rights to access resources, with associated resource usage billing. Thus, the Identity and Access Management component covers all tiers in this layered model.
Here you will find essential tools, documentation, and references to support your work with or implementation of the EOEPCA system. Whether you're looking for technical details or project guidelines, these resources are designed to help streamline your use of and contributions to EOEPCA+.
Tutorials
Exploratory Studies
Training
BiDS 2023 - Water Body Detection
Learn how to effectively package, share, and execute Earth observation workflows using the Common Workflow Language (CWL) standard. In particular there is a comprehensive example of a Water Body Detection Application package.
An investigation has been conducted into potential areas of integration between openEO and EOEPCA Processing capabilities. The report defines the full investigation and suggests two potential means to integrate the platforms. Learn more
openEO API EOEPCA Integration
This activity investigated offering an openEO API from an ADES Component, demonstrated in an EOPCA installation. The solution uses a python application to read and execute openEO Graphs, deployed as an EOEPCA Application Package.
In 2022 an in-depth EOEPCA operator training session was given. To find out more, access the official recording and download relevant documentation, visit here
The resulting Exploitation Platform is where scientific and value adding activities are conducted, to generate targeted outputs for end-users. The goal of the "Common Architecture" is to define and agree the technical interfaces for the future exploitation of Earth Observation data in a distributed environment. The Common Architecture will thus provide the interfaces to facilitate the federation of different EO resources into a "Network of EO Resources". The "Common Architecture" will be defined using open interfaces that link the different resources (building blocks) so that a user can efficiently access and consume the disparate services of the "Network of EO Resources".
Telespazio VEGA UK - a Leonardo and Thales company will lead the definition of the Common Architecture through an open process of public discussion and consensus building with the EO community. It will be promoted as a Reference Architecture that will be designed to meet a broad set of use cases that cover Federated Identity Management, Processing & Chaining, and Data Access and Management. Leveraging free and open source software, a reusable Reference Implementation will be developed and deployed operationally, to act as a validation of the architecture and to provide an existing solution to third-parties.
