Conference Paper

Earth Science data are composite, multidimensional and of significant size, and as such, pose a number of challenges upon their management and handling. With the amount of data sources (e.g, drones, sensor networks or higher level products generators) and the rate of generated data continuously increasing, yet another challenge is arising: making the information existing in multiple heterogeneous resources available efficiently. Currently, Array Database Management Systems provide the mandatory set of tools - storage and processing oriented - for managing unlimited-size multi-dimensional arrays, while OGC standards (e.g., WCS and WCPS) provide the guidelines for the design and exposed behavior of web accessible Earth Science data management engines. Through those approaches, the realization of standardized Earth Science big-data oriented infrastructures is facilitated, as shown in a number of cases.
Our approach, namely the EarthServer Search Engine, builds on top of these elements and manages to bring together diverse data sources while also enhancing the user-friendliness of the underlying array database management systems. An abstract data model is introduced, that extends and combines metadata from different standards, and enables the construction of an one-stop shop, with the use of tools that discover and harvest such data, in order to facilitate access to all of them for the end-user. All the information retrieved is exposed through well-established standards such as OGC CSW, OAI-PMH or OpenSearch. Additionaly, a query language is proposed, which adds a rich set of features on top of the Web Coverage Processing Service (WCPS) Language Interface Standard, by adopting the FLOWR expression syntax popularized by XQuery. The new language, with the assistance of a number of coordinating services, merges the path these two widely adopted standards have paved, to offer a more expressive way of querying array data, to allow mixed search over both data and metadata as well as federated search, and to add result manipulation capabilities with the option of generating mixed results.
Through these, the EarthServer Search Engine allows the discovery and processing of large data sets across heterogeneous infrastructures, and provides answers to requests which address metadata while also giving processing directives, offering advanced, and provider independent data analysis in a compact way. Therefore, the flow of scientific data resources among working groups and systems is greatly facilitated and the processing logic is further decoupled from data organization and management.
The effectiveness of this approach has been evaluated in the context of the EarthServer project, by bringing together five high volume array databases.