INSPIRE Download Services with deegree 3.2: Part Three

The second part of this series focussed on harmonized INSPIRE datasets and their GML encoding. It also lined out why the technical features of the deegree WFS make it an excellent choice for serving valid, harmonized, GML-encoded INSPIRE datasets. Why was this important again? As the first part of the series described, data providers have to deal with this at some point in order to meet the upcoming INSPIRE requirement of interoperability.

Hands on!

This part of the series will provide an initial hands-on experience. If you’re ready to take the plunge, you can have an interoperable INSPIRE Download Service running on your machine in about five minutes. Here’s what you need:

* A Windows, Linux, Solaris or Mac OS X system.
* A supported Java version. Recommended are Oracle Java 7 (JDK) or OpenJDK 7.

If you don’t have a compatible Java version installed yet, download Oracle Java 7 (JDK variant), install it and try again. On Linux, you may prefer to install OpenJDK 7 using the package manager of your distribution.

Downloading and installing deegree webservices

First we need to grab a recent version of deegree webservices (it’s free and Open Source, so no license fees charged). Point your browser to http://www.deegree.org/Download and pick the latest stable version (at the moment of writing, this was 3.3.1). For the most hassle-free experience, pick the ZIP version, not the WAR (which requires a separate Java web container installation).

After downloading, simply extract the ZIP archive to a directory of your choice:

deegree-extracted

In the extraction directory, there’s one start script for each supported operating system:

* Windows: start-deegree-windows
* Linux/Solaris: start-deegree-linux.sh
* Mac OS X: start-deegree-osx.cmd

Fire up the starter. You should now see a terminal window on your screen with a lot of log messages running by:

deegree-terminal

(If this doesn’t work out, you most probably have an issue with the Java installation. Feel free to use one of the deegree support options to get help sorting it out.)

You can minimize this window, but don’t close it as long as you want to be able to use the deegree webservices. To access the administration interface, open http://localhost:8080 in your browser. You should see the following page:

deegree-console

deegree webservices are installed now and ready for configuration and usage. To shut deegree webservices down, switch back to the terminal window and press CTRL+C or simply close it.

If you want to learn more about the different flavors and installation options (e.g. how to make deegree webservices start automatically on system startup), please consult the installation chapter of the official documentation.

Activating the INSPIRE workspace

deegree webservices are well suited for creating Direct Access INSPIRE Download Services, but technically this is just a certain configuration for a deegree webservices installation: The software provides generic, highly configurable implementations of the following OGC standards: WFS, WMS, WMTS, CSW and WPS. Read more here. In order to get an INSPIRE Download Service set up, we could now create a deegree WFS and a feature store configuration for INSPIRE GML manually. However, it’s more comfortable to download a ready-to-use configuration for INSPIRE (a so-called “workspace”). A deegree workspace basically bundles configuration files for the different aspects of a deegree webservices instance:

workspace-overview

For now, we don’t need to dive into the different workspace aspects any deeper, but documentation is available, if you would like to learn more.

The deegree INSPIRE workspace is a basic INSPIRE View and Download Services setup. It contains a transactional WFS (2.0.0 and 1.1.0) configured for all Annex I Data Themes and a WMS (1.3.0 and 1.1.1) that is configured for three layers from three Annex I Data Themes. The workspace also contains some harmonized dutch base data for Administrative Units, Cadastral Parcels and Addresses (which we covered in the last part). The WFS is configured to behave as an INSPIRE Download service (Direct Access) that delivers the data as valid, harmonized INSPIRE GML and supports rich querying facilities.

In order to get the pre-configured INSPIRE workspace running, point your browser to http://localhost:8080 and perform the following three easy steps:

Step 1: Click the “workspaces” link (in the general menu section)

deegree-console

Step 2: Click “Import” (next to “deegree-workspace-inspire”)

deegree-workspace-inspire-import

The workspace will now be fetched from the artifact repository of the deegree project and extracted in a folder on your file system (in the deegree workspaces folder). Depending on your internet connection, this may take a while.

Step 3: Click “Start” (next to “deegree-workspace-inspire”)

deegree-workspace-inspire-start

The workspace will be initialized now. This takes a while, as the INSPIRE Annex I GML schemas are parsed and example GML datasets are loaded. The workspace will be removed from the list of inactive workspaces and displayed next to “Active workspace:” (below the deegree logo).

deegree-workspace-inspire-running

Your deegree instance is now running an interoperable, Direct Access INSPIRE Download Service (as well as an accompanying INSPIRE View Service)!

Note: If the machine running deegree webservices uses a proxy to access the internet and you don’t see any workspaces available for download, you most probably have to configure the proxy settings. Ask your network administrator for details and use the proxy menu link to set up proxy settings.

Performing WFS requests

So we now have an interoperable, Direct Access INSPIRE Download Service!? Very well, I hear you ask, but what can I do with it? How do we test that it actually works? Well, as it’s a standard-compliant OGC WFS service, you can connect to it using any compatible OGC WFS 2.0 client. Unfortunately, the client side is still lagging a bit behind: There are many WFS clients out there, but most of them don’t support WFS 2.0 yet (and equally important: they cannot deal with rich GML models). The only exception that I am aware of is Quantum GIS: There’s a WFS 2.0 plugin available which can be used to connect to WFS 2.0/INSPIRE Download Services and even render rich feature types:

deegree-workspace-inspire-qgis

However, you still won’t be able to work with the rich structure of the INSPIRE features using this client setup. In the future, the client situation should improve, but for now, you can only use raw WFS requests to see what your Direct Access Download Service really serves and is capabable of. The deegree INSPIRE workspace comes with some example WFS requests for testing.

To access these requests, click the “send requests” menu link:

deegree-workspace-inspire-send-requests

Use the drop-down menus to select an example request, e.g. to insert some harmonized INSPIRE Address features:

deegree-workspace-inspire-select-insert

Now click “Send”. The WFS response will be displayed in the lower part:

deegree-workspace-inspire-insert-results

This request inserted a few harmonized INSPIRE Addresses, which we can now request from the server. Again, there a some prepared GetFeature requests, which perform queries using different filter constraints.

deegree-workspace-inspire-select-query

The “ByThoroughfareName” example requests select Address features that are located in a specific street (in street “Madame Curiestraat”). Note that the response actually contains harmonized INSPIRE GML (as discussed in the last part of the series). To check schema-validity, you could validate in a schema-aware editor.

deegree-workspace-inspire-query-results

Don’t be shy to play around with the example requests. If you want to learn more about the rich possibilities of the WFS protocol, you may also want to have a look at the WFS 2.0 specification.

What’s next

Maybe you’re wondering: What is the storage backend for the features? Until now, we didn’t use a database. The INSPIRE workspace is pre-configured to use a so-called MemoryFeatureStore. However, for real-world usage, you most likely want to work with features stored in an SQL database with a spatial extension, such as PostgreSQL/PostGIS, Oracle Spatial or Microsoft SQL Server. For that, deegree webservices has a powerful relational mapping language for GML structures as well as a unique, specialized storage mode which is based on database BLOBs. The next part of the series will look into these storage options and the complimentary INSPIRE View Service that’s currently running unnoticed on your machine as well (in case you followed the hands-on instructions). If you don’t want to wait, please have a look at the respective parts of the official deegree webservices documentation. This section should be a good starting point.

Posted in Uncategorized | 2 Comments

Writing processes for deegree WPS

Finally, there’s an updated documentation for writing processes for the deegree WPS. It’s part of the new deegree webservices documentation, which has been released together with deegree 3.2.

deegree WPS is an implementation of the OGC Processing Service specification. Notable features:

  • Implements WPS standard 1.0.0
  • Supports KVP, XML and SOAP requests
  • Pluggable process provider layer
  • Easy-to-use API for implementing Java processes
  • Supports all variants of input/output parameters: literal, bbox, complex (binary and xml)
  • Streaming access for complex input/output parameters
  • Processing of huge amounts of data with minimal memory footprint
  • Supports storing of response documents/output parameters
  • Supports input parameters given inline and by reference
  • Supports RawDataOutput/ResponseDocument responses
  • Supports asynchronous execution (with polling of process status)

workspace-wps

For more information, please have a look at the deegree webservices documentation.

Posted in Uncategorized | Leave a comment

deegree webservices 3.2.1 released

Today, the first maintenance release for deegree webservices 3.2 has been released. Get it here:

http://www.deegree.org/Download

Changelog:

Grab your copy of deegree webservices 3.2.1, try it out, and please provide feedback on the mailing list.

Enjoy!

Posted in Uncategorized | Leave a comment

INSPIRE Download Services with deegree 3.2: Part Two

The first part of this series described the different classes of INSPIRE Download Services and why a Direct Access Download Service is most versatile. It also lined out that a Download Service may be interoperable or non-interoperable and that data providers must harmonize their base data at some point in order to meet the requirement of interoperability.

Harmonized INSPIRE datasets

This is probably one of the most important ongoing efforts in INSPIRE. The goal of harmonization is that the base data from the different member states and data providers will become available as datasets that comply with a single, interoperable data model. This harmonized INSPIRE data model is specified by 34 spatial data themes, which are grouped into three Annexes.

Let’s have a rough look at one of the Annex I data themes: Addresses

ds30

The Addresses data theme deals with: “Location of properties based on address identifiers, usually by road name, house number, postal code.” Its data model captures the concept of georeferenced addresses while integrating the semantical differences from all 27 involved member states. Here’s a part of this data model as an UML diagram (click for larger image):

address_schema

Encoding of harmonized datasets

The UML model is one thing, but which format does an interoperable Download Service use to encode compliant datasets? In the OGC world, the Geographic Markup Language (GML) is the lingua france for encoding geospatial information and geospatial domain objects. As INSPIRE makes heavy use of OGC specifications, it uses GML as interoperable encoding. Each INSPIRE data theme comes with a corresponding GML application schema, which provides a machine-readable encoding of the data model and can also be used for validating the syntax of GML-encoded datasets.

More information on GML encoding rules and GML application schemas can be found in the GML specification:

gml321

Here’s how a single GML-encoded Address object may look like:

<ad:Address xmlns:ad="urn:x-inspire:specification:gmlas:Addresses:3.0" gml:id="FEATURE_bf1fea07-ab3a-4db7-af10-8a669f8ab8ab">
  <ad:inspireId>
    <base:Identifier xmlns:base="urn:x-inspire:specification:gmlas:BaseTypes:3.2">
      <base:localId>0532200000000003</base:localId>
      <base:namespace>NL.KAD.BAG</base:namespace>
    </base:Identifier>
  </ad:inspireId>
  <ad:position>
    <ad:GeographicPosition>
      <ad:geometry>
        <gml:Point gml:id="GEOMETRY_b29eb313-0a93-4faf-8793-fabef02255e7" srsName="urn:ogc:def:crs:EPSG::4258">
          <gml:pos>52.689618 5.246345</gml:pos>
        </gml:Point>
      </ad:geometry>
      <ad:specification>entrance</ad:specification>
      <ad:method>byOtherParty</ad:method>
      <ad:default>true</ad:default>
    </ad:GeographicPosition>
  </ad:position>
  <ad:locator>
    <ad:AddressLocator>
      <ad:designator>
        <ad:LocatorDesignator>
          <ad:designator>1</ad:designator>
          <ad:type>2</ad:type>
        </ad:LocatorDesignator>
      </ad:designator>
      <ad:level>unitLevel</ad:level>
    </ad:AddressLocator>
  </ad:locator>
  <ad:validFrom>2009-01-05T23:00:00.000</ad:validFrom>
  <ad:validTo>2299-12-30T23:00:00.000</ad:validTo>
  <ad:beginLifespanVersion xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:endLifespanVersion xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:component xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#FEATURE_97d67632-2a1d-42ba-8fd0-163147c61e8e"/>
  <ad:component xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#FEATURE_4fb5f232-8045-4997-9922-eeb6e72e68c0"/>
  <ad:component xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#FEATURE_26d4b71a-2436-4a76-9a79-7c30c3e0ad58"/>
</ad:Address>

Note that the information of this address is not complete on its own, as the object references other objects via xlinks. For example, the information on the street name of this address is separated into a different object:

<ad:ThoroughfareName xmlns:ad="urn:x-inspire:specification:gmlas:Addresses:3.0" gml:id="FEATURE_26d4b71a-2436-4a76-9a79-7c30c3e0ad58">
  <ad:beginLifespanVersion xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:endLifespanVersion xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:validFrom xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:validTo xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:name>
    <gn:GeographicalName xmlns:gn="urn:x-inspire:specification:gmlas:GeographicalNames:3.0">
      <gn:language>nld</gn:language>
      <gn:nativeness>Endonym</gn:nativeness>
      <gn:nameStatus>Official</gn:nameStatus>
      <gn:sourceOfName>Het Kadaster, Nederland</gn:sourceOfName>
      <gn:pronunciation>
        <gn:PronunciationOfName/>
      </gn:pronunciation>
      <gn:spelling>
        <gn:SpellingOfName>
          <gn:text>Hugo de Grootsingel</gn:text>
          <gn:script>Latn</gn:script>
        </gn:SpellingOfName>
      </gn:spelling>
    </gn:GeographicalName>
  </ad:name>
</ad:ThoroughfareName>

(if you wonder: it’s the dutch address “Hugo de Grootsingel 1”)

Why is this important again? An Interoperable Download Service has to offer these GML-encoded objects in the end. Unfortunately, most WFS implementation are currently not ready to cope with providing valid GML 3.2-encodings of rich geo objects like the example. They will work perfectly for delivering GML-encodings of simple features, but can they deal with:

  • Multiple properties with the same name
  • Reference to other features or GML objects
  • Properties that contain GML core datatypes which are not geometries (e.g. code types or units of measure)
  • Properties that contain generic XML

The good news is that deegree WFS does. And it has more to offer.

Why deegree WFS stands out from the crowd

The deegree webservices documentation has a long list of features of the WFS implementation. Let me translate the important aspects into human readable language with regard to creating Direct Access Download Services:

  • Implements WFS standards 1.0.0, 1.1.0 and 2.0.0.: WFS 2.0.0 support is required for Direct Access Download Services.
  • High performance and excellent scalability: Important in order to meet INSPIRE Quality-of-Service requirements
  • Designed for rich data models from the bottom up: Required for smooth handling of harmonized INSPIRE datasets
  • Support for GetGmlObject requests and XLinks: References between individual INSPIRE features are treated and can be resolved as needed
  • Backends support flexible mapping of GML application schemas to relational models: It can be used for relational storage and harmonizing of INSPIRE datasets
  • Fully transactional (even for rich data models): Import harmonized INSPIRE datasets without dealing with the storage details

Not mentioned in this list is, that it has an alternative storage mode based on database BLOBs. The rationale for this is that reconstructing a rich feature from several tables can become rather slow, if the number of involved tables becomes too high. With regard to INSPIRE, it can become hard or even impossible to meet the quality-of-service requirements. However, with BLOB mode, you can store every feature in a single table column, regardless of the feature’s complexity.

And of course: It’s free and open. No license fees. Not locked down in any way. You can even change the code, if you like.

To my knowledge, this combination of features is not met by any other WFS implementation out there at the moment.

What’s next

The next part of this blog post series will describe how to install deegree webservices and how to get an interoperable, Direct Access Download Service running in a couple of minutes. Additionally, I will demonstrate how you can send requests against it and how to import INSPIRE datasets.

Posted in Uncategorized | 2 Comments

INSPIRE Download Services with deegree 3.2: Part One

Why has deegree webservices 3.2 been dubbed “INSPIRE release”? This blog post series explains why it is an excellent choice for providing compliant INSPIRE Download Services, especially if you want the full monty: Interoperable Direct Access Download Services that serve valid, harmonized datasets.

deegree
logo_inspire-300x300

What is an INSPIRE Download Service anyway?

From a technical perspective, this is best answered by the official Technical Guidance document for INSPIRE Download Services. The current version is 3.0 (published 12/06/2012):

ds30

To summarize, this document defines three different classes of INSPIRE Download Services:

  • Pre-defined Dataset Download Service based on ATOM
  • Pre-defined Dataset Download Service based on WFS
  • Direct Access Download Service

Basically, a Pre-defined Dataset Download Service based on ATOM provides service and dataset feeds that ultimately contain download links to the actual datasets. Datasets can only be downloaded in full. There are no filtering capabilities to query subsets.

A Pre-defined Dataset Download Service based on WFS is an OGC Web Feature Service 2.0 that complies to “Simple WFS”, “Query” and “HTTP GET” conformance classes. There’s just a single, mandatory stored query for retrieving full datasets using their identifier. Again, there are no filtering capabilities to query subsets.

A Direct Access Download Service extends a Pre-defined Dataset Download Service based on WFS by additional OGC WFS conformance classes: “Basic WFS”, “Ad hoc Query”, “Resource Identification”, “Minimum Standard Filter”, “Minimum Spatial Filter”, “Minimum Temporal Filter” and “Minimum XPath”.

Although Pre-defined Dataset Download Services may seem much simpler to implement, a Direct Access Download Service has the benefit that is offers filtering possiblities to WFS clients: Not only can you download a full dataset, but also a subset (e.g. the geo objects inside a bounding box). The possibilities for filtering are based on the OGC Filter Encoding 2.0 specification.

deegree WFS (contained in deegree webservices 3.2) allows to set up a Direct Access Download Service without any license fees and similar (or less) implementation complexity. Sounds to good to be true? Read on!

Interoperable vs. non-interoperable Download Services

Besides Pre-defined and Direct Access, INSPIRE guidelines make a distinction between Non-interoperable and Interoperable Download Services:

  • Non-interoperable (initial stages): Offered datasets are not required to be compliant
  • Interoperable: Offered datasets have to be compliant

Basically, the Non-interoperable phase is meant to reduce the efforts for data providers to meet the INSPIRE requirements. They can just offer their data in existing formats ‘as-is’. However, in the long run, all data offered by INSPIRE Download Services has to be compliant to the corresponding Data Theme Specification. This means that the data provider needs to transform their existing datasets into harmonized datasets that are valid with respect to the corresponding Data Theme Specification.

The following diagram (taken from the Technical Guidance) provides an overview of the milestones on the road to Interoperable Download Services (click for full size):

ds_roadmap

deegree WFS is suitable both for setting up both Non-interoperable and Interoperable Download Services. It can deal with the INSPIRE Data Model and return valid, harmonized INSPIRE GML datasets. Additionally, deegree’s GML mapping language can be used to perform data harmonization on-the-fly (but you can also perform data harmonization using any other technology).

What’s next?

The next part of this blog post series will discuss what harmonized datasets are exactly and why the deegree WFS implementation stands out for setting up Interoperable Direct Access Download Services.

Posted in Uncategorized | 2 Comments

deegree 3.2.0 webservices released

The deegree team is happy to announce the immediate availability of deegree webservices 3.2.0, available from:

http://www.deegree.org/Download

One of the major improvements to this release is a completely new written documentation, available at:

http://www.deegree.org/Documentation

The deegree 3.2.0 release (dubbed “INSPIRE release”) contains thirteen months worth of work. Major improvements:

  • New documentation (based on Sphinx)
  • New OGC service implementation: deegree WMTS (implements WMTS 1.0.0)
  • TileStore subsystem: A lean and fast data abstraction layer for map tiles
  • TileStore backends: GeoTIFF, File system, Remote WMS, Remote WMTS, EHCache
  • New concept for layer and theme configuration
  • WFS: Extended support for WFS 2.0 specification (e.g. Transactions)
  • WFS: Better compliance with INSPIRE Download Service TG 3.0
  • WMS: Better compliance with INSPIRE View Service TG 3.1
  • SQLDialect: Support for Microsoft SQL Server

Grab your copy of deegree webservices 3.2.0, try it out, and please provide feedback on the mailing list.

Enjoy!

Posted in Uncategorized | 4 Comments

deegree 3.1 webservices released

Some highlights:

  • WFS 2.0 support
  • Mapping of complex GML application schemas to relational models (PostGIS/Oracle)
  • Support for GeoTIFF raster pyramids as coverage stores
  • Support for cascading WMS
  • Better legend support in WMS
  • Improved relational model for CSW/ISO metadata store, added support for Oracle
  • Support for streaming inlined complex parameters in WPS
  • Metadata integration concept for INSPIRE-compliant service capabilities
  • Single download that includes all major OGC services (WMS, WFS, WPS, CSW)
  • Download example configurations (“workspaces”) from web administration interface

To get started, download a deegree webservices package and activate one of the official demo setups (“workspaces”) in the service console. Afterwards, you may want to check out the documentation available in the deegree wiki for learning about deegree’s configuration concepts.

Unfortunately, the wiki currently doesn’t describe some of the most interesting features of this release (e.g. the relational mapping configuration for the SQL feature store). Keep an eye on my blog, I plan to post some articles on these subjects…

Posted in Uncategorized | Leave a comment