Writing processes for deegree WPS

Finally, there’s an updated documentation for writing processes for the deegree WPS. It’s part of the new deegree webservices documentation, which has been released together with deegree 3.2.

deegree WPS is an implementation of the OGC Processing Service specification. Notable features:

  • Implements WPS standard 1.0.0
  • Supports KVP, XML and SOAP requests
  • Pluggable process provider layer
  • Easy-to-use API for implementing Java processes
  • Supports all variants of input/output parameters: literal, bbox, complex (binary and xml)
  • Streaming access for complex input/output parameters
  • Processing of huge amounts of data with minimal memory footprint
  • Supports storing of response documents/output parameters
  • Supports input parameters given inline and by reference
  • Supports RawDataOutput/ResponseDocument responses
  • Supports asynchronous execution (with polling of process status)

workspace-wps

For more information, please have a look at the deegree webservices documentation.

Posted in Uncategorized | Leave a comment

deegree webservices 3.2.1 released

Today, the first maintenance release for deegree webservices 3.2 has been released. Get it here:

http://www.deegree.org/Download

Changelog:

Grab your copy of deegree webservices 3.2.1, try it out, and please provide feedback on the mailing list.

Enjoy!

Posted in Uncategorized | Leave a comment

INSPIRE Download Services with deegree 3.2: Part Two

The first part of this series described the different classes of INSPIRE Download Services and why a Direct Access Download Service is most versatile. It also lined out that a Download Service may be interoperable or non-interoperable and that data providers must harmonize their base data at some point in order to meet the requirement of interoperability.

Harmonized INSPIRE datasets

This is probably one of the most important ongoing efforts in INSPIRE. The goal of harmonization is that the base data from the different member states and data providers will become available as datasets that comply with a single, interoperable data model. This harmonized INSPIRE data model is specified by 34 spatial data themes, which are grouped into three Annexes.

Let’s have a rough look at one of the Annex I data themes: Addresses

ds30

The Addresses data theme deals with: “Location of properties based on address identifiers, usually by road name, house number, postal code.” Its data model captures the concept of georeferenced addresses while integrating the semantical differences from all 27 involved member states. Here’s a part of this data model as an UML diagram (click for larger image):

address_schema

Encoding of harmonized datasets

The UML model is one thing, but which format does an interoperable Download Service use to encode compliant datasets? In the OGC world, the Geographic Markup Language (GML) is the lingua france for encoding geospatial information and geospatial domain objects. As INSPIRE makes heavy use of OGC specifications, it uses GML as interoperable encoding. Each INSPIRE data theme comes with a corresponding GML application schema, which provides a machine-readable encoding of the data model and can also be used for validating the syntax of GML-encoded datasets.

More information on GML encoding rules and GML application schemas can be found in the GML specification:

gml321

Here’s how a single GML-encoded Address object may look like:

<ad:Address xmlns:ad="urn:x-inspire:specification:gmlas:Addresses:3.0" gml:id="FEATURE_bf1fea07-ab3a-4db7-af10-8a669f8ab8ab">
  <ad:inspireId>
    <base:Identifier xmlns:base="urn:x-inspire:specification:gmlas:BaseTypes:3.2">
      <base:localId>0532200000000003</base:localId>
      <base:namespace>NL.KAD.BAG</base:namespace>
    </base:Identifier>
  </ad:inspireId>
  <ad:position>
    <ad:GeographicPosition>
      <ad:geometry>
        <gml:Point gml:id="GEOMETRY_b29eb313-0a93-4faf-8793-fabef02255e7" srsName="urn:ogc:def:crs:EPSG::4258">
          <gml:pos>52.689618 5.246345</gml:pos>
        </gml:Point>
      </ad:geometry>
      <ad:specification>entrance</ad:specification>
      <ad:method>byOtherParty</ad:method>
      <ad:default>true</ad:default>
    </ad:GeographicPosition>
  </ad:position>
  <ad:locator>
    <ad:AddressLocator>
      <ad:designator>
        <ad:LocatorDesignator>
          <ad:designator>1</ad:designator>
          <ad:type>2</ad:type>
        </ad:LocatorDesignator>
      </ad:designator>
      <ad:level>unitLevel</ad:level>
    </ad:AddressLocator>
  </ad:locator>
  <ad:validFrom>2009-01-05T23:00:00.000</ad:validFrom>
  <ad:validTo>2299-12-30T23:00:00.000</ad:validTo>
  <ad:beginLifespanVersion xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:endLifespanVersion xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:component xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#FEATURE_97d67632-2a1d-42ba-8fd0-163147c61e8e"/>
  <ad:component xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#FEATURE_4fb5f232-8045-4997-9922-eeb6e72e68c0"/>
  <ad:component xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#FEATURE_26d4b71a-2436-4a76-9a79-7c30c3e0ad58"/>
</ad:Address>

Note that the information of this address is not complete on its own, as the object references other objects via xlinks. For example, the information on the street name of this address is separated into a different object:

<ad:ThoroughfareName xmlns:ad="urn:x-inspire:specification:gmlas:Addresses:3.0" gml:id="FEATURE_26d4b71a-2436-4a76-9a79-7c30c3e0ad58">
  <ad:beginLifespanVersion xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:endLifespanVersion xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:validFrom xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:validTo xsi:nil="true" nilReason="UNKNOWN"/>
  <ad:name>
    <gn:GeographicalName xmlns:gn="urn:x-inspire:specification:gmlas:GeographicalNames:3.0">
      <gn:language>nld</gn:language>
      <gn:nativeness>Endonym</gn:nativeness>
      <gn:nameStatus>Official</gn:nameStatus>
      <gn:sourceOfName>Het Kadaster, Nederland</gn:sourceOfName>
      <gn:pronunciation>
        <gn:PronunciationOfName/>
      </gn:pronunciation>
      <gn:spelling>
        <gn:SpellingOfName>
          <gn:text>Hugo de Grootsingel</gn:text>
          <gn:script>Latn</gn:script>
        </gn:SpellingOfName>
      </gn:spelling>
    </gn:GeographicalName>
  </ad:name>
</ad:ThoroughfareName>

(if you wonder: it’s the dutch address “Hugo de Grootsingel 1″)

Why is this important again? An Interoperable Download Service has to offer these GML-encoded objects in the end. Unfortunately, most WFS implementation are currently not ready to cope with providing valid GML 3.2-encodings of rich geo objects like the example. They will work perfectly for delivering GML-encodings of simple features, but can they deal with:

  • Multiple properties with the same name
  • Reference to other features or GML objects
  • Properties that contain GML core datatypes which are not geometries (e.g. code types or units of measure)
  • Properties that contain generic XML

The good news is that deegree WFS does. And it has more to offer.

Why deegree WFS stands out from the crowd

The deegree webservices documentation has a long list of features of the WFS implementation. Let me translate the important aspects into human readable language with regard to creating Direct Access Download Services:

  • Implements WFS standards 1.0.0, 1.1.0 and 2.0.0.: WFS 2.0.0 support is required for Direct Access Download Services.
  • High performance and excellent scalability: Important in order to meet INSPIRE Quality-of-Service requirements
  • Designed for rich data models from the bottom up: Required for smooth handling of harmonized INSPIRE datasets
  • Support for GetGmlObject requests and XLinks: References between individual INSPIRE features are treated and can be resolved as needed
  • Backends support flexible mapping of GML application schemas to relational models: It can be used for relational storage and harmonizing of INSPIRE datasets
  • Fully transactional (even for rich data models): Import harmonized INSPIRE datasets without dealing with the storage details

Not mentioned in this list is, that it has an alternative storage mode based on database BLOBs. The rationale for this is that reconstructing a rich feature from several tables can become rather slow, if the number of involved tables becomes too high. With regard to INSPIRE, it can become hard or even impossible to meet the quality-of-service requirements. However, with BLOB mode, you can store every feature in a single table column, regardless of the feature’s complexity.

And of course: It’s free and open. No license fees. Not locked down in any way. You can even change the code, if you like.

To my knowledge, this combination of features is not met by any other WFS implementation out there at the moment.

What’s next

The next part of this blog post series will describe how to install deegree webservices and how to get an interoperable, Direct Access Download Service running in a couple of minutes. Additionally, I will demonstrate how you can send requests against it and how to import INSPIRE datasets.

Posted in Uncategorized | 1 Comment

INSPIRE Download Services with deegree 3.2: Part One

Why has deegree webservices 3.2 been dubbed “INSPIRE release”? This blog post series explains why it is an excellent choice for providing compliant INSPIRE Download Services, especially if you want the full monty: Interoperable Direct Access Download Services that serve valid, harmonized datasets.

deegree
logo_inspire-300x300

What is an INSPIRE Download Service anyway?

From a technical perspective, this is best answered by the official Technical Guidance document for INSPIRE Download Services. The current version is 3.0 (published 12/06/2012):

ds30

To summarize, this document defines three different classes of INSPIRE Download Services:

  • Pre-defined Dataset Download Service based on ATOM
  • Pre-defined Dataset Download Service based on WFS
  • Direct Access Download Service

Basically, a Pre-defined Dataset Download Service based on ATOM provides service and dataset feeds that ultimately contain download links to the actual datasets. Datasets can only be downloaded in full. There are no filtering capabilities to query subsets.

A Pre-defined Dataset Download Service based on WFS is an OGC Web Feature Service 2.0 that complies to “Simple WFS”, “Query” and “HTTP GET” conformance classes. There’s just a single, mandatory stored query for retrieving full datasets using their identifier. Again, there are no filtering capabilities to query subsets.

A Direct Access Download Service extends a Pre-defined Dataset Download Service based on WFS by additional OGC WFS conformance classes: “Basic WFS”, “Ad hoc Query”, “Resource Identification”, “Minimum Standard Filter”, “Minimum Spatial Filter”, “Minimum Temporal Filter” and “Minimum XPath”.

Although Pre-defined Dataset Download Services may seem much simpler to implement, a Direct Access Download Service has the benefit that is offers filtering possiblities to WFS clients: Not only can you download a full dataset, but also a subset (e.g. the geo objects inside a bounding box). The possibilities for filtering are based on the OGC Filter Encoding 2.0 specification.

deegree WFS (contained in deegree webservices 3.2) allows to set up a Direct Access Download Service without any license fees and similar (or less) implementation complexity. Sounds to good to be true? Read on!

Interoperable vs. non-interoperable Download Services

Besides Pre-defined and Direct Access, INSPIRE guidelines make a distinction between Non-interoperable and Interoperable Download Services:

  • Non-interoperable (initial stages): Offered datasets are not required to be compliant
  • Interoperable: Offered datasets have to be compliant

Basically, the Non-interoperable phase is meant to reduce the efforts for data providers to meet the INSPIRE requirements. They can just offer their data in existing formats ‘as-is’. However, in the long run, all data offered by INSPIRE Download Services has to be compliant to the corresponding Data Theme Specification. This means that the data provider needs to transform their existing datasets into harmonized datasets that are valid with respect to the corresponding Data Theme Specification.

The following diagram (taken from the Technical Guidance) provides an overview of the milestones on the road to Interoperable Download Services (click for full size):

ds_roadmap

deegree WFS is suitable both for setting up both Non-interoperable and Interoperable Download Services. It can deal with the INSPIRE Data Model and return valid, harmonized INSPIRE GML datasets. Additionally, deegree’s GML mapping language can be used to perform data harmonization on-the-fly (but you can also perform data harmonization using any other technology).

What’s next?

The next part of this blog post series will discuss what harmonized datasets are exactly and why the deegree WFS implementation stands out for setting up Interoperable Direct Access Download Services.

Posted in Uncategorized | 1 Comment

deegree 3.2.0 webservices released

The deegree team is happy to announce the immediate availability of deegree webservices 3.2.0, available from:

http://www.deegree.org/Download

One of the major improvements to this release is a completely new written documentation, available at:

http://www.deegree.org/Documentation

The deegree 3.2.0 release (dubbed “INSPIRE release”) contains thirteen months worth of work. Major improvements:

  • New documentation (based on Sphinx)
  • New OGC service implementation: deegree WMTS (implements WMTS 1.0.0)
  • TileStore subsystem: A lean and fast data abstraction layer for map tiles
  • TileStore backends: GeoTIFF, File system, Remote WMS, Remote WMTS, EHCache
  • New concept for layer and theme configuration
  • WFS: Extended support for WFS 2.0 specification (e.g. Transactions)
  • WFS: Better compliance with INSPIRE Download Service TG 3.0
  • WMS: Better compliance with INSPIRE View Service TG 3.1
  • SQLDialect: Support for Microsoft SQL Server

Grab your copy of deegree webservices 3.2.0, try it out, and please provide feedback on the mailing list.

Enjoy!

Posted in Uncategorized | 3 Comments

deegree 3.1 webservices released

Some highlights:

  • WFS 2.0 support
  • Mapping of complex GML application schemas to relational models (PostGIS/Oracle)
  • Support for GeoTIFF raster pyramids as coverage stores
  • Support for cascading WMS
  • Better legend support in WMS
  • Improved relational model for CSW/ISO metadata store, added support for Oracle
  • Support for streaming inlined complex parameters in WPS
  • Metadata integration concept for INSPIRE-compliant service capabilities
  • Single download that includes all major OGC services (WMS, WFS, WPS, CSW)
  • Download example configurations (“workspaces”) from web administration interface

To get started, download a deegree webservices package and activate one of the official demo setups (“workspaces”) in the service console. Afterwards, you may want to check out the documentation available in the deegree wiki for learning about deegree’s configuration concepts.

Unfortunately, the wiki currently doesn’t describe some of the most interesting features of this release (e.g. the relational mapping configuration for the SQL feature store). Keep an eye on my blog, I plan to post some articles on these subjects…

Posted in Uncategorized | Leave a comment

JDBC tweaks to enable streaming result sets

In order to be able to serve large-scale deployments, deegree 3 has a full streaming architecture. This allows deegree 3 WFS to handle GetFeature requests that return Gigabytes or Terabytes of GML – without hogging the server. For example on a deegree 3-based INSPIRE Download Service, it’s no problem request all Cadastral Parcels of a member state. Another aspect is the streaming renderer used in deegree 3 WMS which can draw millions of features without running into memory problems.

Technically, deegree 3 uses iterators to realize this. If a query is performed on a feature store that is backed by an SQL database, the JDBC ResultSet is encapsulated in a FeatureInputStream. The Feature instances are only produced on demand (whenever the next instance is accessed).

Unfortunately, it’s the pecularities of the database vendors and their JDBC drivers that can put a spoke in one’s wheel here. Obviously, one should call Statement#setFetchSize(int) in order to give the DB/JDBC driver a hint on the number of result set rows to fetch in a single batch. However, it turns out that this may not be enough to prevent the collecting of all result rows in memory or the driver waiting until all results have been collected by the DB.

PostgreSQL

For PostgreSQL, it’s additionally required to call setAutoCommit(false) on the Connection object. The reason is that PostgreSQL only enables cursors inside a transaction. See here or here for more details.

Microsoft SQL Server

On SQL Server, the trick is to add the option selectMode=cursor to the JDBC URL, e.g. jdbc:microsoft:sqlserver://myHost:1433;selectMethod=cursor;databaseName=myDB. Some info can be found here. This may also be interesting in this regard.

Oracle

Didn’t really investigate this on Oracle yet. Does anybody have to share some insights?

Posted in Uncategorized | Leave a comment