JDBC tweaks to enable streaming result sets

In order to be able to serve large-scale deployments, deegree 3 has a full streaming architecture. This allows deegree 3 WFS to handle GetFeature requests that return Gigabytes or Terabytes of GML – without hogging the server. For example on a deegree 3-based INSPIRE Download Service, it’s no problem request all Cadastral Parcels of a member state. Another aspect is the streaming renderer used in deegree 3 WMS which can draw millions of features without running into memory problems.

Technically, deegree 3 uses iterators to realize this. If a query is performed on a feature store that is backed by an SQL database, the JDBC ResultSet is encapsulated in a FeatureInputStream. The Feature instances are only produced on demand (whenever the next instance is accessed).

Unfortunately, it’s the pecularities of the database vendors and their JDBC drivers that can put a spoke in one’s wheel here. Obviously, one should call Statement#setFetchSize(int) in order to give the DB/JDBC driver a hint on the number of result set rows to fetch in a single batch. However, it turns out that this may not be enough to prevent the collecting of all result rows in memory or the driver waiting until all results have been collected by the DB.

PostgreSQL

For PostgreSQL, it’s additionally required to call setAutoCommit(false) on the Connection object. The reason is that PostgreSQL only enables cursors inside a transaction. See here or here for more details.

Microsoft SQL Server

On SQL Server, the trick is to add the option selectMode=cursor to the JDBC URL, e.g. jdbc:microsoft:sqlserver://myHost:1433;selectMethod=cursor;databaseName=myDB. Some info can be found here. This may also be interesting in this regard.

Oracle

Didn’t really investigate this on Oracle yet. Does anybody have to share some insights?

Posted in Uncategorized | Leave a comment

INSPIRE-compliant data/metadata coupling with deegree 3

INSPIRE sets a high value on metadata and coupling of data and metadata. For example, the GetCapabilities response of an INSPIRE-compliant View Service links every offered layer to the corresponding ISO/INSPIRE metadata record (which is usually served by an INSPIRE Discovery Service). An example snippet from a Layer element (nested in a WMS capabilities document) could look like this:

MetadataURL snippetIn order to support the generation of these MetadataURL elements, deegree WMS and WFS configurations now support the new options MetadataURLTemplate and MetadataSetId:

<deegreeWMS [...]>

  <MetadataURLTemplate>http://discovery.eu/csw?service=CSW\
    &request=GetRecordById&version=2.0.2&id=${metadataSetId}\
    &outputSchema=http://www.isotc211.org/2005/gmd\
    &elementSetName=full
  </MetadataURLTemplate>

  [...]
  <RequestableLayer>
    <Name>AU.AdministrativeUnit</Name>
    <Title>Administrative unit</Title>
    <Abstract>...</Abstract>
    [...]
    <MetadataSetId>550e8400-e29c-11d4-a716-446655440000
    </MetadataSetId>
    [...]
  </RequestableLayer>
  [...]

</deegreeWMS>

The global MetadataURLTemplate element provides a template for generating URLs of ISO metadata records. In the example, this is a KVP-encoded GetRecordById request template to a CSW instance. Every layer (for WFS: feature type) in the configuration can carry a MetadataSetId element to define the actual metadata record identifier (usually a UUID). This allows deegree WMS to generate the MetadataURL element in the GetCapabilities response.

Advanced options:

  • If one omits the MetadataURLTemplate element, deegree will assume that you want to generate URLs that point back to the deegree CSW instance on the same endpoint as the WMS/WFS (deegree offers an INSPIRE-compliant CSW as well). 
  • MetadataStoreId: If this global element is specified, deegree will access the internal metadata store with the specified id to retrieve the metadata record and magically augment the layer metadata reported in the capabilities. For instance, the Title and Abstract elements can be populated this way. Currently, this only makes sense if the same deegree instance hosts the CSW with the metadata records, but we’re working on enabling the access to any ISO/INSPIRE compliant CSW for retrieving the metadata.

For now, the new configuration options provide a first option to generate the MetadataURLs required by INSPIRE. In the near future, real integration between WMS/WFS and CSW should ease metadata configuration for GetCapabilities responses. If a CSW/INSPIRE Discovery Service is available to provide proper service and data metadata, every applicable piece of metadata (Title, Abstract, Keywords, …) should be automatically imported and reused for generating the GetCapabilities responses of deegree WMS and WFS.

Posted in Uncategorized | Leave a comment

Status of WFS 2.0 support in deegree 3

The last few days, I’ve been busy adding WFS 2.0 support to deegree 3 WFS. Current nightly builds already contain a lot of WFS 2.0-related functionality. Our first goal is to reach the following conformance classes from ISO 19142 (WFS 2.0) and ISO 19143 (Filter Encoding 2.0) in the next couple of days:

Simple WFS

In order to achieve this conformance class, a WFS implementation must provide GetCapabilities, DescribeFeatureType, ListStoredQueries, DescribeStoredQueries and GetFeature operation with the stored query GetFeatureById.  Besides providing valid output, such as service only has to support querying by feature identifier.

Basic WFS

This class adds the the  GetPropertyValue operation and the requirement for so-called ad hoc queries to the Simple WFS class.  Ad hoc queries are based on Filter Encoding expressions. A Basic WFS has to support the Minimum Spatial Filter conformance class from ISO 19143 (Filter Encoding 2.0), i.e. BBOX queries. However, deegree implements a lot more filter conformance classes that enable quite powerful retrieval options (see below).

KVP Encoding

The supported operations can be executed using KVP-encoded requests.

XML Encoding

The supported operations can be executed using XML-encoded requests.

Standard Filter

This class implies that all Filter Encoding 2.0 comparison operators are implemented: PropertyIsEqualTo, PropertyIsNotEqualTo, PropertyIsLessThan, PropertyIsGreaterThan, PropertyIsLessThanOrEqualTo, PropertyIsGreaterThanOrEqualTo, PropertyIsLike, PropertyIsNull, PropertyIsNil and PropertyIsBetween. The logical operators (And, Or and Not) have to be provided as well.

Spatial Filter

This class requires that the BBOX spatial operator and one or more of the other spatial operators is implemented. deegree implements all spatial operators defined in Filter Encoding 2.0: Equals, Disjoint, Touches, Within, Overlaps, Crosses, Intersects, Contains, DWithin, Beyond, BBOX.

Sorting

Implements sorting of the resources in a query response.

Minimum XPath

Server has to implement the minimum required set of XPath capabilities. Additionally, deegree implements support for arbitrary XPath 1.0 expressions — although complexer ones must be evaluated in memory as they cannot be mapped to the storage backend.

Posted in Uncategorized | 1 Comment

INSPIRE Download Services with deegree 3: Extended Capabilities

One additional requirement that the INSPIRE Download Service specification adds to WFS 2.0 is a specific ExtendedCapabilities section. deegree 3 WFS configuration now has initial support for that — there’s a new option for specifying ExtendedCapabilities sections which are included literally in the GetCapabilities response. Here’s an example (taken from deegree inspireNode configuration in trunk):

<deegreeWFS  [...]>

  <SupportedVersions>
    <Version>1.1.0</Version>
    <Version>2.0.0</Version>
  </SupportedVersions>

  <EnableTransactions>false</EnableTransactions>

  <QueryCRS>urn:ogc:def:crs:EPSG::4258</QueryCRS>
  <QueryCRS>urn:ogc:def:crs:EPSG::4326</QueryCRS>

  [...]

  <ExtendedCapabilities wfsVersions="1.1.0">
    <inspire_ds:ExtendedCapabilities [...]>
      <inspire_common:MetadataUrl>
        <inspire_common:URL>http://www.nationaal[...]</inspire_common:URL>
        <inspire_common:MediaType>[...]</inspire_common:MediaType>
      </inspire_common:MetadataUrl>
      <inspire_common:SupportedLanguages>
        <inspire_common:DefaultLanguage>
          <inspire_common:Language>dut</inspire_common:Language>
        </inspire_common:DefaultLanguage>
      </inspire_common:SupportedLanguages>
      <inspire_common:ResponseLanguage>
        <inspire_common:Language>dut</inspire_common:Language>
      </inspire_common:ResponseLanguage>
    </inspire_ds:ExtendedCapabilities>
  </ExtendedCapabilities>

  <ExtendedCapabilities wfsVersions="2.0.0">
    <ows:ExtendedCapabilities xmlns:ows="http://www.opengis.net/ows/1.1">
      <!-- This special treatment (container element) for WFS 2.0.0 is required to
           provide schema-valid GetCapabilities responses. 
           Can be removed once correct inspire_ds schemas for OWS commons
           1.1.0/WFS 2.0.0 are available. -->
      <inspire_ds:ExtendedCapabilities [...]>
        <inspire_common:MetadataUrl>
          <inspire_common:URL>http://www.nationaal[...]</inspire_common:URL>
          <inspire_common:MediaType>[...]</inspire_common:MediaType>
        </inspire_common:MetadataUrl>
        <inspire_common:SupportedLanguages>
          <inspire_common:DefaultLanguage>
            <inspire_common:Language>dut</inspire_common:Language>
          </inspire_common:DefaultLanguage>
        </inspire_common:SupportedLanguages>
        <inspire_common:ResponseLanguage>
          <inspire_common:Language>dut</inspire_common:Language>
        </inspire_common:ResponseLanguage>
      </inspire_ds:ExtendedCapabilities>
    </ows:ExtendedCapabilities>
  </ExtendedCapabilities>

</deegreeWFS>

If you wonder why the example config has two different ExtendedCapabilities sections for WFS 1.1.0 and 2.0.0: Currently, the schema for inspire_ds:ExtendedCapabilities relies on OWS 1.0.0, because it was still made for WFS 1.1.0. In order to achieve a schema-valid GetCapabilities response for WFS 2.0.0, I decided to put it below an ows:ExtendedCapabilities element for now (which allows any content). This should be no longer needed onced INSPIRE Download Service Specification 3.0 sees the light of the day…

BTW: deegree 3 WMS and CSW configurations have a similar option to add the mandatory ExtendedCapabilities section for INSPIRE.

Posted in Uncategorized | Leave a comment

Implementation notes on Filter Encoding 2.0: matchAction

Work is undergoing to implement the WFS 2.0 and INSPIRE Download Service specifications in deegree 3. In the last few days, I have been working on Filter Encoding 2.0 (which is a prerequisite for WFS 2.0 support). This gave me some insights and also revealed some curiosities, which I would like to document and share.

This post is about the matchAction attribute introduced in filter 2.0. It disambiguates the evaluation of operators that reference multi properties. Consider the following example feature (from the spec, section 7.7.3.3):

<ex:Building gml:id="b123">
  <gml:name>175 Fifth Ave.</gml:name>
  <gml:name>Flatiron</gml:name>
  <gml:name>Acme Building</gml:name>
  <!-- ... -->
</ex:Building>

Now consider the following filter 1.1.0 expression (namespace bindings omitted for brevity):

<Filter>
  <PropertyIsEqualTo>
    <PropertyName>gml:name</PropertyName>
    <Literal>Flatiron</Literal>
  </PropertyIsEqualTo>
</Filter>

Should this filter evaluate to true or false? Well, it depends… If you assume that the PropertyIsEqualTo operator requires all gml:name properties to be equal to Flatiron then it won’t match. If a single match is sufficient, it will. Up until filter 1.1.0, the correct interpretation doesn’t appear to be specified (in deegree 3 a single successful match is considered sufficient). Filter 2.0 adds the optional matchAction attribute to clarify this situation:

<Filter>
  <PropertyIsEqualTo matchAction="Any">
    <ValueReference>gml:name</ValueReference>
    <Literal>Flatiron</Literal>
  </PropertyIsEqualTo>
</Filter>

(if you wonder: ValueReference replaces PropertyName in 2.0)

Any means that this predicate will evaluate to true since there is at least one gml:name with value Flatiron. Besides Any, one can also specify All or One (exactly one value must match).

I really appreciate this addition to the standard, but I wonder why this attribute is only defined for comparison operators of type fes:BinaryComparisonOpType? I think it would make perfect sense for PropertyIsLike and PropertyIsBetween as well (in some case it might even apply to PropertyIsNil). And what about the spatial operators — in complex application schemas, features often allow for multiple spatial properties. I think the correct evaluation of Intersects referring to a multi geometry property should be clarified as well. Why is matchAction not supported here?

Also, I wonder how operators that reference two multi properties have to be evaluated? Let’s modify the example feature and filter:

<ex:Building gml:id="b123">
  <gml:name>175 Fifth Ave.</gml:name>
  <gml:name>Flatiron</gml:name>
  <gml:name>Acme Building</gml:name>
  <ex:name>Acme Building</ex:name>
  <ex:name>Building I</ex:name>
  <!-- ... -->
</ex:Building>

Filter:

<Filter>
  <PropertyIsEqualTo matchAction="Any">
    <ValueReference>gml:name</ValueReference>
    <ValueReference>ex:name</ValueReference>
  </PropertyIsEqualTo>
</Filter>

Should this filter match the feature? What is the proper semantics of Any, All or One in the multi/multi case? Comments are welcome…

Posted in Uncategorized | Leave a comment

More thoughts about modularizing deegree’s JSF components

As I wrote earlier, it’s actually pretty simple to redirect facelet resolving to JARs. As JSF already includes mechanisms for scanning for JSF beans in JARs, this could enable a very simple approach for modularizing deegree’s JSF codebase.

So far, so good. But wait! deegree’s workspace (configuration) concept allows for putting JARs into module folder to add functionality dynamically (without messing with the deegree webapp) to an existing deegree web service deployment. Currently, this is in use for WPS processes, additional feature store implementations, filter functions, etc. Wouldn’t it be nice to to be able to plug in JSF functionality dynamically by simply putting it into a deegree workspace?

Unfortunately, it seems that this is not possible at the moment 😦

The main problem is that loading code from deegree’s workspace involves a custom classloader. And an even harder problem: it’s vital that deegree workspaces can be started and stopped dynamically, so an administrator can make changes (or even exchange the full workspace) without restarting the deegree webapp in the servlet container. Some details:

  • It’s actually no big deal to retrieve facelets (*.xhtml files) using a custom classloader (such as deegree’s workspace classloader, that takes the JARs below folder modules into account).
  • It’s apparently impossible to add new JSF beans without restarting the whole webapp. The JSF 2.0 specification (section 11.4.2) is pretty clear about the fact, that scanning for JSF annotated classes only happens once on startup of the system. After investigating Mojarra’s (the reference implementation of Java Server Faces) source code a bit more, I found the AnnotationProvider extension point, but this would be vendor specific and not allow for exchanging JSF bean classes after initialization (webapp startup). While hacking around, I even realized that the JSF bean scanning obviously occurs on Servlet instantiation (not initialization), so it’s not possible to force deegree’s servlet to be initialized first using the load-on-startup parameter. This would at least have opened the door to adding JSF modules from the modules directory of the startup workspace…

To conclude: modularizing JSF components into JARs and putting them into WEB-INF/lib is one thing. Loading them using a custom classloader and allowing for code replace after webapp initialization is another — and probably not so easy.

I found some hints on the web that is is possible to OSGify JSF — so maybe moving to an OSGi container would be an option. Another thing which may be worthwhile to look at could be the so-called web fragments introduced by Servlet specification 3.0.

Posted in Uncategorized | Leave a comment

Thoughts about modularizing deegree’s JSF components

In order to conquer complexity in modern systems, modularization is key. deegree 3 uses Maven modules to break the codebase into manageable pieces. However, deegree’s service administration console is still a single Maven module that includes JSF beans and facelets for all administration tasks (defining JDBC connections, style definition, feature store creation wizard, service configuration, etc.). Only some core JSF functionality (including a few tags) is separated into module deegree-jsf-core.

It would be quite helpful, if also beans and facelets for specific functionality (say: the SQL feature store configuration wizard) could be separated into independent JARs. After searching the web a bit, I stumbled upon a very simple and easy solution for that. It basically involves two steps:

  • Ensure that the JAR is considered for scanning for JSF annotated classes (as it’s already done with deegree-jsf-core). JSF’s behavior for finding JSF beans is documented in section 11.4.2 of the JSF 2.0 specification. The most common and straightforward way is to place a file named META-INF/faces-config.xml or a file matching META-INF/*.faces-config.xml into the JAR.
  • Implement and register an extension of ResourceResolver which considers the classpath for retrieving facelets.

Writing a custom ResourceResolver is simple. The implementation below uses /META-INF/deegree/jsf as the common location for facelets inside the JAR.

package org.deegree.console;

import [...]

public class JARFaceletsResolver extends ResourceResolver {

    private static final String JSF_BASE_PATH = "/META-INF/deegree/jsf";

    private final ResourceResolver parent;

    public JARFaceletsResolver( ResourceResolver parent ) {
        this.parent = parent;
    }

    @Override
    public URL resolveUrl( String path ) {

        URL url = parent.resolveUrl( path );

        // not found in webapp -> resolve on classpath
        if ( url == null ) {
            url = JARFaceletsResolver.class.getResource(  JSF_BASE_PATH + path );
        }
        return url;
    }
}

The only thing left to do is tell JSF to use the JARFaceletsResolver for locating requested facelets. This can be done by adding the following to a (already JSF-enabled web.xml):

<web-app xmlns="http://java.sun.com/xml/ns/j2ee" [...]>
[...]
  <context-param>
      <param-name>javax.faces.FACELETS_RESOURCE_RESOLVER</param-name>
      <param-value>org.deegree.console.JARFaceletsResolver</param-value>
  </context-param>
[...]
</web-app>

That was pretty easy 🙂 This could be the option to effectively modularize deegree’s JSF codebase into JARs as well.

Posted in Uncategorized | 1 Comment