INSPIRE Thematic Clusters

Download Service WFS: StoredQuery and ResponsePaging for large datasets?

3718 Views

Download Service WFS: StoredQuery and ResponsePaging for large datasets?

Started by Thijs BRENTJENS Replies (2)

Hello all,

In the Netherlands we have several large datasets served by WFS to the INSPIRE network. One of the TG requirements for WFS is to provide a StoredQuery to download the entire dataset. For large datasets this is a practical challenge: users could (accidentally or intentionally) download an entire dataset, maybe GBs, with a single request to a WFS, which could freeze applications. For the service provider this may cause problems too on their infrastructure.

For datasets with a high update frequency it is not an option to provide pre-processed / cached datasets and/or use ATOM feeds as Download service. So serving them with WFS directly is sometimes the only (good) option. One approach is to add a (low) serverside limit to the maximum number of features to return. E.g. 1000 features at once. A user then could use WFS Response Paging to fetch the other features, if needed. This works on the INSPIRE StoredQuery too, we've tested this, also with WFSes offering more than 1 featuretype.

However Response Paging is not mentioned in the Technical Guidance for Download Services. Clients have more and more support for Response Paging.

What do you think of this approach? Is this approach in line with Download Services?

And is this an issue other organizations have too? If so, what solution(s) are you using?

Replies

    • Public

    By Jari REINI

    Hi Thijs,

    I agree, response paging is currently the only good option. We are also using paging and my interpretation is that it is ok.

    Nvertheless, we must bring up this question in the next MIG-T meeting.

    Regards,
    Jari

    • Public

    By Thijs BRENTJENS

    Hi Jari,

    Thanks, good to know you are doing this too and indeed, it should be discussed in the next MIG-T. I'll ask if a colleague will attend and bring this up. Or will you attend?

     

    Regards,

    Thijs

Topo & Cadastre, Reference Data

Topo & Cadastre, Reference Data

Topo & Cadastre, Reference Data: If themes like Hydrography, Geographical Names, Administrative Units, Cadastral Parcels, Addresses, Buildings, Transport Networks matters to you, join this group!

Featured groups

Elevation, Orthoimagery, Reference Systems and Geographical Grids Cluster

Elevation, Orthoimagery, Reference Systems and Geographical Grids Cluster

INSPIRE Thematic Cluster Elevation, Orthoimagery, Reference systems, Geographical grids - Join this group to share your knowkledge, learn and collaborate in solving issues related to the Elevation, Orthoimagery, Reference systems and Geographical grids themes