News > hale studio
News' image preview

Last week, more than 800 people met in Strasbourg for an event packed with workshops, keynotes and presentations. This somewhat personal retrospective summarizes our impression in broad strokes. First of all, it was a very intense week for our team - with more than 50 meetings and 10 contributions to the programme. For almost all of these, videos are now available online.

Last week, more than 800 people met in Strasbourg for an event packed with workshops, keynotes and presentations. This somewhat personal retrospective summarizes our impression in broad strokes. First of all, it was a very intense week for our team - with more than 50 meetings and 10 contributions to the programme. For almost all of these, videos are now available online.

Participants Keynote session on Wednesday

Strategy

Most of the INSPIRE community is aware of the discussions surrounding the Fitness for Purpose of INSPIRE, and the related efforts to improve the usefulness of INSPIRE network service and data specifications. There is also an ongoing debate to define a list of Priority data themes and data sets to indicate which steps implementers should focus on. Some stakeholders are very critical of the current state of INSPIRE and point to technical difficulties in implementation as well as to limited usefulness for many use cases.

This criticism is somewhat in contrast to the many organisations moving forward on their INSPIRE implementation. There was a substantial number of presentations and workshops about projects that showed how to successfully implement interoperable services. As an example, Christine Najar from Swisstopo presented their feasibility study, which looked at the concrete efforts required to provide both INSPIRE and ELF/ELS data and services, and came to the conclusion that overall efforts are lower than many people anticipated.

At this point, there is a lot of evidence that some parts of the INSPIRE requirements need to be modified or relaxed, to make implementation easier and more robust. One example is the actual data discovery process, which we analysed in the context of the INScope project. In that project, which we presented together with Wageningen University & Research and the European Environmental Agency, we showed that only a few percent of data sets actually meet all requirements according to their metadata. Another example is the simplification of the encoding, for which several suggestions have been made, e.g. by Denmark and by Germany.

Some recommendations from the study commissioned by BKG

It is also important to focus on the usefulness and usability of INSPIRE data. To increase the usefulness, several national initiatives such as the Spatial Planning Act in the Netherlands build on top of the INSPIRE legislation. INSPIRE extensions are one way to piggyback local use cases onto the INSPIRE infrastructure.

Technology

In the technology and tool oriented sessions, the single keyword that was used most was probably Docker. Docker is a container technology that makes the deployment and maintenance of server based applications much easier than virtual machines did. Docker is a core building block of an entire ecosystem with tools such as docker compose, docker swarm and Rancher that allow us to build scalable, robust applications that can be managed much more effectively than previous generations. The paradigm shift is to move away from individual servers that are manually administered (“Pets”) to fully automated cluster deployments (“Cattle”). No more manual patching of individual Application Servers or Oracle Databases! By now, we have Docker images for all relevant Open Source and Closed Source applications available, be it ArcGIS Server, FME Server, GeoServer, deegree or our own hale connect platform.

Closely related to this topic was the second trendy keyword – the Cloud is coming! INSPIRE mandates a relatively high level of availability and performance for all INSPIRE services, with requirements such as 99% availability and 20 WMS requests per second for a 640x480 raster image. For smaller organisations who do not have dedicated staff and hardware, these objectives can be hard to fulfill, so cloud architectures offer a practical, efficient solution.

Joeri Robbrecht from DG ENV gives an introduction to the INSPIRE requirements and their impact

One key consideration that popped up several times was the question of whether cloud services are secure enough. Several presenters including Ken Bragg (Safe Software) explained that “AWS is probably more secure than your data center” in one variant or another. AWS is by now offering basically any certification one could ask for, and is very transparent about security issues. Especially in the context of INSPIRE data, which is intended for sharing and publishing, there are very few reasons left not to use cloud services – be it Software as a Service Solutions (such as haleconnect.com) or Platform as a Service resources. For those organisations with additional requirements about who should access data, there are also solutions in place or being developed, e.g. by the CLARUS project, which develops a Cloud Encryption Gateway and a Cloud Access Security Broker.

Linked Data is mostly a topic of research projects and prototypes. The main promise of Linked Data is to better integrate with “mainstream” IT technology by making resources such as individual spatial objects discoverable through search engines and by embedding fragments of linked data in normal web content. Just changing the encoding from GML to RDF or JSON-LD for all INSPIRE data however is certainly no silver bullet.

A Personal View by Anida

Anida and Andreas happy to give a thumbs-up even on the last day of the conference :)

This year, I attended the INSPIRE conference for the first time, so I am not going to compare it to previous conferences. I would rather like to focus on the key points and takeaways from an INSPIRE Newbie perspective.

The conference brought together many INSPIRE, GIS, Data and Technology experts, as well as lots of people looking for opportunities to learn something new, and to exchange experiences. The conference was also a meeting point for people looking for new career opportunities, and that made me wonder. Was this market not too small to come to a conference looking for new career opportunities? Then I realized that it is not about the market size, it is about the impact of what was going on with INSPIRE and beyond.

I heard a lot about the approach to open data and making it available for citizens and businesses and listened to discussions about how far public administrations should open their data. In my personal view, on its own, INSPIRE will not bring high-end innovations, but combined with Open Data principles, they become feasible. Attending the SMESpire workshop as a representative of a start-up made me think more about the innovations that can be created by implementing INSPIRE.

Can we bring innovation, open data and fulfillment of legal obligations together? In my opinion, we can, but it is very important to understand why are we implementing INSPIRE. As I see it, INSPIRE should not be the ultimate goal: to implement something just for the sake of implementation. It should be a tool to help countries maintain, manage and exchange big amounts of data effectively, to foster international collaboration. That will then lead to innovations created by businesses. Businesses will find a way to create the added value that will then lead to growth. What does it take? Collaboration and communication, and then a bit more of it. It also takes some kind of a joint platform, that will enable SMEs to take part in different projects and address the needs and priorities of INSPIRE implementers.

So it was a week full of learnings and a really great opportunity for exchange and networking, but moreover, it was an opportunity for so many people to find that one solution, implementation or the expert that will bring them further and closer to their goals.

The way forward

We’ve very much enjoyed supporting this year’s conference through our Gold Partnership, and would like to thank the organisers in Germany, France and at the JRC for the great conference.

The INSPIRE GIS partners at our joint booth, together with some JRC staff

There is not much of a break now, though – the next INSPIRE Roadmap milestone is approaching fast: On November 23rd, provision of existing data sets tied to Annex I in INSPIRE interoperable form is required. Many organisations we work with aim to fulfil their obligations in time. Looking beyond this milestone, focus will shift towards annex II and III – a good moment to take a break and evaluate both the major strategic directions and new technology.

We’re looking forward to the 2018 edition in Antwerp! You can bet that we will accept the challenge of the Hunt for the Golden Pineapple!

(more)

It’s almost a tradition for us to release a new hale studio version for the INSPIRE conference, and 2017 is no exception! This year, we’ve created a maintenance release of hale studio. This version fixes several bugs and brings these enhancements:

  • Save transformation projects directly to hale connect
  • Partition GML output into multiple files
  • Support for PostgreSQL Materialized Views
  • Support for Double columns for the CSV reader
  • Improved handling for HTTP/HTTPS codelists and schemas

We’ve also fixed several bugs and added smaller improvements. The whole list is available in the changelog.

Get the latest version, and let us know what you think of it!


It’s almost a tradition for us to release a new hale studio version for the INSPIRE conference, and 2017 is no exception! This year, we’ve created a maintenance release of hale studio. This version fixes several bugs and brings these enhancements:

  • Save transformation projects directly to hale connect
  • Partition GML output into multiple files
  • Support for PostgreSQL Materialized Views
  • Support for Double columns for the CSV reader
  • Improved handling for HTTP/HTTPS codelists and schemas

We’ve also fixed several bugs and added smaller improvements. The whole list is available in the changelog.

Get the latest version, and let us know what you think of it!


Save transformation projects directly to hale connect

hale studio extends its integration with the online collaboration platform by allowing to save projects directly to hale connect. This allows for a true online workflow where your changes to a project that was saved to or loaded from hale connect will be saved directly to the remote project without the need of another export. You can switch back to an offline workflow at any time by saving your project to a local file on your computer.

If you have a private cloud or on premise installation of hale connect or inspire gis, you can also log in to that by changing the application settings.


Partition GML output into multiple files

When exporting data as GML, the exported data can now be partitioned into multiple output files. The number of features per file can be configured during the export. This partitioning approach keeps features that reference each other together in one file, so it is not effective for some networks and hierarchies.


Support for PostgreSQL Materialized Views

The PostgreSQL schema import now supports loading materialized views alongside tables and regular views.

Thanks to the Bundesanstalt für Wasserbau for funding this work.


(more)

GDI-Südhessen is a cooperative at county level, including more than 100 municipalities in the southern part of the German state of Hesse. The goal of this group is to improve internal cooperation and data sharing and to collaboratively tackle larger challenges, such as the implementation of the INSPIRE initiative.

GDI-Südhessen was also one of our first customers. Together with GDI-Südhessen, we’ve built the “GDI InspireUmsetzer” open platform to achieve both goals: Exchange data across all member organisations using shared data models, and fulfill INSPIRE obligations. Especially the latter typically requires significant capacity building, investment in software development and in changes to internal data production processes. Because of the large amount of expert knowledge that is required, there is significant resistance to implementing the INSPIRE directive.

The platform we have built in 2016 and operated since enables data providers without any knowledge of INSPIRE standards to upload data in a very easy process, and the system then transforms the data and provides services in a fully automated workflow. The process also includes publishing in open data catalogues and data and metadata validation. Our intent from the start was to build GDI InspireUmsetzer so that the same solution could be offered to all public authorities in Europe that also need to comply with the INSPIRE directive. In particular, wetransform offers the platform to organisations like municipalities or surveying departments that don’t have their own IT departments and limited expertise in spatial data infrastructures. This INSPIRE implementation platform is available in different deployment modes (public cloud, private cloud, on premise) and is built on trusted open source software such as hale studio and deegree.

A Preview of the whitepaper

To learn more about the GDI-InspireUmsetzer project, its organisation and processes, download our in-depth case study.

Attending the INSPIRE Conference 2017? Andreas von Dömming and Martin Domeyer will present the platform on Friday 8th of September, 09:45 in Strasbourg!

GDI-Südhessen is a cooperative at county level, including more than 100 municipalities in the southern part of the German state of Hesse. The goal of this group is to improve internal cooperation and data sharing and to collaboratively tackle larger challenges, such as the implementation of the INSPIRE initiative.

GDI-Südhessen was also one of our first customers. Together with GDI-Südhessen, we’ve built the “GDI InspireUmsetzer” open platform to achieve both goals: Exchange data across all member organisations using shared data models, and fulfill INSPIRE obligations. Especially the latter typically requires significant capacity building, investment in software development and in changes to internal data production processes. Because of the large amount of expert knowledge that is required, there is significant resistance to implementing the INSPIRE directive.

The platform we have built in 2016 and operated since enables data providers without any knowledge of INSPIRE standards to upload data in a very easy process, and the system then transforms the data and provides services in a fully automated workflow. The process also includes publishing in open data catalogues and data and metadata validation. Our intent from the start was to build GDI InspireUmsetzer so that the same solution could be offered to all public authorities in Europe that also need to comply with the INSPIRE directive. In particular, wetransform offers the platform to organisations like municipalities or surveying departments that don’t have their own IT departments and limited expertise in spatial data infrastructures. This INSPIRE implementation platform is available in different deployment modes (public cloud, private cloud, on premise) and is built on trusted open source software such as hale studio and deegree.

A Preview of the whitepaper

To learn more about the GDI-InspireUmsetzer project, its organisation and processes, download our in-depth case study.

Attending the INSPIRE Conference 2017? Andreas von Dömming and Martin Domeyer will present the platform on Friday 8th of September, 09:45 in Strasbourg!

(more)

News' image preview

Over the course of the last 15 years, Germany developed its own set of GML-based spatial data exchange standards, known as ALKIS-AFIS-ATKIS (or short 3A NAS). Surveying organisations in all states have implemented the standards, thus providing a common foundation for an INSPIRE implementation.

In 2016, the Arbeitsgemeinschaft der Vermessungsverwaltungen der Länder der Bundesrepublik Deutschland (AdV) commissioned wetransform to create a formal data transformation documentation, with 3A NAS and the “Hauskoordinaten” as a source and with 12 INSPIRE Annex I GML schemas as the target. This documentation was to be generated based on hale studio alignments, and validated against data sets from multiple German states.

This project has recently been completed, resulting with the first full, formal and executable data transformation specification. Those results helped German authorities to achieve the breakthrough in the provision of harmonised INSPIRE data sets.

Additional project challenge was the high complexity of the source data models, being much larger than the individual INSPIRE annex data specifications. Furthermore, the 3A NAS schemas use a lot of special constructs to link features, and the German states have implemented individual variants due to different processes and legal requirements.

In this article, we explain how we created these highly complex alignments with up to 450 cells using hale studio, what methodologies we applied, and how implementers, e.g. the state of Rheinland-Pfalz have already picked up the results to create INSPIRE compliant data sets from their 3A NAS production databases.

The baseline for the project was a massive collection of Excel matching tables, equivalent to more than 200 A3 pages when printed out. We used these Excel tables to create the initial Alignments. Furthermore, we worked with the AdV to define common rules for the transformation and for the resulting INSPIRE data sets, such as patterns for gml:id and gml:identifier elements.

Over the course of the last 15 years, Germany developed its own set of GML-based spatial data exchange standards, known as ALKIS-AFIS-ATKIS (or short 3A NAS). Surveying organisations in all states have implemented the standards, thus providing a common foundation for an INSPIRE implementation.

In 2016, the Arbeitsgemeinschaft der Vermessungsverwaltungen der Länder der Bundesrepublik Deutschland (AdV) commissioned wetransform to create a formal data transformation documentation, with 3A NAS and the “Hauskoordinaten” as a source and with 12 INSPIRE Annex I GML schemas as the target. This documentation was to be generated based on hale studio alignments, and validated against data sets from multiple German states.

This project has recently been completed, resulting with the first full, formal and executable data transformation specification. Those results helped German authorities to achieve the breakthrough in the provision of harmonised INSPIRE data sets.

Additional project challenge was the high complexity of the source data models, being much larger than the individual INSPIRE annex data specifications. Furthermore, the 3A NAS schemas use a lot of special constructs to link features, and the German states have implemented individual variants due to different processes and legal requirements.

In this article, we explain how we created these highly complex alignments with up to 450 cells using hale studio, what methodologies we applied, and how implementers, e.g. the state of Rheinland-Pfalz have already picked up the results to create INSPIRE compliant data sets from their 3A NAS production databases.

The baseline for the project was a massive collection of Excel matching tables, equivalent to more than 200 A3 pages when printed out. We used these Excel tables to create the initial Alignments. Furthermore, we worked with the AdV to define common rules for the transformation and for the resulting INSPIRE data sets, such as patterns for gml:id and gml:identifier elements.

ALKIS Data transformed to INSPIRE using hale studio

Base Alignments and Custom Functions

During the initial analysis of the data models, we saw the need for specific functions and common mappings for all the alignments. As both, the source and target models are rich object-oriented models with rich inheritance hierarchies, we can define the common mappings in one alignment and then import these into all others. These so-called base alignments are re-usable components that we then imported into all Annex I alignments:

  • base-functions: Common functions for all themes (extended also by the other base alignments)
  • base-tn: Common functions and mappings for rail transport, road transport, water transport, air transport, cable transport
  • au-basis: Common mappings for all variants of the Administrative Units Alignment

The custom functions we wrote for this project included the following:

  • Creation of Geographical Name objects
  • Specific simplification rules for geometries
  • Generation of local IDs and Identifiers according to the AdV identifier rules
  • Conversion of units of measurement

Using the custom functions, we avoided a lot of redundancy in the alignments and reduced their complexity.

The Annex I Alignments

The core task in the project was to create the 14 concrete alignments used to generate the formal documentation. We applied the following development process:

  1. Implement the alignment according to the provided mapping tables, collecting any ambiguous points and posting questions on the project issue tracker
  2. Provide the alignments, the generated documents and the transformed data for review to the AdV stakeholders
  3. Implement improvements and fixes as suggested

In this project, we learned that the highly detailed matching tables captured only about 30% of all transformation cells in the final projects correctly or fully. Most of the work was to review and improve iterations that followed on the initial implementation. A lot of very important input was provided by the AdV stakeholders, so that the alignments could be improved until they reached sufficient quality on all aspects. The following links lead you to the interactive mapping documentation for some of these:

  1. Hauskoordinaten to INSPIRE Addresses
  2. 3A to INSPIRE Addresses
  3. 3A Flurstücke to INSPIRE Cadastral Parcels
  4. 3A Flurstücke to INSPIRE Administrative Units
  5. 3A Gebiete to INSPIRE Administrative Units
  6. 3A kommunale Gebiete to INSPIRE Administrative Units
  7. 3A to INSPIRE Air Transport Network
  8. 3A to INSPIRE CableTransport Network
  9. 3A to INSPIRE Road Transport Network
  10. 3A to INSPIRE Railway Transport Network
  11. 3A to INSPIRE Water Transport Network
  12. 3A to Geographical Names
  13. 3A to Hydro-Physical Waters
  14. 3A to Hydrography Network

These alignments are currently in the final resolution process of the AdV.

Variants and Derived Alignments

You might have noticed that there are three alignments that have Administrative Units as their target schema: In 3A, the geometry of Administrative Units is derived by creating the union of a set of land parcels. This process reduces redundancy in the data, but can be computationally expensive. As a consequence, we developed an alignment that creates these aggregated geometries for all levels of Administrative Units, but also made two variants that allow the specification of an additional data source with the respective pre-aggregated geometries.

We set up a process to generate derived alignments for subsets of the 3A data models based on the “Modellart”. The “Modellart” is a mix of model and scale – for example, there are landscape models in scales of 1:25.000 to 1:1.000.000. Each “Modellart” includes a subset of the total 3A model, so that the transformation also need to be used on a subset only, and some information is not available. We used annotations to the mapping cells to indicate which cell is relevant for which model. Due to hale’s declarative mapping they can be created easily by excluding mappings for feature types that are not part of the respective model.

We also set up another automated generation process to derive modified alignments that would use the PostNAS database system instead of 3A XML as the source schema. One of the big advantages of a declarative system is that it makes such derivation processes and re-used of transformation mappings feasible.

Continuous Testing

For any kind of complex data processing, continuous testing is necessary. We set up an automated process that transformed and validated more than a dozen different data sets after each change to the mappings. This process was implemented with a Gradle script invoking the hale Command Line Interface. This interface has grown in capabilities with each release and can be used to control almost all aspects of hale – be it the transformation, the generation of artifacts such as the formal documentation or the validation of the results.

Documentation

The final deliverable of the project was the formal documentation. For a long time, hale studio had the capability to generate both matching tables and HTML documentation. Over the development of the last releases we have continuously improved the HTML documentation feature, so that the documentation offers a lot more than any static document could provide. It includes a graphical representation of the mapping, a verbal description, and information on the related schema entities, notes and other information. It is also interactive –search and filter options make it possible to choose what information to display.

Interactive HTML documentation generated from the Hydro Network Alignment

Collaboration

This project was a relatively complex undertaking, with more than 20 stakeholders reviewing the mappings and the transformed data to ensure completeness and correctness of the formal documentation. In the initial project, we used Gitlab as an issue tracker and collaboration platform. Gitlab is a very useful general purpose project and source code management platform, much like GitHub. However, we also found some issues with the usage of Gitlab for this specific use case:

  • Due to the size of the alignments, it was hard to establish context (e.g. which cell, which data) for any reported issues; reviewers used screenshots of the documentation
  • It was hard to keep track of changes made to the same mapping cells, so that repetitive and competing solutions were implemented
  • Standard diffs don’t work well to communicate changes and the history to a mapping cell to the domain experts involved in the projects

We thus implemented collaboration features as part of the documentation itself. These collaboration features enable efficient teamwork in larger groups with diverse backgrounds:

  • Tasks: Create and assign tasks to users of the platform if something needs to be changed in the transformation project.
  • Comments: Start a discussion visible to anybody with access to the transformation project in scope of a single mapping function or in scope of the entire alignment.
  • Notes: Add a private note for yourself to the alignment or any single cell.

These additional features require a central service to function, which we deployed as part of haleconnect.com. We evaluated the use of hale connect to manage our internal transformation projects over the last months. Now, we start to use the same processes with our customers to build better transformation projects faster.

(more)

News' image preview

For a long time, the data harmonisation panel and specifically the Redmine installation at www.esdi-community.org was the home to hale studio. More than 1.400 people registered there over the course of six years and posted questions on INSPIRE and other standards. For reasons such as that no one maintained the existing infrastructure anymore, we have moved most resources away from Redmine over the course of the last year:

  1. The hale studio source code and issue tracking is now hosted at GitHub
  2. The end user and developer documentation has moved to halestudio.org
  3. Since last week, the hale support board is now moved to discuss.wetransform.to

The old website will now be switched off. So, what do we have in stock for you on the new site?

For a long time, the data harmonisation panel and specifically the Redmine installation at www.esdi-community.org was the home to hale studio. More than 1.400 people registered there over the course of six years and posted questions on INSPIRE and other standards. For reasons such as that no one maintained the existing infrastructure anymore, we have moved most resources away from Redmine over the course of the last year:

  1. The hale studio source code and issue tracking is now hosted at GitHub
  2. The end user and developer documentation has moved to halestudio.org
  3. Since last week, the hale support board is now moved to discuss.wetransform.to

The old website will now be switched off. So, what do we have in stock for you on the new site?

The new forum for hale at wetransform.to

The new forum pretty much looks like other discussion sites. It offers threaded discussion on one single board, where you can post anything related to hale studio, hale connect, inspire git and data harmonisation in general. What’s special about it is that it is fully integrated with hale connect, our cloud platform for collaborative data modelling and transformation. When you register at the forum, you automatically get a free account to hale connect as well.

Step by step, we will integrate functions that are only available inside hale connect, such as discussions on individual cells of hale studio projects, with the public forum. We look forward to see more focused and effective teamwork on issues such as challenging transformation projects!

Happy transforming!

(more)