News > groovy

It’s Thursday, or as we at wetransform (thanks to the influence of a certain Venezuelan on the team) call it, Juernes!

In this fourth installment of the Groovy Scripting week, we will tackle another non-trivial case: Building a complete network with nodes and links when your only input data was a bunch of spaghetti lines. This actually requires three scripts. It also shows in detail how to use priorities and other advanced features.

Like the previous posts, note that this article assumes you have working knowledge of hale studio and know the terminology.

In many projects, we did not receive a fully consistent set of Nodes and Links from customers to create a complete Network, or there were no Nodes available at all.

These three snippets extract the first and last Coordinates from each Link (LineString/Curve) and then set up the Node objects, and then create bidirectional references.

Collect Nodes and connected Segment IDs

This is a simple Groovy Retype function that uses the links as a source. The target does not matter, since no target objects are emitted. In this example, the segments are called vaarwegvakken, and the IDs of these segments are stored in the field vwk_id. This function needs to be executed first and should thus have the highest execution priority.

This is the full script:

It’s Thursday, or as we at wetransform (thanks to the influence of a certain Venezuelan on the team) call it, Juernes!

In this fourth installment of the Groovy Scripting week, we will tackle another non-trivial case: Building a complete network with nodes and links when your only input data was a bunch of spaghetti lines. This actually requires three scripts. It also shows in detail how to use priorities and other advanced features.

Like the previous posts, note that this article assumes you have working knowledge of hale studio and know the terminology.

In many projects, we did not receive a fully consistent set of Nodes and Links from customers to create a complete Network, or there were no Nodes available at all.

These three snippets extract the first and last Coordinates from each Link (LineString/Curve) and then set up the Node objects, and then create bidirectional references.

Collect Nodes and connected Segment IDs

This is a simple Groovy Retype function that uses the links as a source. The target does not matter, since no target objects are emitted. In this example, the segments are called vaarwegvakken, and the IDs of these segments are stored in the field vwk_id. This function needs to be executed first and should thus have the highest execution priority.

This is the full script:



You can download this script here.

This snippet is used in a Groovy Create function that has the Node type as a target. This function has to be executed second and needs to have the higher execution priority.



This script also shows how to create a geometry from a coordinate, using a specific coordinate Reference system.

You can download the script here.

In the final step, we add the references from the Links to the nodes created before, so that we have a fully navigable graph in the transport network. This script is inside a normal Groovy Function that has the domain ID of source as the source property and either the startNode or endNode property of the Link as the target property. The surrounding type transformation needs to have a normal execution priority so that it runs after the two other functions.



You can download this script here.

Happy transforming!

(more)

This is the third time to get “Groovy” this week! Today, we will look into how we can simplify Multi-Geometries to simple geometries using a specific script. This is often necessary when the target data model only allows simple geometries, but we can actually have composite and multi-geometries (MultiLineString, MultiPolygon) in the source data.

Like the previous posts, note that this article assumes you have working knowledge of hale studio and know the terminology.

Wednesday’s Script: Create LineStrings from MultiLineStrings (Johanna)

This Groovy Script was developed to aggregate MultiLineString geometries and convert them to LineStrings.

Background: This script was developed to process the 2018 HY-P Swisstopo data. A merge on the feature type level necessitated the use of aggregation to obtain one LineString per feature. The script below was used in a Greedy Groovy script function on the geometry property.

This script also contains a lot of great logging examples!

This is the full script:

This is the third time to get “Groovy” this week! Today, we will look into how we can simplify Multi-Geometries to simple geometries using a specific script. This is often necessary when the target data model only allows simple geometries, but we can actually have composite and multi-geometries (MultiLineString, MultiPolygon) in the source data.

Like the previous posts, note that this article assumes you have working knowledge of hale studio and know the terminology.

Wednesday’s Script: Create LineStrings from MultiLineStrings (Johanna)

This Groovy Script was developed to aggregate MultiLineString geometries and convert them to LineStrings.

Background: This script was developed to process the 2018 HY-P Swisstopo data. A merge on the feature type level necessitated the use of aggregation to obtain one LineString per feature. The script below was used in a Greedy Groovy script function on the geometry property.

This script also contains a lot of great logging examples!

This is the full script:



You can download this script here and import it in hale studio as a Groovy snippet by going to File -> Import -> Groovy Snippet. It assumes a certain geometry attribute name (the_geom) that you might have to change.

Happy transforming!

(more)

This is the second installment of our “Groovy week”. This time, we are going to look into a powerful feature in hale studio that is used by the Spatial Join function, but can also be accessed through scripting - the spatial index.

Like the previous post, note that this article assumes you have working knowledge of hale studio and know the terminology.

Tuesday’s Script: Build Polygons from Lines (Florian)

We used this script to build up references from boundaries to four levels of administrative units. In the original data, such references were not explicit, with the exception of international borders.

The script first identifies possible candidate AdministrativeUnit polygon geometries through the index and then verifies that the boundary object indeed matches the boundary of the polygon. It then constructs references differently, depending on the level of the identified AdministrativeUnit. As a result of this script, an AdministrativeBoundary target object can have references to up to six AdministrativeUnits - if it happens to be a boundary that separates 5th level, 4th level and 2nd level on both sides.

The execution context is in a Groovy Retype function.

This is the full script:

This is the second installment of our “Groovy week”. This time, we are going to look into a powerful feature in hale studio that is used by the Spatial Join function, but can also be accessed through scripting - the spatial index.

Like the previous post, note that this article assumes you have working knowledge of hale studio and know the terminology.

Tuesday’s Script: Build Polygons from Lines (Florian)

We used this script to build up references from boundaries to four levels of administrative units. In the original data, such references were not explicit, with the exception of international borders.

The script first identifies possible candidate AdministrativeUnit polygon geometries through the index and then verifies that the boundary object indeed matches the boundary of the polygon. It then constructs references differently, depending on the level of the identified AdministrativeUnit. As a result of this script, an AdministrativeBoundary target object can have references to up to six AdministrativeUnits - if it happens to be a boundary that separates 5th level, 4th level and 2nd level on both sides.

The execution context is in a Groovy Retype function.

This is the full script:



You can download this script here and import it in hale studio as a Groovy snippet by going to File -> Import -> Groovy Snippet. Please note that it uses some protected functions, so you need to “Lift Groovy Restrictions” to execute the script. It assumes a certain geometry attribute name (the_geom) that you might have to change. The additional attributes accessed here, such as BFS_NUMMER, BEZIRKSNUM, KANTONSNUM and ICC, will not be available in your source data.

We have thus included a simpler call to the spatial index as a second example:



Happy transforming!

(more)

News' image preview

When you try to do something in hale studio that isn’t possible with the out-of-the-box functionality, hale offers several ways of creating that functionality yourself. The most accessible of these is to add your own transformation functions using Groovy scripting. There are several points where Groovy functions can be added:

  • Type Transformation Functions
  • Property Transformation Functions
  • Custom Functions
  • Post-processing scripts (only through the CLI)

Groovy is superset of Java, i.e. any valid Java program is also a valid Groovy program. What it makes much easier than Java is the creation of data structures – no boilerplate constructors and initialisation and the like. Here are some key resources for learning Groovy:

Often, you will be looking for some recipes and snippets to get started, and that is exactly what we will provide during this week: We will publish one Groovy Snippet per day. Each of my colleagues has selected their favourite script and will share it with all of you 😊.

Please note that this article assumes you have working knowledge of hale studio and know the terminology.

When you try to do something in hale studio that isn’t possible with the out-of-the-box functionality, hale offers several ways of creating that functionality yourself. The most accessible of these is to add your own transformation functions using Groovy scripting. There are several points where Groovy functions can be added:

  • Type Transformation Functions
  • Property Transformation Functions
  • Custom Functions
  • Post-processing scripts (only through the CLI)

Groovy is superset of Java, i.e. any valid Java program is also a valid Groovy program. What it makes much easier than Java is the creation of data structures – no boilerplate constructors and initialisation and the like. Here are some key resources for learning Groovy:

Often, you will be looking for some recipes and snippets to get started, and that is exactly what we will provide during this week: We will publish one Groovy Snippet per day. Each of my colleagues has selected their favourite script and will share it with all of you 😊.

Please note that this article assumes you have working knowledge of hale studio and know the terminology.

The Groovy Logo, by Zorak1103, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=13358930

Monday’s Script: Build Polygons from Lines (Thorsten)

This script is my personal favorite. While relatively complex, it shows a lot of useful approaches how to work with geometries in a Groovy script.

This function was built for creating waterbody Polygons from LineString shore segments. It is executed in the context of a Merge type cell as a Greedy Groovy Script function. It takes a MultiLineString geometry with unsorted individual LineStrings and builds one to many polygons in a MultiPolygon from that input.

This is the script:



You can download this script here and import it in hale studio as a Groovy snippet by going to File -> Import -> Groovy Snippet. Please note that it uses some protected functions, so you need to “Lift Groovy Restrictions” to execute the script. It furthermore assumes a certain geometry attribute name (the_geom) that you might have to change.

Happy transforming!

(more)

News' image preview

Collectors are a powerful feature that we introduced in hale studio 3.1.0 and have since then expanded on. So, for what use cases should you be looking at collectors?

  • Your target schema has a collection object, such as a Network, that needs to reference many or all objects of a different type, e.g. NetworkLinks
  • You are building up a hierarchy of objects such as AdministrativeUnits, with their upperLevelUnit and lowerLevelUnit references
  • You need to use Merge or Join operations to determine relationships between objects, but these are computationally too expensive

With a Collector, you can collect values in one place in your transformation project and then use these values in another place in the transformation process. Let’s look at a recent project we’ve worked on to see how they work in practice.

Please note that this article assumes you have working knowledge of hale studio and know the terminology.

Collectors are a powerful feature that we introduced in hale studio 3.1.0 and have since then expanded on. So, for what use cases should you be looking at collectors?

  • Your target schema has a collection object, such as a Network, that needs to reference many or all objects of a different type, e.g. NetworkLinks
  • You are building up a hierarchy of objects such as AdministrativeUnits, with their upperLevelUnit and lowerLevelUnit references
  • You need to use Merge or Join operations to determine relationships between objects, but these are computationally too expensive

With a Collector, you can collect values in one place in your transformation project and then use these values in another place in the transformation process. Let’s look at a recent project we’ve worked on to see how they work in practice.

Please note that this article assumes you have working knowledge of hale studio and know the terminology.

A big collection of keys - which one to use?

To use a collector, there are two to three steps:

  1. Define the collector
  2. Define where to apply values from the collector
  3. Optionally, define Cell Execution priorities to make sure collected values are available when used

We always collect values in the context of another transformation function. As of hale studio 3.2.0, these are the transformation functions that support the definition of collectors:

  1. Groovy Scripts and Groovy Script (Greedy)
  2. Groovy Retype
  3. Groovy Merge
  4. Groovy Join
  5. Custom Functions

In all of these functions plus the following ones, you can apply the collected values:

  1. Groovy Create
  2. Assign Collected Values

With the upcoming 3.3.0 release, you’ll see more widespread support for the feature. Basically, most existing functions will allow collecting values, and we’ll add more functions to assign them.

The Use Case

This is the use case we are going to work on for this tutorial:

We need to create an INSPIRE Hydrographical Network dataset from UK Meridian 2 data encoded in a specific schema, using a GML 2.1 encoding. Each river segment from the source will be transformed to a WatercourseLink, and in addition, we’ll create a Network object that references all created WatercourseLink features.

The source and target schemas for this project
The source and target schemas for this project

You can take a look at the hale transformation project for this tutorial and download it, including source data, here at haleconnect.com.

Implementation

Step 1: Define the Collector in a Groovy Script

  1. As always, first define the Type-level transformation function. On the source schema, pick River, on the target side, select WatercourseLink. Click on the double arrow icon and select Retype. Use the default values for the function.
  2. Now, select fid on the source and id on the target feature type. Click on the double arrow icon and select Groovy Script. Leave the parameters on the first page as they are and click Next to proceed to the actual script editor. After you’ve entered the script, click Finish to let the transformation execute.
Use a Groovy Script function to create and collect IDs for the new features
Use a Groovy Script function to create and collect IDs for the new WatercourseLink features

This is the actual script to use:



Step 2: Use the collected values in an Assign collected values function

The easiest way to use values from a collector is to use the Assign collected values function. Follow these steps to use it:

  1. In the target schema, select the Network feature type. Click on the arrow icon and select Create. This function will create one or more objects of the target type from thin air. Create exactly one object.
  2. Next, click on the elements property in Network and then on the arrow icon. Choose Assign collected values and click Next. Enter the name of the collector we’ve defined in the script above (linkIDs) so that it can be accessed.

The Assign collected values function has some special behaviour to automatically identify and create local references. If you inspect the created network, you’ll see it now has 982 references that all look like this:

Assign collected IDs to any ReferenceType to get local references
Assign collected IDs to any ReferenceType to get local references features

Step 3: Transform in the right order

hale studio, in principle, automatically determines execution order of all cells. In some cases, this may not have the desired effect, so you need to provide hints to the transformation engine what should happen first, and what should happen last. For collectors, it’s important that the engine first completes collecting values before it tries to apply them in a different place. We do plan to recognize these cases automatically but for now, you’ll have to assign cell execution priorities to make sure everything always works as expected.

To ensure that the described steps are executed in the correct sequence, the execution priority has to be defined accordingly. The second mapping cell (Create on Network) has therefore to be set to a lower priority than the first mapping cell. This can be done via context menu in hale studio.

Edit the Cell Priority so that the function using the collector is executed last
Set the Cell Priority to lowest so that the function using the collector is executed last

Summary

With these three steps, you learned how to use the collector feature in hale studio. Let us know what you think of this feature and what we can do to improve its usability!

Happy transforming!

(more)