In our fourth installment of the 2020 Groovy Week, we are going to take a look at how to map attributes in Groovy type transformation functions and how to use that to create multiple target objects for a single source object. Like in the previous Groovy Week posts, note that this article assumes you have working knowledge of hale studio and know the terminology.

Thursday’s Script: Mapping Attributes on the Type Level to Split a Feature (Florian)

The normal approach to map values of source attributes to the target is to invoke an attribute transformation function like Rename, Classification or Groovy Script. These attribute-level functions let you create a target value based on the value(s) of the source attribute(s). Here is an example for a simple Rename that maps the source description attribute to the target attribute of the same name. The function is part of the Retype of the source type River to the target type Watercourse.

If we replace the type transformation function Retype by a Groovy Retype, we can achieve the same result as with the Rename function by using the _target instruction in the script:



A typical use case for this is if you need to split one feature into more than one target object with different target attribute values. Here is an example from an INSPIRE Hydrography alignment where one target instance is created for every source geometry:



You can download the script snippets here and here and import them in hale studio as a Groovy snippet by going to File -> Import -> Groovy Snippet. Please note that some scripts use protected functions, so you might need to “Lift Groovy Restrictions” to execute the script. Make sure you replace the placeholder attribute names with your own attribute names.

Happy transforming!

In our fourth installment of the 2020 Groovy Week, we are going to take a look at how to map attributes in Groovy type transformation functions and how to use that to create multiple target objects for a single source object. Like in the previous Groovy Week posts, note that this article assumes you have working knowledge of hale studio and know the terminology.

Thursday’s Script: Mapping Attributes on the Type Level to Split a Feature (Florian)

The normal approach to map values of source attributes to the target is to invoke an attribute transformation function like Rename, Classification or Groovy Script. These attribute-level functions let you create a target value based on the value(s) of the source attribute(s). Here is an example for a simple Rename that maps the source description attribute to the target attribute of the same name. The function is part of the Retype of the source type River to the target type Watercourse.

If we replace the type transformation function Retype by a Groovy Retype, we can achieve the same result as with the Rename function by using the _target instruction in the script:



A typical use case for this is if you need to split one feature into more than one target object with different target attribute values. Here is an example from an INSPIRE Hydrography alignment where one target instance is created for every source geometry:



You can download the script snippets here and here and import them in hale studio as a Groovy snippet by going to File -> Import -> Groovy Snippet. Please note that some scripts use protected functions, so you might need to “Lift Groovy Restrictions” to execute the script. Make sure you replace the placeholder attribute names with your own attribute names.

Happy transforming!

(more)

In our third installment of the 2020 Groovy Week, we are going to cover a typical use case of hale studio’s Collector feature: building a bounding box covering the geometries of all source objects. For an introduction to the powerful Collector feature, make sure to check out our previous post on the subject. Like in the previous Groovy Week posts, note that this article assumes you have working knowledge of hale studio and know the terminology.

Wednesday’s Script: Building a Bounding Box using a Collector (Florian)

A typical use case for calculating a bounding box covering the geometries of all source objects is creating an alignment to the INSPIRE Species Distribution theme. Transforming to this theme typically comprises creating several SpeciesDistributionUnit instances and a single SpeciesDistributionDataset instance which references those units.

When creating the SpeciesDistributionDataset instance, one of the mandatory target attributes to fill is domainExtent which should contain the geographic extent of all contained SpeciesDistributionUnit instances. To achieve this, we will create a bounding box covering all SpeciesDistributionUnit geometries on the fly during the transformation using a Collector. While there exists the specialized function Compute Extent for calculating bounding boxes, using it would require a Merge over all source instances. Using a Collector is more efficient in this case.

Instead of using the Rename function to map the source geometry to the SpeciesDistributionUnit, use the following Groovy script:



To assign the calculated bounding to the domainExtent property of the SpeciesDistributionDataset type, use the script below:



Make sure to reduce the priority of the SpeciesDistributionDataset transformation to make sure that it is executed last, i.e. after all SpeciesDistributionUnits are created.

You can download the script snippets here and here and import them in hale studio as a Groovy snippet by going to File -> Import -> Groovy Snippet. Please note that some scripts use protected functions, so you might need to “Lift Groovy Restrictions” to execute the script. Make sure you replace the placeholder attribute names with your own attribute names.

Happy transforming!

In our third installment of the 2020 Groovy Week, we are going to cover a typical use case of hale studio’s Collector feature: building a bounding box covering the geometries of all source objects. For an introduction to the powerful Collector feature, make sure to check out our previous post on the subject. Like in the previous Groovy Week posts, note that this article assumes you have working knowledge of hale studio and know the terminology.

Wednesday’s Script: Building a Bounding Box using a Collector (Florian)

A typical use case for calculating a bounding box covering the geometries of all source objects is creating an alignment to the INSPIRE Species Distribution theme. Transforming to this theme typically comprises creating several SpeciesDistributionUnit instances and a single SpeciesDistributionDataset instance which references those units.

When creating the SpeciesDistributionDataset instance, one of the mandatory target attributes to fill is domainExtent which should contain the geographic extent of all contained SpeciesDistributionUnit instances. To achieve this, we will create a bounding box covering all SpeciesDistributionUnit geometries on the fly during the transformation using a Collector. While there exists the specialized function Compute Extent for calculating bounding boxes, using it would require a Merge over all source instances. Using a Collector is more efficient in this case.

Instead of using the Rename function to map the source geometry to the SpeciesDistributionUnit, use the following Groovy script:



To assign the calculated bounding to the domainExtent property of the SpeciesDistributionDataset type, use the script below:



Make sure to reduce the priority of the SpeciesDistributionDataset transformation to make sure that it is executed last, i.e. after all SpeciesDistributionUnits are created.

You can download the script snippets here and here and import them in hale studio as a Groovy snippet by going to File -> Import -> Groovy Snippet. Please note that some scripts use protected functions, so you might need to “Lift Groovy Restrictions” to execute the script. Make sure you replace the placeholder attribute names with your own attribute names.

Happy transforming!

(more)

News entry thumbnail
Groovy Week 2020: Logging and Error Handling
05.05.2020 by Florian Esser, Thorsten Reitz

In our second installment of the 2020 Groovy Week, we are going to cover the topic of adding messages to the transformation log as well as showing how to handle situations where an error needs to be raised, e.g. due to incomplete or unexpected source data. Like the previous post, note that this article assumes you have working knowledge of hale studio and know the terminology.

Tuesday’s Script: Handling and logging Errors and other relevant information (Florian)

When using a Groovy script for a specific mapping task, you often need to add checks that verify that the incoming source data can actually be transformed correctly. In cases where the transformation cannot proceed (e.g. because of missing or erroneous source data), the cause needs to be reported. To achieve that, you can add error, warning and informational messages to the transformation log as well as throw exceptions to signal a critical error.

The following script is part of an alignment that transforms road data to the INSPIRE Transport Network Roads schema. It first defines two conditions hatFahrbahnachse and hatStrassenachse based on if the source instance contains a link to a AX_Fahrbahnachse or AX_Strassenachse source instance. If neither condition is met, the transformation cannot proceed. An error message is added to the transformation log and a NoResultException is thrown to indicate that the script did not create a result in this case, which is a critical error.



Not all situations are equally critical, of course. If a warning or informational message is sufficient, use one of the two other log levels that are supported by hale studio: warning and info. To emit a warning to the transformation log, use _log.warn. For informational messages, use _log.info.

All messages emitted from a Groovy script can be accessed in hale studio via the “Instance transformation” entry of a transformation run in the “Report List”, as shown in the following picture:

You can download the script snippet here and import it in hale studio as a Groovy snippet by going to File -> Import -> Groovy Snippet. Please note that some scripts use protected functions, so you might need to “Lift Groovy Restrictions” to execute the script. Make sure you replace the placeholder attribute names with your own attribute names.

Happy transforming!

In our second installment of the 2020 Groovy Week, we are going to cover the topic of adding messages to the transformation log as well as showing how to handle situations where an error needs to be raised, e.g. due to incomplete or unexpected source data. Like the previous post, note that this article assumes you have working knowledge of hale studio and know the terminology.

Tuesday’s Script: Handling and logging Errors and other relevant information (Florian)

When using a Groovy script for a specific mapping task, you often need to add checks that verify that the incoming source data can actually be transformed correctly. In cases where the transformation cannot proceed (e.g. because of missing or erroneous source data), the cause needs to be reported. To achieve that, you can add error, warning and informational messages to the transformation log as well as throw exceptions to signal a critical error.

The following script is part of an alignment that transforms road data to the INSPIRE Transport Network Roads schema. It first defines two conditions hatFahrbahnachse and hatStrassenachse based on if the source instance contains a link to a AX_Fahrbahnachse or AX_Strassenachse source instance. If neither condition is met, the transformation cannot proceed. An error message is added to the transformation log and a NoResultException is thrown to indicate that the script did not create a result in this case, which is a critical error.



Not all situations are equally critical, of course. If a warning or informational message is sufficient, use one of the two other log levels that are supported by hale studio: warning and info. To emit a warning to the transformation log, use _log.warn. For informational messages, use _log.info.

All messages emitted from a Groovy script can be accessed in hale studio via the “Instance transformation” entry of a transformation run in the “Report List”, as shown in the following picture:

You can download the script snippet here and import it in hale studio as a Groovy snippet by going to File -> Import -> Groovy Snippet. Please note that some scripts use protected functions, so you might need to “Lift Groovy Restrictions” to execute the script. Make sure you replace the placeholder attribute names with your own attribute names.

Happy transforming!

(more)

News entry thumbnail
Groovy Week 2020: More Free scripts for hale studio
04.05.2020 by Florian Esser, Thorsten Reitz, Johanna Ott

Two years ago, we published a series of posts around using Groovy scripts to extend the functionality that hale studio offers. These posts have been a favorite among you, so we’ve decided to do a reload of the original “Groovy Week” this week!

Let’s start with the basics: When you try to do something in hale studio that isn’t possible with the out-of-the-box functionality, hale studio offers several ways of creating that functionality yourself. The most accessible of these is to add your own transformation functions using Groovy scripting. There are several points where Groovy functions can be added:

  • Type Transformation Functions
  • Property Transformation Functions
  • Custom Functions
  • Post-processing scripts (only through the CLI)

Groovy is a superset of Java, i.e. any valid Java program is also a valid Groovy program. What it makes much easier than Java is the creation of data structures – no boilerplate constructors and initialisation and the like. Here are some key resources for learning Groovy:

The following posts should help you with some common challenges. Please note that this article assumes you have working knowledge of hale studio and know the terminology.

Two years ago, we published a series of posts around using Groovy scripts to extend the functionality that hale studio offers. These posts have been a favorite among you, so we’ve decided to do a reload of the original “Groovy Week” this week!

Let’s start with the basics: When you try to do something in hale studio that isn’t possible with the out-of-the-box functionality, hale studio offers several ways of creating that functionality yourself. The most accessible of these is to add your own transformation functions using Groovy scripting. There are several points where Groovy functions can be added:

  • Type Transformation Functions
  • Property Transformation Functions
  • Custom Functions
  • Post-processing scripts (only through the CLI)

Groovy is a superset of Java, i.e. any valid Java program is also a valid Groovy program. What it makes much easier than Java is the creation of data structures – no boilerplate constructors and initialisation and the like. Here are some key resources for learning Groovy:

The following posts should help you with some common challenges. Please note that this article assumes you have working knowledge of hale studio and know the terminology.

The Groovy Logo, by Zorak1103, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=13358930

Monday’s Script: Create a persistent ID for your target object (Johanna)

Sometimes, you can’t just use a property of a source object to create a unique, persistent ID on the target side. The source object might not have an unique ID, or you might have to merge several source objects. If you just use Generate sequential ID or Generate unique ID, the ID will not be persistent across multiple transformation runs.

If a combination of attributes of the source objects can be found that is unique and stable over time, the values of those attributes can be used to generate an unique ID on the target side that is persistent.

To generate an ID from several source attributes, you can use the function _.crypto.sha256 which is part of the standard Groovy helper functions. The function creates a hash for a string and, as long as the input String is stable, the generated hash value will also be stable.

When using the generated hash as an ID, you should always use a prefix (e.g. the feature type name followed by an underscore) as the hash value might start with a number which is not allowed in a gml:id.

Here is an example for mapping the gml:id attribute of a Road object of in the INSPIRE Transport Network Roads schema:



You can download this script here and import it in hale studio as a Groovy snippet by going to File -> Import -> Groovy Snippet. Please note that some scripts use protected functions, so you might need to “Lift Groovy Restrictions” to execute the script. Make sure you replace the placeholder attribute names with your own attribute names.

Happy transforming!

(more)

At wetransform, we fully support INSPIRE implementers since we think that key problems can be better solved through cross-border, accessible and useable, harmonised data. In February, we have started a new project that heavily builds on harmonised INSPIRE data to solve a critical issue: The massive ecological and economic impact of climate change on our forests.

Forests are subject to numerous stressors: extreme weather events such as heat, drought and heavy rainfall, pests such as bark beetles and air pollutants. Even tree species that were considered to be stable have suffered from these stressors in the last years. As a result, forest experts face new challenges. They have to identify stands that are at a high risk of being endangered. For each location, they need to find optimal forest development types, tree species and intraspecific varieties. Their measures must ensure that the biodiversity of forests and economic yields are maintained despite climate change.

The factors that influence the general vitality and productivity of forests (climate, weather, soil, geology, morphology, biodiversity, age mixing, etc.) are complex and interconnected. Forest experts consider these in the planning process. However, an in-depth analysis of these variables and their interconnectedness is still a big challenge.

A keystone of new approaches is, like in many other industries, a shift to data-driven decision support. Data on the aforementioned factors can form a basis for more effective and faster decision-making, but gathering useful data is not an easy task. The sheer size and complexity of environmental data related to forests ensures that. To compound matters, this data is held by different organizations, which usually have diverse use cases and use different schemas, formats and semantics. For example, one organization may represent the topographic data of a forest region in a shapefile, and another organization may represent the topographic data of a neighbouring region in the same forest as a GML file.

Forest experts thus depend on a wide range of data sets that are acquired from different sources. This interdisciplinary knowledge transfer is a challenge, but is also mission critical. High quality data acquisition and data integration are a precursor to the forest experts’ data driven decision-making and ultimately to the survival of our forests.

To help forest experts and owners to deal with these issues, wetransform has initiated the FutureForst project. FutureForst is a Phase I Artificial Intelligence Lighthouse project co-funded by the German Zukunft-Umwelt-Gesellschaft (ZUG) association. Through FutureForst, forest owners receive comprehensive decision support that has been adapted to their specific situation and goals. These recommendations are based on harmonized data such as the forest inventory, weather, pest development and air pollution.

In order to provide this decision support, we currently evaluate deep-learning methods and “Explainable AI” approaches such as Semantic Reasoning and Bayesian Belief Networks together with our partners, Minerva Intelligence GmbH and the Forstliche Versuchsanstalt Baden-Württemberg.

With the “Explainable AI” approach, the system can generate comprehensible results with accessible recommendations for action. Explainable AI methods enable users to see which variables of the input data lead to which outcome. These can be adjusted by experts and laypersons in any depth and checked for plausibility. On this basis, experts and laymen can then decide which forest development types and tree species can be established as “future forests” - forest ecosystems that can withstand climate change.

In a further step, different climate forecasts can then be taken into account and forest conversion scenarios can be simulated. In addition, a solution forum will be offered where users and partners can exchange information on their approach.

The data that is core to the approach is harmonised and published with wetransform’s tools, hale connect and hale studio. These tools have already been used by hundreds of organizations to consolidate heterogenous stacks into harmonized data that can be easily analysed. Afterwards, components of Minerva’s AI suite are used for the analysis and development of explainable recommendations based on the harmonized data.

When this project is finished, we aim to provide a FutureForst solution that offers:

  • Always up-to-date, homogeneous data basis
  • Complete picture of the environment including real-time stressors such as pest infestation
  • Highly local, explainable recommendations for action based on international data
  • Solution forum for users and partners We are currently executing open remote workshops on this project every two weeks.

These workshops provide opportunities to learn about what the concrete outcomes of the project are, and to contribute experiences and requirements. The workshops are themed as follows:

  • 15.04.2020: End user scenario development
  • 29.04.2020: Existing and missing data
  • 13.05.2020: AI Approaches and Algorithms

Reach out to us to stay updated on the project’s progress and let us know how your country is dealing with this challenge:


At wetransform, we fully support INSPIRE implementers since we think that key problems can be better solved through cross-border, accessible and useable, harmonised data. In February, we have started a new project that heavily builds on harmonised INSPIRE data to solve a critical issue: The massive ecological and economic impact of climate change on our forests.

Forests are subject to numerous stressors: extreme weather events such as heat, drought and heavy rainfall, pests such as bark beetles and air pollutants. Even tree species that were considered to be stable have suffered from these stressors in the last years. As a result, forest experts face new challenges. They have to identify stands that are at a high risk of being endangered. For each location, they need to find optimal forest development types, tree species and intraspecific varieties. Their measures must ensure that the biodiversity of forests and economic yields are maintained despite climate change.

The factors that influence the general vitality and productivity of forests (climate, weather, soil, geology, morphology, biodiversity, age mixing, etc.) are complex and interconnected. Forest experts consider these in the planning process. However, an in-depth analysis of these variables and their interconnectedness is still a big challenge.

A keystone of new approaches is, like in many other industries, a shift to data-driven decision support. Data on the aforementioned factors can form a basis for more effective and faster decision-making, but gathering useful data is not an easy task. The sheer size and complexity of environmental data related to forests ensures that. To compound matters, this data is held by different organizations, which usually have diverse use cases and use different schemas, formats and semantics. For example, one organization may represent the topographic data of a forest region in a shapefile, and another organization may represent the topographic data of a neighbouring region in the same forest as a GML file.

Forest experts thus depend on a wide range of data sets that are acquired from different sources. This interdisciplinary knowledge transfer is a challenge, but is also mission critical. High quality data acquisition and data integration are a precursor to the forest experts’ data driven decision-making and ultimately to the survival of our forests.

To help forest experts and owners to deal with these issues, wetransform has initiated the FutureForst project. FutureForst is a Phase I Artificial Intelligence Lighthouse project co-funded by the German Zukunft-Umwelt-Gesellschaft (ZUG) association. Through FutureForst, forest owners receive comprehensive decision support that has been adapted to their specific situation and goals. These recommendations are based on harmonized data such as the forest inventory, weather, pest development and air pollution.

In order to provide this decision support, we currently evaluate deep-learning methods and “Explainable AI” approaches such as Semantic Reasoning and Bayesian Belief Networks together with our partners, Minerva Intelligence GmbH and the Forstliche Versuchsanstalt Baden-Württemberg.

With the “Explainable AI” approach, the system can generate comprehensible results with accessible recommendations for action. Explainable AI methods enable users to see which variables of the input data lead to which outcome. These can be adjusted by experts and laypersons in any depth and checked for plausibility. On this basis, experts and laymen can then decide which forest development types and tree species can be established as “future forests” - forest ecosystems that can withstand climate change.

In a further step, different climate forecasts can then be taken into account and forest conversion scenarios can be simulated. In addition, a solution forum will be offered where users and partners can exchange information on their approach.

The data that is core to the approach is harmonised and published with wetransform’s tools, hale connect and hale studio. These tools have already been used by hundreds of organizations to consolidate heterogenous stacks into harmonized data that can be easily analysed. Afterwards, components of Minerva’s AI suite are used for the analysis and development of explainable recommendations based on the harmonized data.

When this project is finished, we aim to provide a FutureForst solution that offers:

  • Always up-to-date, homogeneous data basis
  • Complete picture of the environment including real-time stressors such as pest infestation
  • Highly local, explainable recommendations for action based on international data
  • Solution forum for users and partners We are currently executing open remote workshops on this project every two weeks.

These workshops provide opportunities to learn about what the concrete outcomes of the project are, and to contribute experiences and requirements. The workshops are themed as follows:

  • 15.04.2020: End user scenario development
  • 29.04.2020: Existing and missing data
  • 13.05.2020: AI Approaches and Algorithms

Reach out to us to stay updated on the project’s progress and let us know how your country is dealing with this challenge:


(more)