News > opensource
News' image preview

The next version of hale studio, your friendly data transformation tool, is here! This time, our focus was on the integration of hale connect and hale studio. We believe that as a community, we could work together effectively by sharing transformation projects in a dedicated environment. The new capabilities enable you to browse and download shared projects and to upload your own projects directly from hale studio.

As usual, we also improved the software in many other aspects:

  • Arbitrary SQL Statements as Schema or Data Source: If you don’t have the option to define a view on a database you’re accessing, you can now use an arbitrary SQL statement both as schema and data source
  • Spatial Operations: We’ve now made the spatial index hale creates when loading data accessible through Groovy Scripting, and provided helper functions to make usage in scripts easier. We also added in a Spatial Join function that supports all modes of the 9-intersection-model.
  • Updated mapping documentation for help example projects
  • Import/Export hale schema definitions as JSON
  • DMG image for macOS installation

The next version of hale studio, your friendly data transformation tool, is here! This time, our focus was on the integration of hale connect and hale studio. We believe that as a community, we could work together effectively by sharing transformation projects in a dedicated environment. The new capabilities enable you to browse and download shared projects and to upload your own projects directly from hale studio.

As usual, we also improved the software in many other aspects:

  • Arbitrary SQL Statements as Schema or Data Source: If you don’t have the option to define a view on a database you’re accessing, you can now use an arbitrary SQL statement both as schema and data source
  • Spatial Operations: We’ve now made the spatial index hale creates when loading data accessible through Groovy Scripting, and provided helper functions to make usage in scripts easier. We also added in a Spatial Join function that supports all modes of the 9-intersection-model.
  • Updated mapping documentation for help example projects
  • Import/Export hale schema definitions as JSON
  • DMG image for macOS installation

hale studio 3.3.0 Connect with your Community!

We’ve also fixed several bugs and added smaller improvements. The whole list is available in the changelog.

Get the latest version, and let us know what you think of it!


Share Transformation Projects on haleconnect.com

In infrastructure projects such as INSPIRE, we’re all working with the same set of target data models. A request from implementers we hear often is that they want to see how it’s done - they want to build on how others have created compliant INSPIRE data sets. We’ve now made it really easy to share transformation projects and to work together on them with the hale studio/hale connect integration. This integration currently enables you to do three things:

  1. Log in to your haleconnect.com account from within hale studio
  2. Browse and download hale transformation projects from haleconnect.com
  3. Share your own hale transformation projects to the community

If you have a private cloud or on premise installation of hale connect or inspire gis, you can also log in to that by changing the application settings.

On haleconnect.com itself, you can discuss transformation projects with others, such as project partners or customers. They can create tasks or just leave comments on any level of the transformation project, even on an individual cell.

Going forward, we will provide more capabilities, such as working with schemas shared on haleconnect.com and also to share custom functions. We’re looking forward to your feedback on this!


Arbitrary SQL Statements as Source Schema and Source Data

When you’re working with databases as sources, it can be highly efficient to combine views with hale studio’s transformation capabilities. The main upsides include that some processing, filtering and joining can be done close to the data, and that only data really required for the transformation is transferred to hale studio. The issue is that sometimes you don’t have the option to create views on the database. In these cases, you can now define your own using an arbitrary SQL statement, which can include joins, WHERE clauses and anything else your database will allow you to pack into said statement.

You can also use that statement to load exactly the right data for your transformation from the database.

Thanks to the Bundesanstalt für Wasserbau for funding this work.


Spatial Operations Support

Have you ever wanted to join objects based on the spatial relationship of their geometries or perform other typical spatial operations in hale? Now, you can finally do so. We’ve added several features at different levels towards that purpose:

  1. hale studio has an internal spatial index that is now accessible through an API and exposed to Groovy Scripting as well as Transformation Functions.
  2. To make usage of the index in Groovy Scripts and Custom Functions easy, we’ve added two helper functions called spatialIndexQuery and boundaryCovers.
  3. There is also a new cousin to the existing Join function called Spatial Join available now, which lets you join source objects based on the spatial relationship between its geometries. The Spatial Join function supports the relation types contains, covered by, covers, crosses, equals, intersects, overlaps, touches, and within.

Thanks to swisstopo for supporting the work on this feature.


Working with hale schema definition files

We’ve also made another internal capability of hale available to you - the ability to export any data model in a common format, called a hale schema definition. This can be exported both as XML or JSON. So if you’ve ever looked for a tool to convert XML Schema to JSON or a shapefile schema to a XML descriptor, look no further.


(more)

News' image preview

With the latest version of hale studio, we’ve improved support for Web Feature Services (WFS). Our goal is to make it a fully functional WFS client. In this article, we’ll describe those capabilities in detail and outline what limitations there still are - both in hale studio and in WFS implementations.

hale studio has several features related to working with Web Feature Services:

  • Read Schema from WFS via DescribeFeatureType operations
  • Read Source Data from WFS via GetFeature operations
  • Write transformed data (or source data) to a transactional WFS via the WFS-T export configurations

With the latest version of hale studio, we’ve improved support for Web Feature Services (WFS). Our goal is to make it a fully functional WFS client. In this article, we’ll describe those capabilities in detail and outline what limitations there still are - both in hale studio and in WFS implementations.

hale studio has several features related to working with Web Feature Services:

  • Read Schema from WFS via DescribeFeatureType operations
  • Read Source Data from WFS via GetFeature operations
  • Write transformed data (or source data) to a transactional WFS via the WFS-T export configurations

hale studio 3.2.0 Load from WFS

The typical workflow involves these steps:

  1. Go to File -> Import -> Source Schema
  2. Pick the “From WFS” tab
  3. Click on the “…” button to the right of the URL input
  4. Enter the WFS GetCapabilities URL of your service, click “Next”
  5. Pick which types from the WFS you’d like to load, then click “Finish”
  6. Back in the main import dialog, click “Finish” to load the schema

hale studio 3.2.0 Enter capabilities URL

If you hit an issue during this process, you might need to configure your proxy setup to correctly handle the requests to the WFS. After you’ve loaded the schema, the next step is to load some data to work with. In this process, you have several options to control what exactly should be loaded, and how.

  1. Go to File -> Import -> Source Data
  2. Pick the “From WFS” tab
  3. Click on the “…” button to the right of the URL input
  4. Enter the WFS GetCapabilities URL of your service, click “Next”
  5. Pick for which types from the WFS you’d like to load data, then click “Next”
  6. You can now set a bounding box for your query by clicking into the map twice, once for the upper left corner, once for the lower right corner. Click “Next” again.
  7. In the final Query definition step, you can decide to limit the number of features retrieved from the WFS. By default, this option is deactivated and the limit is set to 10.000. Click “Finish”

hale studio 3.2.0 set Bounding Box for WFS

Please note that the integrated GetFeature request builder doesn’t support all options of the WFS 2.0 Query conformance class. You can always edit the actual query and paste it into the main import tab. After you’ve set up the query, you have some more configuration options to control how hale studio will request data from the WFS.

  1. In the main Import Data from WFS tab, click “Next”.
  2. If your schema was imported from the same WFS as this data, you can safely leave the first option unchecked. The second option (“order must strictly match the Schema”) is usually not required. Creating an instance for the root element is necessary if the root element is not a collection, such as a FeatureCollection.
  3. You can also decide whether hale studio should paginate requests. Instead of requesting all data matching the query in one request, this will instruct hale studio to create smaller requests, e.g. requesting 500 features at a time. In this way, load on both the server and the client is lower, and response time is faster. Request pagination thus allows loading larger datasets from a WFS where previously this may have been limited by a server-side limit of retrievable features per request. There are some limitations to using this feature that we’ll talk about below. When you’ve configured pagination, click “next”.
  4. In the next screen, you can define how arc geometries should be handled during import. hale studio doesn’t fully support Arcs in its functions, so we provide the option of interpolating them to polygons or linestrings with linear segments. When you’ve set up interpolation, click “Finish” and let hale studio import the data.
  5. Explore the data via the Source Data view, the Map View, and collect statistics via the Schema Explorer and the the Properties View!

hale studio 3.2.0 partition WFS requests

There are also some known limitations of using hale studio as a WFS client. Some of these can be fixed by implementing improvements in hale studio and are already on our roadmap, others need to be fixed in the WFS implementations:

  • Pagination will only work if the queried WFS server correctly supports the query parameters required for paging, which is not the case for all WFS 2.0.0 implementations out there.
  • Pagination can work with some WFS 1.x implementations as well. The following implementations support paging via the STARTINDEX parameter also for WFS 1.0 and 1.1:
  • There is some overhead involved when issuing a request to the WFS, so if you set the features per request too low (edge case: 1) you may get worse performance than with a higher value. On the other hand, if the limit is too high you may run into timeouts, e.g. when hale studio times out when reading from a WFS because the response was not returned after a certain amount of time (30 seconds by default).
  • hale studio will never fetch more features than reported by the WFS in the numberMatched attribute. If the WFS does not report the correct number of matched features but instead, for example, a server-side limit of features per request, hale studio will not be able to fetch more features than that.
  • hale studio uses the STARTINDEX parameter defined in WFS 2.0.0 to offset the different requests. If this parameter is not supported, the WFS will always return the same set of features in every request.
  • If the WFS does not report the number of matched features at all and does not support STARTINDEX correctly, hale studio will run into a loop of perpetually fetching the same features.

As explained in the beginning, hale studio can also write source or transformed features to WFS supporting this. We’ll cover this in one of our next posts.


(more)

News' image preview

Our team working on hale studio has increased in size, and so we can now already deliver the next hale studio release to you. We plan to release four new versions of hale studio per year - if you’d like to know what the next releases will bring, check out the roadmap on Github!

In this release, we focused on smaller improvements to make some existing features really work:

  • Added support for several Arc-based GML geometry types to be interpolated when read: Arc, ArcString, Circle, CircleByCenterPoint. The interpolation is based on two algorithms that can be selected on import.
  • New transformation function Assign collected values allows the assignment of all values collected by a Groovy transformation function: The new function automatically converts collected values to references if the target property takes references.
  • Added a Project Validator: Checks if instances (features) are valid during live transformation using project-wide validator configuration (e.g. rules or schemas).
  • Improved Alignment layout: Better usage of available space in Alignment and Mapping views.
  • Content assist for project variables: In transformation function wizards such as Formatted String, Regex Analysis and Assign, you can now get auto-fill help to insert project variables.
  • Request pagination for WFS requests: You can now choose what size requests to WFS sources should have.
  • Faster alignment documentation: The HTML alignment documentation is displayed much faster for large mappings due to lazy loading and rendering.
  • Scalable Merge: The Merge function now uses an iterative approach for merging instances which allows for processing more data in a Merge.
  • Scalable CSV file loading: When loading data from CSV files the data is now streamed (similar to XML data sources) and not loaded at once into memory.

Our team working on hale studio has increased in size, and so we can now already deliver the next hale studio release to you. We plan to release four new versions of hale studio per year - if you’d like to know what the next releases will bring, check out the roadmap on Github!

In this release, we focused on smaller improvements to make some existing features really work:

  • Added support for several Arc-based GML geometry types to be interpolated when read: Arc, ArcString, Circle, CircleByCenterPoint. The interpolation is based on two algorithms that can be selected on import.
  • New transformation function Assign collected values allows the assignment of all values collected by a Groovy transformation function: The new function automatically converts collected values to references if the target property takes references.
  • Added a Project Validator: Checks if instances (features) are valid during live transformation using project-wide validator configuration (e.g. rules or schemas).
  • Improved Alignment layout: Better usage of available space in Alignment and Mapping views.
  • Content assist for project variables: In transformation function wizards such as Formatted String, Regex Analysis and Assign, you can now get auto-fill help to insert project variables.
  • Request pagination for WFS requests: You can now choose what size requests to WFS sources should have.
  • Faster alignment documentation: The HTML alignment documentation is displayed much faster for large mappings due to lazy loading and rendering.
  • Scalable Merge: The Merge function now uses an iterative approach for merging instances which allows for processing more data in a Merge.
  • Scalable CSV file loading: When loading data from CSV files the data is now streamed (similar to XML data sources) and not loaded at once into memory.

hale studio 3.1.0 Load from MS SQL Server Database

We’ve also fixed several bugs and added smaller improvements. The whole list is available in the changelog.

Get the latest version, and let us know what you think of it!


Interpolation of Arc Geometries on Import

Until now, hale didn’t support geometries with non-linear interpolation between points, such as arcs. We now added a robust interpolation method that processes Arcs, ArcStrings, Circles, CircleByCenterPoints and creates regular LineStrings and Polygons.

Thanks to the Landesamt für Vermessung und Geobasisinformation Rheinland-Pfalz for funding this work.


Multiple Validators

hale studio’s feature to validate exported XML and GML files has been extended. It is now possible to configure multiple validators. Furthermore, a new type of validator, the Project validator has been added which allows validation of the export result based on rules and schemas (e.g. Schematron schemas) imported into the project.

As soon as the official INSPIRE validator is available, we will also integrate that.


WFS request pagination support

When loading from WFS sources, hale studio now supports request pagination, a capability that was officially introduced with the WFS 2.0 standard. Some implementations such as Geoserver provided support for this before, so you can use this feature with most WFSes. Request pagination allows loading larger datasets from a WFS where previously this may have been limited by a server-side limit of retrievable features per request.

We will add a more detailed post on this feature within the next weeks!


Better UI support for project variables

The use of project variables, a feature that was introduced with hale studio 3.0, is now easier as several wizards in hale studio have been extended to provide content assistance. Simply press Ctrl+Space when inside a text field and choose the desired variable from a list. The content assistant will take care of the correct syntax.


(more)

News' image preview

Just in time for the new year, we have a nice gift for all of you: The latest hale studio release is ready! There were several main areas of work in the past months, largely driven by your requirements and support.

These are the highlight features we’ve added:

  • Read and write Microsoft SQL Server databases, including support for Geography and Geometry column types
  • Add custom validation rules by integrating a Schematron validator
  • Use SKOS code lists
  • Validate against code list values
  • View instance counts for condition contexts instantly
  • Hide optional properties
  • GML Encoding: Specify a number format for geometry ordinates, e.g. if you want a fixed precision
  • GML Encoding: Perform strict Simple Features SF-0 profile encoding for geometries

Just in time for the new year, we have a nice gift for all of you: The latest hale studio release is ready! There were several main areas of work in the past months, largely driven by your requirements and support.

These are the highlight features we’ve added:

  • Read and write Microsoft SQL Server databases, including support for Geography and Geometry column types
  • Add custom validation rules by integrating a Schematron validator
  • Use SKOS code lists
  • Validate against code list values
  • View instance counts for condition contexts instantly
  • Hide optional properties
  • GML Encoding: Specify a number format for geometry ordinates, e.g. if you want a fixed precision
  • GML Encoding: Perform strict Simple Features SF-0 profile encoding for geometries

hale studio 3.1.0 Load from MS SQL Server Database

We’ve also fixed several bugs and added smaller improvements. The whole list is available in the changelog.

Get the latest version, and let us know what you think of it!


MS SQL Server Support

hale studio already supports most of the common relational database management systems, such as MySQL, PostGreSQL and Oracle (via the subscription plugins). The one glaring omission was Microsoft SQL Server, for which we have now added reader and writer support.

We have tested our implementation with SQL Server versions up to 2012. Let us know if you find any issues with newer versions. Thanks to the Bundesanstalt für Wasserbau for funding this work!


SKOS Codelists and Codelist value Validation

Codelists and their cousin Enumerations are a cornerstone of data specifications and data quality. They are used by most of you to ensure that only a set of agreed values is used for certain properties of objects. While the spatial data community, and in particular INSPIRE, has defined some specific formats for such codelists, there are also formats such as SKOS (Simple Knowledge Organisation System) that are in wide use - not only for Linked Data. Now you can load SKOS resources as code lists in hale studio, where a SKOS concept represents an individual code.

In addition, we’ve added codelist value and reference validation to the live validation in hale studio. Thanks to the Bundesanstalt für Wasserbau for funding this work!


Context Counts

To filter source data by the value of one or more properties, you can define contexts for the mapping. When you have loaded source data in hale studio, you can now see directly how many instances and values there are that match these contexts.


Again, Improved Compliance with INSPIRE Data Specifications 4.0

The INSPIRE data specifications mandate usage of the GML SF-0 profile for all of the Annex I themes except Cadastral Parcels. We have now improved the encoding and transformation to better create allowed geometry structures in such cases.


(more)

News' image preview

It’s time for a new major release, and it does come packed with many new features and improvements. First, you might notice the new name and the new logo. HALE by itself was always hard to find with search engines and there is lots of other software called hale. Also, we’ve started to build a full data design and integration platform, and needed to diversify naming. HALE is thus now officially hale studio - but otherwise it remains the same!

These are the highlight features we’ve added:

  • Create Custom transformation functions and use them across projects
  • Write more powerful Groovy/Custom Functions with Collectors, new helper functions and multiple target support
  • Define Priorities for type cells
  • Define Project variables
  • Automate processes with a much improved Command Line Interface
  • Read MS Access (*.mdb) databases
  • Validate XLinks
  • Control Polygon Winding Order for GML output
  • Use Project Generator
  • Use Documentation Generator
  • Improved Transactional WFS support
  • Use any reference map

It’s time for a new major release, and it does come packed with many new features and improvements. First, you might notice the new name and the new logo. HALE by itself was always hard to find with search engines and there is lots of other software called hale. Also, we’ve started to build a full data design and integration platform, and needed to diversify naming. HALE is thus now officially hale studio - but otherwise it remains the same!

These are the highlight features we’ve added:

  • Create Custom transformation functions and use them across projects
  • Write more powerful Groovy/Custom Functions with Collectors, new helper functions and multiple target support
  • Define Priorities for type cells
  • Define Project variables
  • Automate processes with a much improved Command Line Interface
  • Read MS Access (*.mdb) databases
  • Validate XLinks
  • Control Polygon Winding Order for GML output
  • Use Project Generator
  • Use Documentation Generator
  • Improved Transactional WFS support
  • Use any reference map

hale studio 3.0.0 Map View

The majority of these new features were made possible through our contracts with more than 20 customers. We’d like to thank you all for your support!

Note: As this is a major release, projects created with hale studio 3.0 may be incompatible with previous versions of HALE.


Custom transformation functions

HALE has had support for Groovy scripts for type and property transformation, but individual scripts couldn’t be re-used. With custom transformation functions, you can define your own functions using Java or Groovy and re-use them throughout multiple mappings or even multiple projects.

The current implementation of Custom Functions allows you to define input parameters and input/output data structures. In the function editor, you can use lots of helper functions to quickly get results. Please see the Custom Transformation Functions help for more information on how to define and use custom transformation functions.


New Groovy Function Features

Groovy Functions help when the functions we ship with hale studio don’t match specific requirements. To make them more powerful, we have added several new capabilities. The first of these is to output multiple features from a single Groovy Retype, Groovy Merge or Groovy Join function. You can use this functionality to create multiple partial feature from one large feature, or to create copies of a feature in the target data set.

The second improvement we’ve added are Collectors. When you declare a collector, you tell hale that it should create a list of all objects it creates during transformation, for later access in this or another Groovy function. You can use this for post-processing of objects, or to easily create references from a large object, such as as <pre>TransportNetwork</pre>, to small objects, such as <pre>TransportLinks</pre>.

We have also added several new helper functions, in particular for working with geometries. </p>


Project Variables

Sometimes you need to use a value in multiple functions that is a constant during one execution of the alignment, but might need to be changed later on, such as a dataset identifier. So far you’d have to modify all functions that use the constant value. Now, you can just define a global project variable and use that in many functions, such as <pre>Formatted String</pre> and in all Groovy Functions.

You can also set Project variables through the updated hale command line interface. Please see the Project variables section for more information.


Use any Basemap you want

When MapQuest stopped providing acces to their nice tile services, that came as a surprise. We decided to not just drop in a new basemap like OSM Tiles, but insted improved the Map View. While we’ve set the new Stamen Terrain map as the default basemap, you are now free to configure any WMS-based or Tile-based map. Check the Map View section for more instructions on how to configure basemaps.

A big thanks to Fraunhofer IGD’s Competence Center GEO, who contributed the CityServer3D Map View to make these improvements possible.


Read MS Access Databases

MS Access databases are one of the most common formats used in reporting and statistics. In hale studio, you can now use MS Access databases to load source schemas and source data. Thanks to the European Environmental Agency for funding this work.


Improved Compliance with INSPIRE guidance

All GML writers now have an option to control and unify the winding order of any closed rings. You can either leave rings as they are, or unify them to clockwise or counter-clockwise winding order, with the latter being required for GML 3.2 and 3.3. The algorithm we employ takes into account axis order and direction, as reverses order for each level of inner rigns. Furthermore, we have added an Xlink validator to make sure all <pre>href</pre> properties point to valid resources. Thanks to the European Environmental Agency for funding this work.


Much improved Interactive Documentation

The interactive alignment documentation is really helpful to get input from domain experts on an alignment. During several major projects this year, we’ve seen it being used a lot and have improved it according to the feedback. In particular, even very large alignments with hundreds of cells can now be displayed, they can be searched and filtered, and much more.

Thanks to the Arbeitsgemeinschaft der deutschen Vermessungsverwaltungen for funding part of this work.

(more)