It’s time to address the ever-present elephant in the room: data harmonization. Without the right tools and knowledge, geodata harmonizations can be complex and time-consuming tasks. There are a few common patterns that contribute to the complexity of geodata harmonizations, and here’s a list of five things that you should avoid when undertaking a data harmonization project.
It’s time to address the ever-present elephant in the room: data harmonization. Without the right tools and knowledge, geodata harmonizations can be complex and time-consuming tasks. There are a few common patterns that contribute to the complexity of geodata harmonizations, and here’s a list of five things that you should avoid when undertaking a data harmonization project.
Misunderstanding differences between the source data and the target data have to be mapped: The mappings between the source data and the target data need to be mapped well. It’s good to have a visual representation of this process. Having a diagram of what the data initially looks like, how it will be mapped, and what exactly it should look like at the end makes things easier to interepret and analyze. Apart from that, an issue that arises often is when there are two attributes with the same name but have different namespaces. In this case, make sure you use the right namespace.
Having the wrong tools: Tools that are currently in place for data harmonization workflows may need to be tweaked or changed. Not every ETL tool can keep up with the increasing complexity and size of datasets. It’s important to assess the capabilities of current tools and see if their capacity can match up to the task at hand. If it can’t, it may be time to look into a new toolset that can better serve your needs and help you perform your tasks easily and efficiently.
Combining software inefficiently: There are different parts of the data harmonization process that include publishing original datasets, metadata generation and validation, data transformation and publishing and viewing of services. There are various tools that you can use for each of these stages, but this doesn’t necessarily mean that the output from each tool will be extensible and guarantee a successful input for the next tool. There is a loss of data consistency if there are too many application breaks, that eventually must be dealt with manually. Ensure that you’re using the right tools that have the right extensibility, or better yet, find a tool that can help you deal with all these steps in an integrated manner.
Not creating documentation: Documentation can give you the edge you need when you’re running low on project times and deadlines are looming. Imagine having an internal go-to-repository of transformation projects to help you out whenever you seem to run into an issue, or already having a specific roadmap laid out for a complex transformation project. This documentation can exist in different forms, for example, a best practices sheet or a target data model prototype set.
Placing high manual efforts in every harmonization process: Even with the correct tools, knowledge, and documentation, project times and costs could still be high because of the amount of manual work that goes in. Harmonization projects often have opportunities for automation; it’s important to identify the parts of the project that don’t need manual intervention. These segments can then be capitalized on by looking into how the process can be automated; this can either be done manually or with the right toolset.
Have any more ideas that could benefit the GIS Analyst Community? Check out our discussion forum to add on your ideas and link with like-minded GIS Analysts.
Looking for an integrated tool through which you can perform all operations relevant to data transformations? Try hale connect for free!
(more)
We want to make 2019 the best year ever for the users of our software, and here’s the first contribution towards this: we’ve just released hale studio 3.5.0!
This version brings several changes you need to know about:
- hale studio now uses Eclipse Photon, which improves performance by up to 100% and removes several issues on non-windows platforms such as low performance on Linux with GTK 3.18 and start-up issues on macOS
- The hale pro plug-ins are now distributed through an update site. This also removed the error messages related to drop-in plug-ins.
- The GeoServer plug-in is not anymore part of the default distribution and needs to be installed through an update site.
This enables independent update cycles of the plug-in. It also realizes a clearer separation of concerns – the plug-in will now be maintained and supported by GeoSolutions.
Installing Geoserver and Pro Plug-Ins
You can find instructions on how to install plug-ins from updates sites here. Please note that for the Pro Plug-Ins you still need a license. These licenses are provided to all customers of wetransform who have a support contract.
Important note for Windows users
In case you experience problems with hale studio after upgrading from a previous version (e.g. a missing menu bar or other UI glitches), please make sure to delete or rename the workspace folder HALE in %APPDATA%\wetransform as this folder may contain data incompatible with the new Photon client platform.
Deegree Workspace configuration
A bit like the Geoserver app-schema and Xtraserver configuration, it is now also possible to generate deegree workspace configurations from hale studio alignments. This feature is only available through the user interface, not through the command line interface. Its development was funded by the Landesvermessung und Geobasisinformation Brandenburg.
Partition output by spatial extent
The GML/XML writer in hale studio already supported several partitioning modes, e.g. by Feature Type, or simply by feature count. We’ve now added support to partition output by spatial extent, using either a custom or a standard tile grid. The development of this feature was supported by swisstopo.
Other changes and fixes
- Improved Xtraserver export
- Migration support for join conditions and context filters
- Support for alignments with inline transformations in headless mode
- Summary in transformation reports
- Support for matching shortened Shapefile properties
- Upgraded Groovy to 2.4
- Upgraded PostgreSQL driver to 42.2.4, PostGIS driver to 2.2.1
- Fixed file names when partitioning by feature type in a GML export
- Improved performance for inline transformations where inlined transformation contains Groovy scripts
- Prevent change of project resources paths if project is exported to hale connect
- Prevent removal of existing source data when loading additional source data
- Allow loading hale schema definition file even if constraint type can’t be recreated
The complete changelog is available here.
Plans for 2019
Our current planning includes three more releases for 2019:
- 3.5.1: Bugfixes that were originally planned for 3.5.0 but didn’t make the cut; might also include updates to the GeoJSON writer to include model transformation rules (then it will become 3.6.0).
- 4.0.0: Migration to OpenJDK 11, Improved SSL certificate handling, many significant usability improvements
- 4.1.0: CRS handling improvements, remaining issues after 4.0.0 release
Download hale studio
Download the latest version and send us your feedback:
Thanks to our customers for funding this work on the 26th release of hale studio!
We want to make 2019 the best year ever for the users of our software, and here’s the first contribution towards this: we’ve just released hale studio 3.5.0!
This version brings several changes you need to know about:
- hale studio now uses Eclipse Photon, which improves performance by up to 100% and removes several issues on non-windows platforms such as low performance on Linux with GTK 3.18 and start-up issues on macOS
- The hale pro plug-ins are now distributed through an update site. This also removed the error messages related to drop-in plug-ins.
- The GeoServer plug-in is not anymore part of the default distribution and needs to be installed through an update site.
This enables independent update cycles of the plug-in. It also realizes a clearer separation of concerns – the plug-in will now be maintained and supported by GeoSolutions.
Installing Geoserver and Pro Plug-Ins
You can find instructions on how to install plug-ins from updates sites here. Please note that for the Pro Plug-Ins you still need a license. These licenses are provided to all customers of wetransform who have a support contract.
Important note for Windows users
In case you experience problems with hale studio after upgrading from a previous version (e.g. a missing menu bar or other UI glitches), please make sure to delete or rename the workspace folder HALE in %APPDATA%\wetransform as this folder may contain data incompatible with the new Photon client platform.
Deegree Workspace configuration
A bit like the Geoserver app-schema and Xtraserver configuration, it is now also possible to generate deegree workspace configurations from hale studio alignments. This feature is only available through the user interface, not through the command line interface. Its development was funded by the Landesvermessung und Geobasisinformation Brandenburg.
Partition output by spatial extent
The GML/XML writer in hale studio already supported several partitioning modes, e.g. by Feature Type, or simply by feature count. We’ve now added support to partition output by spatial extent, using either a custom or a standard tile grid. The development of this feature was supported by swisstopo.
Other changes and fixes
- Improved Xtraserver export
- Migration support for join conditions and context filters
- Support for alignments with inline transformations in headless mode
- Summary in transformation reports
- Support for matching shortened Shapefile properties
- Upgraded Groovy to 2.4
- Upgraded PostgreSQL driver to 42.2.4, PostGIS driver to 2.2.1
- Fixed file names when partitioning by feature type in a GML export
- Improved performance for inline transformations where inlined transformation contains Groovy scripts
- Prevent change of project resources paths if project is exported to hale connect
- Prevent removal of existing source data when loading additional source data
- Allow loading hale schema definition file even if constraint type can’t be recreated
The complete changelog is available here.
Plans for 2019
Our current planning includes three more releases for 2019:
- 3.5.1: Bugfixes that were originally planned for 3.5.0 but didn’t make the cut; might also include updates to the GeoJSON writer to include model transformation rules (then it will become 3.6.0).
- 4.0.0: Migration to OpenJDK 11, Improved SSL certificate handling, many significant usability improvements
- 4.1.0: CRS handling improvements, remaining issues after 4.0.0 release
Download hale studio
Download the latest version and send us your feedback:
Thanks to our customers for funding this work on the 26th release of hale studio!
(more)
The 2020 deadline for INSPIRE implementation is approaching. Many implementers are deciding on the toolset to aid them with this implementation. Currently, organizations are preparing Annexx II and II implementation, or are working hard to deliver INSPIRE Priority data sets.
To ensure your success with the implementation, we are offering critical capabilities to a successful implementation as a discounted bundle to help you reduce risks, cut costs and implement INSPIRE with maximum efficiency. Give your INSPIRE implementation a boost with this special package:
Hale Connect Micro Organisation License
Metadata validation, publishing and viewing of services and data harmonization: These three processes form the cornerstones of the INSPIRE Implementation process. The complexity of each process along with the aplication breaks results in significant challenges for preserving data quality and full standard compliance. With hale connect, you can perform all of these tasks on an integrated platform. With the hale connect micro organisation license, you get:
- Unlimited transformation projects, data models and themes
- 3 Named User accounts
- 500 capacity points, which allow you to publish approximately 20 WMS/WFS datasets of 100 MB volume each.
Public Cloud Hosting
hale connect is available in on premise, private cloud and public cloud hosting. Public cloud hosting is the most cost-effective way to host small to midscale amounts of services. Since your services are being hosted on a shared platform, the application management and maintenance will be handled. The cloud that your data will be hosted on complies with C5 Standards which offer the highest level of security, reliability and scalability.
INSPIRE SLA
We guarantee that the data sets we create as well as the service we provide are fully INSPIRE compliant. This means that services hosted on our platform meet the INSPIRE service quality requirements such as availability, throughput and response time. We also guarantee updates to any components of the system to meet updated INSPIRE specifications within six months of them being released.
Support from the INSPIRE experts
Our goal is to make INSPIRE work for you. For this purpose, we offer you our expertise stemming from more than 200 INSPIRE implementation projects. As part of this package, you get access to our support and can request specific support, e.g. for:
- Set up of publishing and metadata configuration
- Development of transformation projects
- General training and workshops
The package includes 40 hours of support to allow you to reach your full potential as fast as possible, and reduce time taken on critical projects. This is enough to support you with implementing 2-3 INSPIRE themes, and will provide you with a strong foundation for future implementations.
If you’d like to try it out yourself before making a decision, you can sign up for a free trial here.
The bundle is priced at €8.000 per year.
The minimum contract duration is 12 months. All prices are without VAT; reverse charge may apply.
Contact us to learn more about what this offer can do for you and see how you can implement INSPIRE with ease.
The 2020 deadline for INSPIRE implementation is approaching. Many implementers are deciding on the toolset to aid them with this implementation. Currently, organizations are preparing Annexx II and II implementation, or are working hard to deliver INSPIRE Priority data sets.
To ensure your success with the implementation, we are offering critical capabilities to a successful implementation as a discounted bundle to help you reduce risks, cut costs and implement INSPIRE with maximum efficiency. Give your INSPIRE implementation a boost with this special package:
Hale Connect Micro Organisation License
Metadata validation, publishing and viewing of services and data harmonization: These three processes form the cornerstones of the INSPIRE Implementation process. The complexity of each process along with the aplication breaks results in significant challenges for preserving data quality and full standard compliance. With hale connect, you can perform all of these tasks on an integrated platform. With the hale connect micro organisation license, you get:
- Unlimited transformation projects, data models and themes
- 3 Named User accounts
- 500 capacity points, which allow you to publish approximately 20 WMS/WFS datasets of 100 MB volume each.
Public Cloud Hosting
hale connect is available in on premise, private cloud and public cloud hosting. Public cloud hosting is the most cost-effective way to host small to midscale amounts of services. Since your services are being hosted on a shared platform, the application management and maintenance will be handled. The cloud that your data will be hosted on complies with C5 Standards which offer the highest level of security, reliability and scalability.
INSPIRE SLA
We guarantee that the data sets we create as well as the service we provide are fully INSPIRE compliant. This means that services hosted on our platform meet the INSPIRE service quality requirements such as availability, throughput and response time. We also guarantee updates to any components of the system to meet updated INSPIRE specifications within six months of them being released.
Support from the INSPIRE experts
Our goal is to make INSPIRE work for you. For this purpose, we offer you our expertise stemming from more than 200 INSPIRE implementation projects. As part of this package, you get access to our support and can request specific support, e.g. for:
- Set up of publishing and metadata configuration
- Development of transformation projects
- General training and workshops
The package includes 40 hours of support to allow you to reach your full potential as fast as possible, and reduce time taken on critical projects. This is enough to support you with implementing 2-3 INSPIRE themes, and will provide you with a strong foundation for future implementations.
If you’d like to try it out yourself before making a decision, you can sign up for a free trial here.
The bundle is priced at €8.000 per year.
The minimum contract duration is 12 months. All prices are without VAT; reverse charge may apply.
Contact us to learn more about what this offer can do for you and see how you can implement INSPIRE with ease.
(more)
Wetransform is currently participating in an exchange programm for young scientists from countries in the West Balkan. Nikolina is researching how to use INSPIRE to exchange and display information on atmospheric conditions relevant for geodesic applications. This article is based on a recently published IJSDIR paper she has authored.
Currently, the Republika Srpska and the Federation of Bosnia and Herzegovina have strategies at the state level, and national level strategies are under development. The idea for the future is that after adopting this national strategy, the iGEO portal will play the role of the core component for NSDI services.
Towards a national strategy: The Role of INSPIRE
Eleven years after INSPIRE was conceived and five years after its formal adoption, we can see clear results and benefits. Every day, more and more spatial data within the framework of INSPIRE becomes available throughout the EU e.g. Geodata Services for European soil data. They may not yet be harmonized, since that process has just started, but data is already available for discovery and viewing.
The INSPIRE process has involved hundreds of experts and professionals from all over Europe. This participatory approach is certainly innovative, not only in relation to the development of SDIs, but also for the formulation of public policy at the European level.
Current Situation
The online geoportal (named “iGEO”) makes it easier for citizens, governments, and private-sector users to find and access vast quantities of national geographic information and related services. The iGEO portal is currently an experimental research application, but it may have the potential to become the official national Bosnian and Herzegovinian portal for SDI.
The geoportal has already proven its value as an essential component of civil infrastructure planning and design. An actively operating geoportal means that other governmental organizations can use iGEO data and make their data accessible. This is the first step in establishing a Bosnia-Herzegovinian national geoportal as part of a national SDI.
The geoportal is modeled after the EU INSPIRE Directive to share geographic information across Europe. This geoportal was created by small and medium-sized enterprises (SMEs) in Bosnia and Herzegovina. The data available through the geoportal includes digital orthophotos, base maps, administrative units, and land survey information (Figure 1).
Wetransform is currently participating in an exchange programm for young scientists from countries in the West Balkan. Nikolina is researching how to use INSPIRE to exchange and display information on atmospheric conditions relevant for geodesic applications. This article is based on a recently published IJSDIR paper she has authored.
Currently, the Republika Srpska and the Federation of Bosnia and Herzegovina have strategies at the state level, and national level strategies are under development. The idea for the future is that after adopting this national strategy, the iGEO portal will play the role of the core component for NSDI services.
Towards a national strategy: The Role of INSPIRE
Eleven years after INSPIRE was conceived and five years after its formal adoption, we can see clear results and benefits. Every day, more and more spatial data within the framework of INSPIRE becomes available throughout the EU e.g. Geodata Services for European soil data. They may not yet be harmonized, since that process has just started, but data is already available for discovery and viewing.
The INSPIRE process has involved hundreds of experts and professionals from all over Europe. This participatory approach is certainly innovative, not only in relation to the development of SDIs, but also for the formulation of public policy at the European level.
Current Situation
The online geoportal (named “iGEO”) makes it easier for citizens, governments, and private-sector users to find and access vast quantities of national geographic information and related services. The iGEO portal is currently an experimental research application, but it may have the potential to become the official national Bosnian and Herzegovinian portal for SDI.
The geoportal has already proven its value as an essential component of civil infrastructure planning and design. An actively operating geoportal means that other governmental organizations can use iGEO data and make their data accessible. This is the first step in establishing a Bosnia-Herzegovinian national geoportal as part of a national SDI.
The geoportal is modeled after the EU INSPIRE Directive to share geographic information across Europe. This geoportal was created by small and medium-sized enterprises (SMEs) in Bosnia and Herzegovina. The data available through the geoportal includes digital orthophotos, base maps, administrative units, and land survey information (Figure 1).
Technically, the iGEO geoportal offers web services allowing access to the most precise base map data. This data can be accessed by Bosnia-Herzegovinian local municipalities, government agencies and utility infrastructure organizations. The portal provides full catalogue service web (CSW) functionality via the CSW plug-in. The following standard CSW operations are currently supported:
- GetCapabilities
- GetRecords
- GetRecordById
- GetDomain
- DescribeRecord
The Internal Catalogue Store supports two metadata schemes:
- Dublin Core
- ISO Metadata Profile
iGEO raster and vector layers are registered using metadata, which describes the location, age, quality, and other characteristics of the resources (Figure 2). With access to this mapping material information, organizations can make decisions based on the best resources available. iGEO represents the geoportal component of the NSDI initiative in Bosnia and Herzegovina.
The Geoportal provides the datasets concerning the following aspects:
- administrative boundaries
- buildings
- cadastral parcels
- colored DTM
- scanned maps (regardless of map scale)
- geographical names
- hydrography networks
- ortho-photographies (regardless of resolution)
- railway networks
- road networks
- airports runways
- governmental buildings
- cadastral trigonometric points (levels I-IV)
Currently, we are working towards establishing interfaces with the GeoNetwork platform to enable setting up an INSPIRE view service with GeoNetwork and GeoServer.
The impact of iGEO - One Infrastructure, Many use cases
At this point iGEO-related experimental web services already handle data requests from local municipalities and utility companies, such as the Municipality of Banja Luka, the Municipality of Prijedor, the Municpiality of Gradiška, the Municipality of Teslić, M:Tel (telecommunications provider based in the Republic of Srpska), Vodovod i kanalzacija (RS-based local water and sewer company), Elektro-Krajina (regional electricity company), and many others.
There are two software projects under development based on the iGEO services geoportal as a principal data source: AreaCAD-GIS and BrownInfo. More iGEO-based specialized applications are envisioned and their planned architectures are at different stages of completeness by utility infrastructure organizations regionwide.
The web services provided carry detailed metadata on accompanying data layers, according to ISO/TS 19139-2:2012 Geographic information – Metadata standard based on OpenGIS Consortium (OGC) rules (Figure 3):
- Web Map Service (WMS)
- Web Feature Service (WFS)
- Web Coverage Service (WCS)
- Web Map Tiled Service (WMTS)
- Catalogue Services for the Web (CSW)
- Streaming KML
- Geography Markup Language (GML)
All data provided by iGEO are serviceable by all the before mentioned web services.
AreaCAD-GIS
AreaCAD-GIS is INOVA informaticki inzenjering’s project representing a multi-tier GIS technology enabling organization-wide authoring and publishing spatial plans, metadata and design information quickly and easily for distribution, internally or on the web. The application targets municipal spatial (urban) planning departments’ productivity improvement and reduction of the time necessary for the issuing of building permits and tracking of their execution, as well as building legalization issues (Figure 8). It also serves as a bridge between a centralized municipal GIS repository and spatial planner acquisition tools by using various types of thin clients – to display acquired cadastral and urban data.
AreaCAD-GIS utilizes iGEO-based web services to query and visualize Bosnia-Herzegovinian administrative data and topographic maps (Figure 9).
BrownInfo
BrowInfo is a University of Banja Luka research project initiated with the aim of defining universal standards of identification, inventory and classification of brownfield sites. It aims to devise sets of effective methods for the collection, systematization, presentation and storage of brownfield-related data for use in the future as a basis for decisions on starting brownfield regeneration projects. Figure 10: BrownInfo software implementation related to Business Zone Incel, Banja Luka
BrownInfo software utilizes iGEO-based web services to query and visualize utility infrastructure data and orthophoto maps for targeted Banja Luka region (Poslovna zona Incel/Business Zone Incel).
Conclusion and outlook
The main stakeholders have already been “inspired” in each country as well as at the European level, which has triggered many other initiatives. Awareness has been raised throughout the public sector and individuals and organizations have been encouraged to think about their roles and the datasets for which they are responsible.
All eyes are now clearly focused on the MS. Their experiences of implementing INSPIRE in the coming period will be crucial for the planning and preparatory activities already started for the review of the INSPIRE Directive. The efficient implementation of INSPIRE also involves cost–benefit considerations, which are a core part of the process. It is important to raise awareness, especially among users, of the need to monitor the benefits, which will not immediately be obvious.
In Bosnia and Herzegovina there is no official strategy for establishing NSDI, there are just borderline strategies at the entity level for Republika Srpska and the Federation of Bosnia and Herzegovina. Administrative borders and complex relations between these two entities have resulted in a situation in which we are still waiting for Bosnia and Herzegovina to officially adopt a national strategy for NSDI. Our idea for the future is that after adopting this strategy, iGEO portal will play the role of the core component for NSDI services.
The Federation of Bosnia and Herzegovina and Republika Srpska have not yet assigned a national geoportal, but significant progress has been made with the help of iGEO. The primary challenge faced by them in their INSPIRE Implementation is one that is faced by the majority of INSPIRE Implementers: data harmonization. This challenge, though, can be dealt with easily if the right tools such as Hale Studio are at hand.
(more)
.e8bb.png)
A few months back, we described how you can setup hale connect to fully automate metadata generation, re-use and validation. In the past months, we have worked together with a reference customer, the LGL Baden-Württemberg, to extend the metadata tools in hale connect substantially. LGL Baden-Württemberg is one of 16 state-level cadastral, mapping and spatial planning authorities in Germany and maintains the state spatial data infrastructure.
The Problem
The problem that the LGL approached us with is one that we have seen many times already: there is a specific metadata profile, but both in terms of process and tools, the maintenance and implementation of this profile requires a lot of extra effort. The LGL used a massive Excel Table with more than 400 rows to describe their profile. This Excel table conflated profile rules from different levels (ISO, INSPIRE, national, and local), which all needed to be updated and kept in sync.
In implementation of the profile, there were often ambiguities or different interpretations of certain aspects of the profile. Furthermore, the documentation was a bit unwieldy and hard to read for most people outside the immediate circle of editors.
Thus, LGL reached out to us to understand how they could improve on this situation. Wetransform originally conducted a UML and Metadata workshop to explore whether developing the profile in a UML Tool such as Enterprise Architect would make sense. However, some aspects such as a nice differential documentation and the generation of an executable test suite directly from the profile proved to be quite hard.
Thus, in early 2018, LGL asked wetransform to develop a toolset that would support the creation of a fully formal profile definition of the ISO metadata schemas. This toolset would enable the definition of profiles on any XML-based schema, be it the CSW 2.0.2 Application Profile ISO 1.0.0 or a GML Application Schema.
The Solution
There were several key requirements that the profile toolset should resolve:
- Definition of type and property constraints (e.g. on cardinality, allowed content and allowed property types.)
- Definition of consistency constraints (e.g. “If A exists, B must also exist”) and other advanced validation rules.
- Addition of documentation via tagged values.
- Automated generation of printable documentation.
- Automated generation of differential documentation.
- Automated generation of Executable Testsuites (ETS) for the INSPIRE Reference Validator (ETF.)
- Automated generation of example XML instance documents for the profile.
- Be easy to use for people without in-depth knowledge in technologies such as XML, XML Schema, Schematron and ETF.
A few months back, we described how you can setup hale connect to fully automate metadata generation, re-use and validation. In the past months, we have worked together with a reference customer, the LGL Baden-Württemberg, to extend the metadata tools in hale connect substantially. LGL Baden-Württemberg is one of 16 state-level cadastral, mapping and spatial planning authorities in Germany and maintains the state spatial data infrastructure.
The Problem
The problem that the LGL approached us with is one that we have seen many times already: there is a specific metadata profile, but both in terms of process and tools, the maintenance and implementation of this profile requires a lot of extra effort. The LGL used a massive Excel Table with more than 400 rows to describe their profile. This Excel table conflated profile rules from different levels (ISO, INSPIRE, national, and local), which all needed to be updated and kept in sync.
In implementation of the profile, there were often ambiguities or different interpretations of certain aspects of the profile. Furthermore, the documentation was a bit unwieldy and hard to read for most people outside the immediate circle of editors.
Thus, LGL reached out to us to understand how they could improve on this situation. Wetransform originally conducted a UML and Metadata workshop to explore whether developing the profile in a UML Tool such as Enterprise Architect would make sense. However, some aspects such as a nice differential documentation and the generation of an executable test suite directly from the profile proved to be quite hard.
Thus, in early 2018, LGL asked wetransform to develop a toolset that would support the creation of a fully formal profile definition of the ISO metadata schemas. This toolset would enable the definition of profiles on any XML-based schema, be it the CSW 2.0.2 Application Profile ISO 1.0.0 or a GML Application Schema.
The Solution
There were several key requirements that the profile toolset should resolve:
- Definition of type and property constraints (e.g. on cardinality, allowed content and allowed property types.)
- Definition of consistency constraints (e.g. “If A exists, B must also exist”) and other advanced validation rules.
- Addition of documentation via tagged values.
- Automated generation of printable documentation.
- Automated generation of differential documentation.
- Automated generation of Executable Testsuites (ETS) for the INSPIRE Reference Validator (ETF.)
- Automated generation of example XML instance documents for the profile.
- Be easy to use for people without in-depth knowledge in technologies such as XML, XML Schema, Schematron and ETF.
To fulfil these, we developed a formal model that could capture all the required information, as well as a set of editing functions that extend the existing schema modelling tools in hale connect. The formal profile contains type and property constraints, consistency constraints and tagged values:
To create a profile, you first have to create a schema using any of the methods supported in hale connect. For creating a metadata profile, use either the ISO 19139 preset or the CSW 2.0.2 Application Profile ISO 1.0.0 preset.
After this step, you have to create a profile of that schema. This is also done through the “Create new Schema” process, using the “Create Profile” option. When the profile has been created (which takes up to 30 seconds), you can go to the Feature Types section and start applying type and property constraints.
To make a single type mandatory, you can just toggle it in the type header. Making it mandatory means that an object of this type has to occur at least once in any document. This is particularly useful in GML Feature Collection, where you want to state which types of objects have to be present.
Most of the constraints are set using the Edit Mode, which you can access through the Pen icon in the type header. Clicking this icon brings you to a powerful editor, where you can set a number of constraints:
- Content Required: If not already true, you can set this to true to make sure that the given property always has to have content. The semantics of this constraint are equivalent to setting the nillable flag to false.
- Minimum Cardinality: The minimum number of times this property needs to occur in any given type. It needs to be higher than the one defined in the schema, but lower or equal than the value given for maximum cardinality.
- Maximum Cardinality: The maximum number of times this property needs to occur in any given type. It needs to be lower than the one defined in the schema, but higher or equal than the value given for minimum cardinality.
- Allowed Values: In this field you can define an enumeration of the actually allowed values. If the property uses an enumeration or code list, you will be able to activate or deactivate options from that enumeration or codelist.
- Allowed Types: If this field uses a Choice, Substitution Group or Union, you can limit the allowed types by deactivating options.
In addition, you can add any number of tagged values to each property. You can use any keys you want, but some control specific behavior:
- label-testcase: Label for this test case in the ETF
Testsuite - description-testcase: Description for this test case in the ETF
Testsuite - module-id: ID of the
TestModulegroup to which the test case generated for this property will be added - example: Content to be used for the generation of XML sample files. If the property is a complex element, setting this value to “true” leads to the element being generated.
.0355.png)
The second big block of functionality is to be found in the “Consistency Conditions” section of the profile. When you go there, you will be able to define various sets of rules, such as:
- If A exists, B must/must not exist
- If A has value “X”, B must have value “Y”
You can combine any number of such rules through logical operators (AND, OR, XOR). To each Consistency Condition, you can add a label and a description, as well as tagged values.
When you want to review your profile at any time, the differential documentation is a handy feature, especially in big schemas like the ISO 19139 schema. It hides all elements (types and properties) for which no change or addition was made in the profile, resulting in a compact, well-readable format.
Either the full profile or the differential documentation can be printed to any device, specifically to PDF. This enables you to get a document that you can send around to any user.
By going to the “Files” section of the profile, you can also export sample XML instance documents for your profile.
And finally, you want to make sure that any data you receive complies with your profile. For this purpose, hale connect can also export an Executable Test Suite, again through the Files section. This ETS can be run in the ETF (INSPIRE Reference Validator).
The profile tools described here can also be used in conjunction with the metadata generation tools introduced with hale connect 1.0. Ask us to learn how to do so, especially if you are already a hale connect user.
If you want to know more about how to use these tools, check out the online help.
Getting Access to the Metadata Tools
We presented the new metadata tools at the INSPIRE conference 2018 in Antwerp just a few weeks back and can now announce general availability.
The new metadata tools are part of all hale connect licenses, independent of deployment mode and license level. While Metadata Generation has been part of hale connect since version 1.0, the profile management tools have been released with version 1.11.0, which is available since October 17th, 2018.
The tools are available starting from 2.900 € per year (Micro Org Public Cloud license). Special pricing is available for multi-year contracts, partners, and customers with support subscriptions.
(more)