Create and publish data sets

Create a new data set

Create new data set

To create a new data set, follow this workflow as a logged in data manager or theme manager:

  1. Go to «Data» on the main toolbar
  2. Go to «Create new Data Set» at the bottom of the data set list
  3. Select a theme to link to your data set.
    • You can also create a data set without linking it to a theme, but without the link, you can’t publish services or execute transformation projects for that data set.
    • Superusers who belong to more than one organisation can select the organisation in which to create a dataset. The dropdown menu of available organisations enables Superusers to create datasets for organizations that they are not currently logged into.
  4. Select the data source. There are a few ways to upload the files belonging to your dataset:
    • Click «Select file(s)» and navigate to the desired dataset.
    • Click «Enter URL(s)» and navigate to the desired URL or WFS.
    • Drag a dataset to «Drop file(s) here»

    The size limit for single file upload is 2 GB.


    Tip: We recommend that you zip your files to reduce upload times and to ensure that no files are lost. To upload a Shapefile, you should zip all individual files (*.shp, *.dbf, *.prj, *.shx, …) in an archive. To upload an XML file such as ALKIS, ISYBAU or GML, we also recommend that you upload them as a zip archive.


    To upload your files from a database click «Connect to database» and enter the following information:

    • Database type (MS SQL, PostgreSQL and PostGIS are currently supported)
    • Host (:Port)
    • Database (Provide the database name)
    • Username
    • Password
  5. Give your data set a name. This name should enable you to quickly und clearly identify the data set in the system. If the linked theme’s metadata configuration is set up to use the name in publishing, the name is also used for data set metadata.
  6. Provide metadata for your data set. Metadata entry is determined by the theme associated with your dataset.

    If the theme uses the metadata editor, enter required and optional metadata for your dataset or services:

    • Metadata categories display a red badge with the number of missing required metadata elements. Click the chevron at the end of any metadata category to expand it.
    • Required input fields are marked by red symbols. Optional input fields are marked by orange symbols.
    • Metadata categories display a green check mark once you have provided all required metadata.
    • Click «Continue» to save the metadata.


    If the theme uses Republish existing metadata:

    • Upload your existing metadata file.

    If the theme uses Link to existing metadata:

    • Provide a link to your metadata file.
  7. In the final step, you can review all desired steps of the publishing and transformation process configured for the theme. By default, you will see up to five toggle switches to publish Download and View Services, to transform the new data set, and to publish services for the transformed data set. If you know you need to work on the data set before it is to be published, deactivate the service publishing.
If you close the browser or the tab, or you click a different link, you will be asked whether the work in progress data set should be saved or deleted. If you pick save, you can continue editing it later.

Create new data set with multiple resource types

hale»connect currently supports a range of file formats which can be used to create data sets. The size limit for single file upload is 2 GB. The size limit for attachments on haleconnect.com is 750 MiB. A data set may contain:

Vector data

  • Supported vector data formats include *.gml and *.shp
  • Multiple *.shp files can be uploaded to create a data set
  • Currently the upload of one *.gml file per data set is supported
    • INSPIRE, 3A, CityGML, XPlanung and ISYBAU are fully supported

Non-spatial data

  • Supported data formats include *.xml

Raster data

  • Supported formats include *.png, and *.GeoTIFF
  • Currently the upload of one raster data resource per data set is supported
  • A separate raster layer is published in the view and download services

Attachments

  • Supported file attachment formats include *.pdf, *.png, *.GeoTIFF, or even textures for 3D data
  • There are multiple ways to upload and associate attachments to your data set
  • Attachments uploaded during data set creation are added to the attachments section automatically
  • The attachment files uploaded to hale»connect have a publicly available link to the attachment file. The links to attachment files can be included in your Predefined data set download service. The links to attachment files and external URLs can also be viewed in the Feature Explorer Tool.
  • Separate service layers are not created for attachment files
  • Visit our help page for more information about working with attachments

A combination of these data resources

  • Data sets can include *.gml, raster and attachment data resources. View and download services for vector and raster data are published as separate layers.
  • Currently the upload of *.shp with combined data resources is not supported

To create a new data set with multiple resource types, follow this workflow as a logged in data manager or theme manager:

  1. Go to «Data» on the main toolbar
  2. Go to «Create new Data Set» at the bottom of the data set list
  3. Select a theme to link to your data set. The theme must be configured to allow additional layers from raster sources.
    • You can also create a data set without linking it to a theme, but without the link, you can’t publish services or execute transformation projects for that data set.
  4. Select the data sources. There are a few ways to upload the files belonging to your data set:
    • Click «Select file(s)» and navigate to the desired data set.
    • Click «Enter URL(s)» and navigate to the desired URL or WFS.
    • Drag a data set to «Drop file(s) here»


    Add a *.gml file, a raster file and as many associated attachment files as necessary.


    Tip: We recommend that you zip your files to reduce upload times and to ensure that no files are lost. To upload a *.png, you should zip all individual files (*.png, *.pgw, …) in an archive. To upload an XML file such as ALKIS, ISYBAU or GML, we also recommend that you upload them as a zip archive.

  5. Give your data set a name. This name should enable you to quickly und clearly identify the data set in the system. If the linked theme’s metadata configuration is set up to use the name in publishing, the name is also used for data set metadata.
  6. Provide metadata for your data set. Metadata entry is determined by the theme associated with your data set.

    If the theme uses the metadata editor, enter required and optional metadata for your data set or services:

    • Metadata categories display a red badge with the number of missing required metadata elements. Click the chevron at the end of any metadata category to expand it.
    • Required input fields are marked by red symbols. Optional input fields are marked by orange symbols.
    • Metadata categories display a green check mark once you have provided all required metadata.
    • Click «Continue» to save the metadata.


    If the theme uses Republish existing metadata:

    • Upload your existing metadata file.

    If the theme uses Link to existing metadata:

    • Provide a link to your metadata file.
  7. In the final step, you can review all desired steps of the publishing process configured for the theme. By default, you will see up to five toggle switches to publish Download and View Services, to transform the new data set, and to publish services for the transformed data set. If you know you need to work on the data set before it is to be published, deactivate the service publishing.
If you close the browser or the tab, or you click a different link, you will be asked whether the work in progress data set should be saved or deleted. If you pick save, you can continue editing it later.

Viewing data sets in the resource list

The data set resource list provides a hierarchical display of data set series and transformed data sets. Dataset series are identified with a ‘Series’ label and transformed resources are identified with a ‘Transformations’ label. Data set series can be expanded by clicking on the ‘>’ sign, which displays a nested list of the datasets in the series. The data set resource list in hale»connect supports a range of filter and sorting options which enable users to quickly locate the resources they are looking for.

Log in to hale»connect and click the Data resource tab to access the list. The tabular listing enables users to filter, sort, select and view datasets. Users have the added option to configure the column layout using the column widget.

The search bar at the top of the resource list filters data sets across all fields. The global scope button enables users to limit the scope of the data sets shown in the resource list. Global scope displays all available data sets, and the Organisation scope displays only those data sets which belong to the organisation of the current, logged in user.

The resource list can also be sorted by column. Click on the column name to sort ascending and click again to sort descending.

Select any resource in the list to navigate directly to the resource overview page.

The status column provides an overview of the system status and validation state of a hale»connect resource. Green indicates that the resources are valid and operating normally. Yellow indicates that the resources have validation warnings and are operating with errors. Red indicates that the resources have validation errors and are not operating normally. Grey indicates that the resources are not published. Click on any of the icons to navigate directly to the resource overview page, the metadata, the download services, or the view services.

The column widget at the top right enables users to configure the column layout of the resource list. Click on the widget to open the dialog. In the dialog, users can manage the column visibility of default columns, however default columns cannot be edited or deleted. Users can also manage the column properties of custom columns including column name, visibility and column content. Users have the additional option to add and delete custom columns. Custom column content is populated using the autofill assistant, which provides access to schema elements. Custom columns are populated with the data value present in the selected schema element, for each resource listing.

All changes must be saved before exiting the column widget dialog.

Edit metadata

Edit metadata for a data set

Editing metadata can be a time-consuming and sometimes tiresome process. For this reason, the system allows automation of about 95% of the process. The remaining fields still have to be filled manually, or edited. Please keep the following hints in mind when manually editing metadata:

  • Your input has to be in the correct format. Some fields will allow any kind of text, while others expect numbers, web links, e-mail addresses, JSON objects or Well-Known-Text (WKT) objects. The round symbols to the left of each input indicate which type of input the system expects. When you hover over these symbols, you’ll get more information.
  • Some fields may be set more than once. To set multiple values, e.g. on «Keywords», click the «+» button below the input field. To remove a field again, click the remove symbol right next to the field.
  • When you edit metadata for any existing data set, each change is saved by the system as soon as you press enter or go to another field. The system indicates this by showing a green border and checkmark on the modified input.

To edit metadata, follow these steps:

  1. Go to «Data» on the main toolbar
  2. Pick the data set from the resource list for which you’d like to edit metadata
  3. Go to «Metadata»
  4. Enter required and optional metadata for your dataset or services. Open one of the metadata categories by clicking on the title or the chevron.
  5. Provide a value for any field marked with a red symbol. You can also edit existing values.
    • Red symbols turn green once required metadata has been entered, and the number of missing required metadata elements is updated in the category header.
  6. Close the category again and proceed with the next category

Create a new data set series

Create new dataset series

A dataset series is a collection of datasets sharing the same product specification [ISO 19115]. All files in a dataset series have the same schema and the same spatial coordinate reference system. It is also possible to create derived dataset series by means of transformation projects. A dataset series has its own INSPIRE, ISO or national metadata.

hale»connect supports the creation of dataset series and enables customers to upload multiple datasets easily using batch upload.

Dataset series are enabled on a per organisation basis. Please contact us at info@wetransform.to for more information about dataset series.

A range of file formats can be used to create a dataset series. A dataset series may contain:

Vector data

  • Supported vector data formats include *.gml and *.shp
  • Multiple *.shp files can be uploaded to create a dataset
  • Currently the upload of one *.gml file per dataset is supported
    • INSPIRE, 3A, CityGML, XPlanung and ISYBAU are fully supported

Non-spatial data

  • Supported data formats include *.xml

Raster data

  • Supported formats include *.png, and *.GeoTIFF
  • Currently the upload of one raster data resource per dataset is supported
  • A separate raster layer is published in the view and download services

Attachments

  • Supported file attachment formats include *.pdf, *.png, *.GeoTIFF, or even textures for 3D data
  • There are multiple ways to upload and associate attachments to your dataset
  • Attachments uploaded during dataset creation are added to the attachments section automatically
  • The attachment files uploaded to hale»connect have a publicly available link to the attachment file. The links to attachment files can be included in your Predefined dataset download service. The links to attachment files and external URLs can also be viewed in the Feature Explorer Tool.
  • Separate service layers are not created for attachment files
  • Visit our help page for more information about working with attachments

A combination of these data resources

  • Datasets can include *.gml, raster and attachment data resources. View and download services for vector and raster data are published as separate layers.
  • Currently the upload of *.shp with combined data resources is not supported

Creating a new dataset series

Follow this workflow to publish a dataset series, logged in as a theme manager or superuser:

  1. Create a schema based on your source data and toggle the switch to create a theme.
  2. Navigate to your newly created theme, and click on the dataset series section.
  3. In the dataset series section of your theme, toggle the switch to Publish aggregate services for all datasets using this theme.
  4. If you plan to add data to your series using batch upload, click on the metadata section in the theme and add a dataset name pattern using the autofill assistant to provide your datasets with pre-configured names. This step is optional and recommended for batch upload if you need a special naming pattern for your datasets.

Once the theme is configured, any user belonging to your organisation or a sub-organisation of your organisation can create and publish a dataset series.

  1. Go to «Data» on the main toolbar
  2. Go to «Create new dataset» at the bottom of the dataset list
  3. Select a series enabled theme to link to your dataset. The option to use batch upload appears when a series enabled theme is selected.
  4. Users have the option to create a dataset series that contains a single dataset, or a dataset series that contains any number of datasets. Dataset series are automatically named using the following pattern: User organisaton (Theme name).

    Dataset series with a single dataset:

    • Select the data sources for the individual dataset. There are a few ways to upload the files belonging to your dataset:
      • Click «Select file(s)» and navigate to the desired dataset.
      • Click «Enter URL(s)» and navigate to the desired URL or WFS.
      • Drag a dataset to «Drop file(s) here»
    • Name your dataset and enter any required metadata elements. Metadata entry is determined by the metadata configuration in the theme associated with your dataset.

    • When a dataset series using individual files is created, the dataset series, the dataset belonging to the series and the dataset series metadata and dataset metadata are published. In the final step of dataset series creation, toggle Create services to publish dataset series services. Navigate to the dataset contained in your series to publish the individual dataset’s services.

    • If your theme contains a transformation configuration, you can use the toggles in the final step to execute the transformation and create services for a series enabled target theme.

    Dataset series using batch upload:

    • Toggle the switch to use batch upload. There is only one way to upload files using batch upload. Creating a dataset series using batch upload requires the upload of a zip that directly contains the datasets as zips.batch upload

    • The dataset name and metadata steps are not available for dataset series creation using batch upload. We recommend configuring a dataset name pattern autofill rule in your theme to automatically name datasets belonging to a series. You can edit the metadata on a per dataset basis after series creation.

    • The mass publishing of services for dataset series using batch upload is not currently supported. When a dataset series using batch upload is created, the dataset series, the datasets belonging to the series and the dataset series metadata and dataset metadata are published. You can navigate to the resources to publish services after series creation.

    • The final step in the data series using batch upload workflow displays a list of the datasets added to the series.

  5. Dataset series can be viewed in the dataset resource list. Dataset series are identified by a ‘Series’ label. Click on the chevron next to the ‘Series’ label to expand the series and view the individual dataset resources which belong to the series. Visit the Viewing datasets in the resource list documentation page for more information about working with the resource list and optimizing the display of fields to best suit your needs.

  6. Click on the dataset series to navigate to the resource. Dataset series are organized similarly to datasets. The overview panel of a dataset series provides a summary overview of the resource including ownership information and a status panel which displays the system status of the series metadata and services. Users can edit series metadata in the metadata section, and series services in the view and download services sections. The dataset series section contains a list of all datasets which belong to a series. Every dataset that is published using a series enabled theme is added to the same dataset series.

  7. To delete a dataset series, navigate to the series and click «Delete». The dataset series and all sub-datasets are deleted.

Publish services

Publish Services for a data set

You can publish view and download services if the theme linked to your data set has a valid service configuration. Publishing services is done either by triggering the automated publishing process at the end of the data set creation, or through a manual command later on. When you publish services, the system always creates all configured services types at once; in other words, if your theme configuration includes a view- and a download service, it is not possible to publish just a download service. The reason for this is that the metadata of each has references to the other.

When you’re logged in as a Data Manager or Theme manager, follow these steps to publish services manually:

  1. Go to «Data» on the main toolbar
  2. Pick the data set from the resource list for which you’d like to publish services
  3. Go to «View Services» or «Download Services»
  4. Activate the toggle switch
  5. Click «Publish services» to create and start the service, and also run any configured test suites
    • This process runs in the background and can take a while. A progress bar displaying the status of service creation, service testing and service publishing appears.
  6. As each process completes, additional information will become available:
    • Links to service and data set metadata
    • For view services, you’ll also get a preview link
    • Test reports
    • Service settings
    • Service log
  7. To unpublish a service, deactivate the toggle switch and click «Delete services».
Please note the following limitations of the current version:
  • Display properties configured for the theme are not used for the GetFeatureInfo in the WMS.
  • The ATOM Feed is not completely INSPIRE compliant, since elements for GeoRSS and Open Search are missing.