Tutorial 8: AreaMapper and grid visualization

In this tutorial, we'll learn how to use grid visualization with different indicator scales. Currently, there are two ways of using a grid in CleverMaps:

• connecting your geometryPoint dataset using h3Grid datasets with grid geometries generated on the fly

• mapping your geometryPoint dataset using AreaMapper and importing a grid data dimension with vector tiles

This tutorial covers both cases. It's up to you if you want to learn only one of them or both.

What you'll create

We will add a grid visualization and some more indicator scales.

grid visualization, heatmap scale	grid visualization, magma scale
grid visualization, positive scale	grid visualization, negative scale

Using H3 grid

CleverMaps natively supports visualization using the H3 grid spatial index developed by Uber. This hierarchical spatial index consists of several resolutions of grid cells which cover the whole Earth, and are related to each other (similar to lower and higher administrative units). The differences between this approach and using AreaMapper and data dimension are:

• The geometries are not in vector tile format, but are generated on the fly by the application frontend

• There is no need for pre-computed DWH datasets in the project - H3 grids have generated cell names

• The grid can be used for any location on Earth (in any resolution)

The usage is also much simpler - to use the H3 grid visualization we need to:

1. add some h3Grid datasets (each for one resolution)

2. add these datasets to the ref.h3Geometries array of the geometryPoint dataset we want to visualize

So, let's add these 4 h3Grid datasets:

{
    "name": "h3_grid_6",
    "type": "dataset",
    "title": "Cell edge 3.2 km (36 km2)",
    "ref": {
        "type": "h3Grid",
        "resolution": 6,
        "zoom": {
            "min": 2,
            "optimal": 8,
            "max": 18
        }
    }
}

{
    "name": "h3_grid_7",
    "type": "dataset",
    "title": "Cell edge 1.2 km (5 km2)",
    "ref": {
        "type": "h3Grid",
        "resolution": 7,
        "zoom": {
            "min": 2,
            "optimal": 9,
            "max": 18
        }
    }
}

{
    "name": "h3_grid_8",
    "type": "dataset",
    "title": "Cell edge 460 m (0.7 km2)",
    "ref": {
        "type": "h3Grid",
        "resolution": 8,
        "zoom": {
            "min": 2,
            "optimal": 10,
            "max": 18
        }
    }
}

{
    "name": "h3_grid_9",
    "type": "dataset",
    "title": "Cell edge 170 m (0.1 km2)",
    "ref": {
        "type": "h3Grid",
        "resolution": 9,
        "zoom": {
            "min": 2,
            "optimal": 11,
            "max": 18
        }
    }
}

Use addMetadata to add these datasets. Now add the h3Geometries reference to the customers dataset:

Customers dataset excerpt

   ...
        "ref": {
            "type": "dwh",
            "subtype": "geometryPoint",
            "h3Geometries": [
                "h3_grid_6",
                "h3_grid_7",
                "h3_grid_8",
                "h3_grid_9"
            ],
            "visualizations": [
                {
                    "type": "dotmap"
                },
   ...

And use pushProject to push the changes. You can review the references to the h3Grid datasets in the data model.

Now you can skip right to Tutorial 8: AreaMapper and grid visualization#Grid visualization, or explore the second way - AreaMapper and data dimension.

Advanced Tips for H3 grids

1. Is H3 grid insufficient in performance? The use case described here is a basic one. It's possible to use dataset with materialized H3 grid ID columns which makes the responses up to 2-3x times faster.

2. If you need connect multiple fact tables, it is necessary to use materialized H3 grid dimension with materialized DWH datasets. In the example bellow, the dataset cz_uc4_grid_res9_dwh can be connected with multiple fact datasets from the left side:

Contact us at support@clevermaps.io for more information about H3 grids.

Using AreaMapper and data dimension

Running AreaMapper

CleverMaps AreaMapper is one of our data preparation tools. AreaMapper computes geometric intersection between points and polygons (areas, grids) and then adds IDs of polygons to the points table if an intersection exists.

It can be run in Docker or at Keboola. We will use Docker because not everyone has a Keboola account and it can be run locally.

You can get Docker here. Select Docker Desktop and download the installation file according to your platform.

Then use the following command to get the AreaMapper image:

docker pull clevermaps/areamapper:latest

We will create an empty directory on our local drive (e.g. /home/user/CleverMaps/AreaMapper) and copy the customers.csv file from the first tutorial into the folder.

Now, we will create a configuration file for AreaMapper. There are two possible ways of running AreaMapper:

• using polygons CSV on your local drive

• using polygons CSV prepared and hosted by us

The difference is - when using local CSV, you can use any CSV you want. When using the hosted CSV, the file has to be hosted by us first. For more info, contact us at info@clevermaps.io. The current list of hosted dimensions can be found here.

The CSV we will use in this tutorial is already hosted. In case you want to try the "local" way, or just have a look at the CSV, you can download it here.

{
    "input_points": {
        "filename": "/workdir/customers.csv",
        "delimiter": ",",
        "geom_fieldname_x": "lng",
        "geom_fieldname_y": "lat",
        "join_fieldname": ""
    },
    "input_areas": {
        "filetype": "candim",
        "filename": "can-dim-grid-uber-9-cz"
    },
    "output": {
        "filename": "/workdir/customers_mapped.csv",
        "delimiter": ",",
        "not_matched_value": ""
    },
    "temp_directory": "/workdir"
}

{
    "input_points": {
        "filename": "/workdir/customers.csv",
        "delimiter": ",",
        "geom_fieldname_x": "lng",
        "geom_fieldname_y": "lat",
        "join_fieldname": ""
    },
    "input_areas": {
        "filetype": "csv",
        "filename": "/workdir/cz_grid_res9_wkt.csv",
        "geom_fieldname": "wkt",
        "primary_key_fieldname": "hex_id",
        "delimiter": ",",
        "join_fieldname": ""
    },
    "output": {
        "filename": "/workdir/customers_mapped.csv",
        "delimiter": ",",
        "not_matched_value": ""
    },
    "temp_directory": "/workdir"
}

We will run AreaMapper using the following command. Please note that you have to modify the paths according to your system.

user@host:/home/user/CleverMaps/AreaMapper$ sudo docker run -t -v /home/user/CleverMaps/AreaMapper:/workdir clevermaps/areamapper:latest --config /workdir/config-hosted.json

Importing /workdir/customers.csv to temp database...
Importing /workdir/cz_grid_res9_wkt.csv to temp database...
Creating geometry on points table...
Creating geometry on polygons table...
Creating spatial index on points table...
Creating spatial index on polygon table...
Calculating intersection...
Area mapping was successfully finished.

Copy the output file you specified in the JSON config (customers_mapped.csv) to your project dump /data folder and rename it to customers.csv to overwrite the older customers CSV.

Importing the grid dimension

Let's import the grid data dimension using the importProject command.

tomas.schmidl@secure.clevermaps.io/project:k5t8mf2a80tay2ng/dump:2020-06-23_17-48-00$ importProject --project nbutdscdghhg25v4 --datasets
Importing project nbutdscdghhg25v4...

Here, we omit a significant amount of the importProject command output for the sake of readability.

Import finished!
Source project: nbutdscdghhg25v4 (can-dim-grid-cz-uber)
Destination project: k5t8mf2a80tay2ng (First project)
Imported datasets: 9
Imported metadata objects: 0
Imported CSV files: 4

To view detailed imported content, use status command.
Now you can make changes, or import another project.
When you're done, use addMetadata command to add the metadata to the project.
To push the data to the project, use pushProject command.

Add the imported files using addMetadata and pushProject.

Now we have to edit the customers dataset. AreaMapper added one column to the end of the CSV file - hex_id. We have to add this property to the end of our dataset. Its foreign key will point to the lowest grid dataset - cz_grid_res9_dwh.

Customers dataset excerpt

  ...
        {
            "name": "lng",
            "title": "Address longitude",
            "column": "lng",
            "type": "longitude",
            "filterable": false
        },
        {
            "foreignKey": "cz_grid_res9_dwh",
            "name": "hex_id",
            "title": "Hex grid ID",
            "column": "hex_id",
            "type": "string",
            "filterable": false
        }
    ],
  ...

Take a look at the data model to see that the customers and cz_grid_res9_dwh are joined. 

Grid visualization

Enter the Business overview view, and select the new granularity from the granularity drop-down menu.

Now, we have a new visualization available - a grid.

Alternatively, select one of the lower levels to get to a bigger detail.

Indicator scales

Let's take a look at more indicator scales. So far we've used only the standard (blue) scale. 

To try some of them, set the following scales for these indicators:

• for online_turnover_indicator, set content.scale to "heatmap"

• for offline_turnover_indicator, set content.scale to "magma"

• for online_ratio_indicator, set content.scale to "positive"

• for offline_ratio_indicator, set content.scale to "negative"

Use pushProject to push the indicators and enter the view. Select the grid visualization, select one of the lower levels (e.g. "Cell edge 460 m") and click "Visualize" on the "Online channel turnover" indicator.

But indicators offer way more scales. Feel free to explore them and use them in your projects! 

 That's it! The last part of CleverMaps tutorial is a bit more technical and focused on Shell project management.