Data Toolkit

We highly recommend understanding Main Concepts first. (3 min read)

Set of optional data components that you can use in various stages of your data pipeline, among other things, to transform raw Data or feed transformed data into a Data model. These components are supposed to be integrated into your existing data pipeline stack, not to replace it. Usually, you will implement the data pipeline using your favourite data tools (SQL, Python, dbt, ...), and our components are here to help you with specific tasks.

Besides the main CleverMaps platform components, we also provide a set of single-purpose data components that can be deployed in different stages of a data pipeline, from data transformation to dashboard validations. Each component has a different purpose and its usage is optional. But it can make your data pipelines easier to set up. All components were designed mainly for automation because we believe that analytics-as-a-code is the best way how to build sustainable and reusable analytical solutions.

From a technical perspective, most of our data components are Docker images publicly available on Docker Hub. We believe that this is the most universal way of sharing the components. Docker images should be easily integrated into your existing data orchestration frameworks (e.g. Dagster, Prefect, AirFlow). Furthermore, we are gradually trying to implement some of our data components as native plugins in standards ELT/ETL platforms (e.g. AirByte, Fivetran, Stitch). Currently, our components are available in Keboola Connection.

We can divide our data components into these categories: Components around the CleverMaps Semantic Layer and Components around the CleverMaps Semantic Layer.

Components around the CleverMaps Semantic Layer

They help you to:

Automate data flows into (Writer) or out of the Semantic Layer (Extractor). Typically you will use the Writer to load data from your Data Warehouse into the CleverMaps Semantic Layer. Optionally you can use Extractor to extract the data and metrics from the Semantic Layer and use them in other systems.

Deploy an automated pipeline that can validate your map dashboards (Project Validator). This can be useful to ensure that nothing was broken in the data or configuration during the updates.

Deploy any custom logic on top of the Semantic Layer (Python SDK), for example, to directly ingest data or metrics in your custom data app.

General data processing components

They help you with the following:

Specific geospatial data transformation (AreaMapper or SpatialUtils).

Create or update vector tiles in Mapbox (MapboxUtils). This is a more CleverMaps-specific task because we use Mapbox as the engine for the map visualizations.