Demo Walkthrough
Scroll through the complete platform workflow — from citizen letter upload to geospatial analysis.
A municipality wants to know where to place new EV charging stations. Citizens can submit requests via handwritten letters, a web form, or a phone call. Each request is processed by AI to extract the address, geocoded to exact coordinates, stored in a multi-layer data warehouse, and visualized on interactive maps — all using fully sovereign, open-source tooling with zero cloud lock-in.
This demo uses real data from CBS (Statistics Netherlands) for the municipality of Zoetermeer, retrieved via their open API. The same approach can be applied to any Dutch municipality.
Letter Upload
A citizen uploads a handwritten or typed letter requesting an EV charging point through a simple drag-and-drop frontend. Metadata such as name, district, and address are filled in.
FastAPI Upload Portal
File Storage in RustFS
The uploaded letter file is stored in RustFS, a self-hosted S3-compatible object store. Files are organized by upload date and can be browsed through the RustFS web console. This ensures all raw documents are preserved alongside the platform with no external cloud dependency.
RustFS
File Conversion & LLM Extraction
If the upload is a PDF, it's first converted to PNG via a Python script. The image is then sent to Mistral's open-source vision LLM "Pixtral" hosted on Scaleway, which reads the handwritten text and extracts the address as structured JSON.
Pixtral (Scaleway)
Geocoding
The extracted address JSON is sent to a self-hosted Nominatim container — an open-source geocoder based on OpenStreetMap — which enriches it with precise longitude and latitude coordinates within the municipality.
Nominatim
Landing in ClickHouse
All results (letters, web and phone requests) land in ClickHouse following a multi-layer model: bronze (raw data + metadata), silver (cleaned and combined across channels), and gold (final analytics-ready tables with dimensions, facts, and marts).
ClickHouse
Prefect Orchestration
Prefect orchestrates the entire pipeline — from upload processing, LLM extraction, and geocoding to loading into ClickHouse and running data quality checks. A scheduled "poll pending letters" flow checks every few minutes for unprocessed letters.
Prefect
dbt Transformation & Quality
Data tests (postal code ranges, coordinate bounds, not-null checks) are defined in dbt. The dbt docs site visualizes the full lineage from bronze through silver to gold, showing table schemas and which tests apply to each model.
dbt
Authenticated REST API
A FastAPI application with Swagger documentation exposes endpoints for uploading letters, retrieving GIS results, and querying analytics. Access requires a time-limited API key generated through the admin portal, routed through the Tyk API gateway.
Tyk + FastAPI
Grafana Monitoring
Five Grafana dashboards monitor the platform: API request rates and latency, geocoding success rates, average LLM extraction time, token usage, confidence scores, upload volumes over time, and uptime checks every 20 seconds.
Grafana + Prometheus
GIS Services & Visualization
GeoServer publishes geospatial layers: CBS neighbourhood boundaries, existing charging points, requests per neighbourhood, and proximity analysis results. These can be downloaded in various formats or connected directly to ArcGIS or QGIS. GeoNode provides a portal for sharing these GIS resources.
GeoServer + GeoNode + PostGIS
Apache Hop (Optional ETL)
As a successor to Pentaho, Apache Hop offers a visual, point-and-click way to build additional pipelines on top of the bronze tables — especially useful for users familiar with Pentaho who prefer a graphical interface over code.
Apache Hop
The Complete Pipeline
Upload
Storage
LLM
Geocoding
ClickHouse
Prefect
dbt
API
Grafana
GIS
Hop