Proposal for a heatmap for Martin Metrics web [DRAFT]

[AbelVM]

Proposal for a heatmap for Martin Metrics web

The idea is to provide a heatmap of the areas that are hot for a given layer, while being compliant with any arbitrary privacy policy.

Starting point

• A hot area is defined not only by the number of visits but also because the users zoom deeper within those areas
• Let's say we have a (centralized) metrics table like this for a given layer

z	x	y	hits

• We want to display a heatmap that
  • represents the most visited areas for a given layer based on the tile hits table
  • is zoom-limited to an arbitrary value heatmap_max_zoom due to privacy concerns
  • we want to be able to provide the most detailed information
• The target layer have a max zoom layer_max_zoom as per its setup (typically layer_max_zoom >> heatmap_max_zoom)
• The heatmap user is at a given moment using a zoom level z' (z' < heatmap_max_zoom) when browsing our heatmap

Concerns

• As we're limiting the heatmap to heatmap_max_zoom, we're losing detail for zoom levels in the range [heatmap_max_zoom, layer_max_zoom]

Alternative Approach

First, for the sake of simplicity, let's use slippy tiles at the current zoom level, plus an integer delta, as the cells for our heatmap. So, our cells would be defined by [zº,x,y], being zº = z' + delta, and the heatmap layer can be easily built.

Instead of displaying the number of hits per tile, let's show the most common zoom level that users browse when visiting the area covered by the given tile.

1. Get the iterative list of subtiles for each [zº,x_n,y_n] tile in the view. Something like this JavaScript pseudocode:

const subtiles = tile => {
    const 
        z = tile[0] + 1
        x = tile[1],
        y = tile[2];
    return [
        [z, 2 * x, 2 * y],
        [z, 2 * x + 1, 2 * y],
        [z, 2 * x, 2 * y + 1],
        [z, 2 * x + 1 , 2 * y +1]
    ]
};
const descendants = (tile, layer_max_zoom) =>{
    cache = cache || {};
    const 
        tiles = [tile],
        hash = tile.join('_');
    if cache.hasOwnProperty(hash){
        return cache[hash]
    }else{
        for(let zi = ztile[0]; zi <= layer_max_zoom; zi++){
            const 
                ztiles = tiles.filter(t => t[0] == zi);
                subt = ztiles.map(t => subtiles(t)).flat();
            tiles.push(...subt);
        }
        cache[hash] = tiles;
        return tiles;
    }
}

const requestObject = tilesInViewport.map(t => {
    return {
        "tile": t,
        "subtiles": descendants(t, layer_max_zoom)
    }
});

2. Now we have an object like this for the current view and use it to request the database

[
    ...,
    {
        tile: [zº,x_n,y_n],
        subtiles: [
            [zº,x_n,y_n],
            ...
        ]
    },
    ...
]

3. Query the hits table for the given list of tiles, aggregating for zoom level, and then get the most common value. Something like:

with
input as(
    select
        t->'tile' as tile,
        json_array_elements(t->'subtiles') as subtile
    from json_array_elements({input_object}) as j(t)
)
tiles as(
    select
        i.tile,
        (i.tile->>0)::int as z,
        sum(t.hits) as hits
    from
        metrics_table t,
        input i
    where json_build_array(t.z, t.y, t.x) = i.subtile
    group by 1, 2
),
mostcommonz as(
    select
        tile,
        sum(hits) as hits,
        first_value(z) within group ( order by hits desc ) as zoom
    from tiles,
    group by 1
),
geometries as(
    select
        zoom,
        hits,
        st_tileenvelope(tile[0]::int,tile[1]::int,tile[2]::int) as geom
    from mostcommonz
)
select
    st_asgeojson(g.*)::json
from geometries g;

4. This GeoJSON is then rendered on our metrics map, using a bivariate styling in the line of:

• Color scale based on zoom value
• Opacity based on hitsvalue

6. That's it.

Final notes

The first half of the logic in this proposal runs on the client side, and the result is a GeoJSON that is recreated with every zoom event in the heatmap. This is obviously suboptimal, and it's been done that way for laziness' sake 😅 as I hate recursive CTEs logic

The right way to implement this would be to move the logic in step 1 to the server side, and store the results of step 1 + step 3 as a view that is served as any other table by Martin

[CommanderStorm]

I am a bit unsure if collecting more data over layer_max_zoom..heatmap_max_zoom is actually helping or how to visualise this.

=> Do you want to know that users in china zooming differently to europe?
=> Is a histogram for the zoom and one at heatmap_max_zoom for the geografic heatmap enough?

The advantage of the simpler approach is that we don't need to use so much more memory

[AbelVM]

My target is to get insights about where my users are most interested in.

• Not interested in users profiling, but places profiling. Per layer! That means insights based on the layer theme!
• You'd need a histogram per cell, a bit overkill.

And the interest is usually represented not only by the number of visits, but also that the higher the interest, the higher the zoom level.

But higher zoom levels are not significant if the number of hits in that area is low. And high hit numbers but with low zoom level are also not significant. That's why the proposed bivariate style

[CommanderStorm]

We should discuss what format to store these histograms:

• tile-indexes
• H3
• S2

Between these, H3 likely looks the nicest and has the least misleading artefacts (webmercator tiles cover different size in meters near 85° and the äquator).
We would need to do approximate maths which of them is affected by how much, though.
I am not quite sure how that math would look like.

https://h3geo.org/docs/core-library/restable/
The equivalent of Z7 (probaly a lower bound for privacy) would mean resolution 2 or 3
Resolution 0..=3 would have 41,162+5,882+842+122 = 48008 tiles.
At 4 Bytes for an u32 this would mean 200 KB per source

[AbelVM]

Being an H3 fan myself, I find that tile indexes are the best suited for this use case:

• Source data for tile hits metrics is already tile-based
• No need for reprojection. And we need speed, as the metrics are live data
• H3 is intended for trajectories, not coverage. And this is clearly a coverage case