MapComplete/latlon2country
4
0
Fork 0
Code Issues Pull requests Activity

Refactoring: use promises, now works with nodejs

Browse source
This commit is contained in:
Pieter Vander Vennet 2021-12-06 20:31:22 +01:00
parent ee6a747632
commit 3df2f45ea4
14 changed files with 4456 additions and 2554 deletions

50
README.md
View file

@ -1,8 +1,8 @@
LatLon2Country
==============
LatLon2Country
==============
LatLon2Country is a reverse geocoder, whose goal is to convert a location (latlon) into the country code in which the location lies in the browser.
LatLon2Country is a reverse geocoder, whose goal is to convert a location (latlon) into the country code in which the
location lies in the browser.
It is specifically designed to work with opening_hours.js and MapComplete and borrows ideas from codegrid.
@ -19,22 +19,25 @@ Run the code:
coder.CountryCodeFor(lon, lat)
`
If you want to selfhost the tiles (which is highly recommended), download [tiles.zip](tiles.zip), extract the zip to your server and point the constructor above to the correct path.
If you want to selfhost the tiles (which is highly recommended), download [tiles.zip](tiles.zip), extract the zip to
your server and point the constructor above to the correct path.
Base architecture
-----------------
The client side downloads tiles with information to determine the country of a point. These tiles are statically generated by the 'generate'-scripts and are used to determine the country.
Note that no country might be returned (e.g. the international waters), one country might be returned (most of the cases) or _multiple_ countries might be returned (contested territory). In the latter case, be _very_ careful which one to show!
The client side downloads tiles with information to determine the country of a point. These tiles are statically
generated by the 'generate'-scripts and are used to determine the country. Note that no country might be returned (e.g.
the international waters), one country might be returned (most of the cases) or _multiple_ countries might be returned (
contested territory). In the latter case, be _very_ careful which one to show!
The code is specifically designed to handle multiple calls efficiently (e.g.: if a 100 points close to each other are requested at the same moment, these should only cause the correct tiles to downloaded once).
The code is specifically designed to handle multiple calls efficiently (e.g.: if a 100 points close to each other are
requested at the same moment, these should only cause the correct tiles to downloaded once).
Tile format
-----------
The requested tiles all have the extension '.json'.
THey however break down into two types of tiles:
The requested tiles all have the extension '.json'. THey however break down into two types of tiles:
- Leaf tiles
- 'Go Deeper'-tiles
@ -43,32 +46,39 @@ THey however break down into two types of tiles:
A 'leaf'-tile is a tile with which the actual country can be determined without downloading more tiles.
If the result is uniform accross the tile (most commonly: the tile is completely within a single country), then the tile will consist of `["country_code"]`
If the result is uniform accross the tile (most commonly: the tile is completely within a single country), then the tile
will consist of `["country_code"]`
If a border goes trhough the tile, the tile will contain the actual geometries and the leaftile is a standard geojson file.
If a border goes trhough the tile, the tile will contain the actual geometries and the leaftile is a standard geojson
file.
The client library will enumerate _all_ the polygons and determine in which polygons the requested points lie. Multiple countries can be returned.
The client library will enumerate _all_ the polygons and determine in which polygons the requested points lie. Multiple
countries can be returned.
Note that the geometry-leaf-tiles have a fixed upper bound in size, in order to make processing fast (that is the entire point of this library)
Note that the geometry-leaf-tiles have a fixed upper bound in size, in order to make processing fast (that is the entire
point of this library)
### Go-Deeper-tiles
THe other are tiles which contain an overview of the next step to take, e.g.
`
z.x.y.json
[ "be", z + 1, z + 5, 0 ]
`
This tile is a single array, which contains the next actions to take for the upper left, upper right, bottom left and bottom right subtile respectively.
This tile is a single array, which contains the next actions to take for the upper left, upper right, bottom left and
bottom right subtile respectively.
The upper left tile falls completely within belgium (so no more action is needed). To know the country of upper right one, one should download the appropriate tile at zoomlevel (z + 1).
The bottom left tile is split multiple times, and the biggest subtiles there are way deeper, at 'z + 5' - so we skip all the intermediate layers and fetch the deeper tile immediately.
For the bottom right tile, nothing (0) is defined, meaning international waters.
The upper left tile falls completely within belgium (so no more action is needed). To know the country of upper right
one, one should download the appropriate tile at zoomlevel (z + 1). The bottom left tile is split multiple times, and
the biggest subtiles there are way deeper, at 'z + 5' - so we skip all the intermediate layers and fetch the deeper tile
immediately. For the bottom right tile, nothing (0) is defined, meaning international waters.
Note that in some edge cases, _multiple_ country codes are given, e.g.:
[ "RU;UA", ... ]
This would indicate disputed territory, e.g. Crimea which is disputed and claimed by both Russia and Ukraine. The client returns those in alphabetical order. It is the responsibility of the program using this code to determine the "correct" country.
This would indicate disputed territory, e.g. Crimea which is disputed and claimed by both Russia and Ukraine. The client
returns those in alphabetical order. It is the responsibility of the program using this code to determine the "correct"
country.

136
client/countryCoder.ts
View file

@ -1,136 +0,0 @@
import * as turf from "turf";
export default class CountryCoder {
public static runningFromConsole = false;
/* url --> ([callbacks to call] | result) */
private static readonly cache = {}
private readonly _host: string;
constructor(host: string) {
this._host = host;
}
lon2tile(lon: number, zoom: number): number {
return Math.floor((lon + 180) / 360 * Math.pow(2, zoom));
}
lat2tile(lat: number, zoom: number): number {
return Math.floor((1 - Math.log(Math.tan(lat * Math.PI / 180) + 1 / Math.cos(lat * Math.PI / 180)) / Math.PI) / 2 * Math.pow(2, zoom));
}
Fetch(z: number, x: number, y: number, callback: ((data: any) => void)): void {
const path = `${z}.${x}.${y}.json`;
// @ts-ignore
let cached: { callbacks: ((data: any) => void)[], data: any } = CountryCoder.cache[path];
if (cached !== undefined) {
if (cached.data !== null) {
// O, the data has already arrived!
callback(cached.data);
} else {
// We'll handle this later when the data is there
cached.callbacks.push(callback);
}
// Nothing more to do right now
return;
}
// No cache has been defined at this point -> we define the cache + callbacks store
cached = {
data: null,
callbacks: []
};
// @ts-ignore
CountryCoder.cache[path] = cached;
const url = this._host + "/" + path;
cached.callbacks.push(callback);
const xhr = new XMLHttpRequest();
xhr.open('GET', url, true);
xhr.responseType = 'json';
xhr.onload = function () {
var status = xhr.status;
if (status === 200) {
cached.data = xhr.response;
for (const callback of cached.callbacks) {
callback(xhr.response);
}
} else {
console.error("COULD NOT GET ", url)
}
};
xhr.send();
}
determineCountry(lon: number, lat: number, z: number, x: number, y: number, callback: ((countries: string[]) => void)): void {
this.Fetch(z, x, y, data => {
if(data === undefined){
throw `Got undefined for ${z}, ${x}, ${y}`;
}
if (data.length !== undefined) {
// This is an array
// If there is a single element, we have found our country
if (data.length === 1) {
callback(data)
return;
}
// The appropriate subtile is determined by zoom level + 1
const dx = this.lon2tile(lon, z + 1);
const dy = this.lat2tile(lat, z + 1);
// Determine the quadrant
// We determine the difference with what 'x' and 'y' would be for the upper left quadrant
const index = (dx - x * 2) + 2 * (dy - y * 2);
const state = data[index];
if (state === 0) {
// No country defined, probably international waters
callback([]);
}
const nextZoom = Number(state);
if (isNaN(nextZoom)) {
// We have found the country!
callback([state]);
return;
}
// Last case: the next zoom level is given:
this.determineCountry(lon, lat, nextZoom, this.lon2tile(lon, nextZoom), this.lat2tile(lat, nextZoom), callback)
} else {
// We have reached an actual leaf tile with actual geometries
const geojson = data;
const countries = [];
for (const feature of geojson.features) {
const inPolygon = turf.inside(turf.point([lon, lat]), feature);
if (inPolygon) {
const country = feature.properties.country;
countries.push(country)
}
}
countries.sort();
callback(countries);
}
})
}
CountryCodeFor(lon: number, lat: number, callback: ((countries: string[]) => void)): void {
// We wrap the callback into a try catch, in case something goes wrong
const safeCallback = (countries) => {
try {
callback(countries);
} catch (e) {
console.error("Latlon2country: the dev of this website made a call with CountryCodeFor, however, their callback failed with " + e)
}
}
this.determineCountry(lon, lat, 0, 0, 0, safeCallback);
}
}

2
generator/BuildMergedTiles.ts
View file

@ -51,7 +51,7 @@ export default class BuildMergedTiles extends Step {
// Nothing more to do, this is fine!
continue;
}
if (state === TileState.NOT_DEFINED) {
throw "This should not happen"
}

2
generator/DownloadBoundary.ts
View file

@ -12,7 +12,7 @@ export default class DownloadBoundary extends Step {
private static readonly servers = [
"https://nominatim.openstreetmap.org/search.php?",
"https://nominatim.geocoding.ai/search?",
"https://nominatim.geocoding.ai/search?",
]
private lastUsedServer = 0;

10
generator/GeoJsonSlicer.ts
View file

@ -54,7 +54,7 @@ export default class GeoJsonSlicer extends Step {
sum += container.length;
}
return sum;
}else if(geojson.type === "Point"){
} else if (geojson.type === "Point") {
return 0;
}
throw "Unknown type " + geojson.type;
@ -65,7 +65,7 @@ export default class GeoJsonSlicer extends Step {
const tileOverview = new TileOverview(this._countryName, this._countryName);
tileOverview.Add(0, 0, 0);
const zeroTile = {z: 0, x: 0, y: 0};
if(!fs.existsSync(tileOverview.GetPath( ))){
if (!fs.existsSync(tileOverview.GetPath())) {
fs.mkdirSync(tileOverview.GetPath(), {recursive: true});
}
fs.writeFileSync(tileOverview.GetPath(zeroTile), geoJsonString, {encoding: "utf8"});
@ -74,9 +74,9 @@ export default class GeoJsonSlicer extends Step {
while (queue.length > 0) {
const tile = queue.pop();
const geoJSON = tileOverview.GetGeoJson(tile);
if(this.Complexity(geoJSON) > this._maxComplexity){
const newTiles = tileOverview.BreakTile(tile);
queue.push(...newTiles);
if (this.Complexity(geoJSON) > this._maxComplexity) {
const newTiles = tileOverview.BreakTile(tile);
queue.push(...newTiles);
}
}

10
generator/TileOverview.ts
View file

@ -65,7 +65,7 @@ export default class TileOverview {
const x = xyz.x;
const y = xyz.y;
console.log("Breaking ",z,x,y, this._countryName);
console.log("Breaking ", z, x, y, this._countryName);
const status = this.DoesExist(z, x, y);
if (status !== TileState.EXISTS) {
throw "Attempting to split a tile which doesn't exist"
@ -80,8 +80,8 @@ export default class TileOverview {
function onNewTile(b: { z: number, x: number, y: number }) {
results.push(b);
}
this.Add(z,x,y, TileState.ZOOM_IN_MORE);
this.Add(z, x, y, TileState.ZOOM_IN_MORE);
this.IntersectAndWrite({z: z + 1, x: x * 2, y: y * 2}, geojson, onNewTile);
this.IntersectAndWrite({z: z + 1, x: x * 2 + 1, y: y * 2}, geojson, onNewTile);
@ -218,7 +218,7 @@ export default class TileOverview {
return;
}
// @ts-ignore
if (intersection.type === "Point") {
this.Add(z, x, y, TileState.NOT_DEFINED);
@ -240,7 +240,7 @@ export default class TileOverview {
return `${dir}${xyz.z}.${xyz.x}.${xyz.y}.geojson`;
}
DoesExistT(tileXYZ: { z: number; x: number; y: number }) : TileState {
DoesExistT(tileXYZ: { z: number; x: number; y: number }): TileState {
return this.DoesExist(tileXYZ.z, tileXYZ.x, tileXYZ.y);
}
}

13
generator/Utils.ts
View file

@ -5,13 +5,14 @@ export default class Utils {
public static Download(url: string, onSuccess: (data: string) => void, onFail?: (msg: string) => void,
format = "application/json") {
if(onFail === undefined){
if (onFail === undefined) {
onFail = console.error
}
const chunks = []
console.warn("Downloading "+url);
console.warn("Downloading " + url);
const req = https.request(url, {
headers: {"Accept": format, "User-agent":"latlon2country generator (Pietervdvn@posteo.net)"}}, function (res) {
headers: {"Accept": format, "User-agent": "latlon2country generator (Pietervdvn@posteo.net)"}
}, function (res) {
res.setEncoding('utf8');
res.on('data', data => {
chunks.push(data)
@ -20,11 +21,11 @@ export default class Utils {
req.on('error', function (e) {
onFail(e.message)
});
req.on("close",
req.on("close",
() => {
onSuccess(chunks.join(""))
onSuccess(chunks.join(""))
}
)
)
req.end();
}
}

14
index.ts
View file

@ -1,14 +0,0 @@
import CC from "./client/countryCoder"
export default class CountryCoder {
private readonly cc: CC;
constructor(host: string) {
this.cc = new CC(host);
}
public GetCountryCodeFor(lon: number, lat: number, callback: ((countries: string[]) => void)): void {
this.cc.CountryCodeFor(lon, lat, callback)
}
}

6533
package-lock.json generated
View file

File diff suppressed because it is too large Load diff

22
package.json
View file

@ -1,9 +1,16 @@
{
"name": "latlon2country",
"version": "1.1.2",
"version": "1.2.3",
"description": "Convert coordinates into the containing country",
"main": "index.ts",
"typings": "",
"main": "dist/index.js",
"typings": "dist/index.d.ts",
"files": [
"/dist",
"README.md",
"tiles.zip",
"generator",
"LICENSE"
],
"scripts": {
"build": "tsc -project .",
"publish": "npm run build && npm publish",
@ -21,16 +28,15 @@
"license": "GPL-3.0-or-later",
"dependencies": {
"@turf/boolean-point-in-polygon": "^6.0.1",
"@turf/turf": "^5.1.6",
"@types/node": "^14.14.10",
"jquery": "^3.5.1",
"ts-node": "^9.1.1",
"@turf/turf": "^6.3.0",
"turf": "^3.0.14"
},
"devDependencies": {
"@types/turf": "^3.5.32",
"@types/node": "^14.14.10",
"jquery": "^3.5.1",
"js-yaml": "^3.14.0",
"osmtogeojson": "^3.0.0-beta.4",
"ts-node": "^9.1.1",
"typescript": "^4.1.2"
}
}

163
src/CountryCoder.ts Normal file
View file

@ -0,0 +1,163 @@
import * as https from "https";
import * as turf from "turf";
export class CountryCoder {
/* url --> ([callbacks to call] | result) */
private static readonly cache: Map<string, Promise<any>> = new Map<string, Promise<any>>()
private readonly _host: string;
private static runningFromConsole = typeof window === "undefined";
constructor(host: string) {
this._host = host;
}
public async GetCountryCodeAsync(lon: number, lat: number): Promise<string[]> {
return this.GetCountryCodeForTile(lon, lat, 0, 0, 0);
}
public GetCountryCodeFor(lon: number, lat: number, callback: ((countries: string[]) => void)): void {
this.GetCountryCodeAsync(lon, lat).then(callback)
}
private static lon2tile(lon: number, zoom: number): number {
return Math.floor((lon + 180) / 360 * Math.pow(2, zoom));
}
private static lat2tile(lat: number, zoom: number): number {
return Math.floor((1 - Math.log(Math.tan(lat * Math.PI / 180) + 1 / Math.cos(lat * Math.PI / 180)) / Math.PI) / 2 * Math.pow(2, zoom));
}
private static FetchJsonNodeJS(url, headers?: any): Promise<any> {
return new Promise((resolve, reject) => {
try {
headers = headers ?? {}
headers.accept = "application/json"
console.log("ScriptUtils.DownloadJson(", url.substring(0, 40), url.length > 40 ? "..." : "", ")")
const urlObj = new URL(url)
https.get({
host: urlObj.host,
path: urlObj.pathname + urlObj.search,
port: urlObj.port,
headers: headers
}, (res) => {
const parts: string[] = []
res.setEncoding('utf8');
res.on('data', function (chunk) {
// @ts-ignore
parts.push(chunk)
});
res.addListener('end', function () {
const result = parts.join("")
try {
resolve(JSON.parse(result))
} catch (e) {
console.error("Could not parse the following as JSON:", result)
reject(e)
}
});
})
} catch (e) {
reject(e)
}
})
}
private static FetchJsonXhr(url): Promise<any> {
return new Promise((accept, reject) => {
const xhr = new XMLHttpRequest();
xhr.open('GET', url, true);
xhr.responseType = 'json';
xhr.onload = function () {
var status = xhr.status;
if (status === 200) {
accept(xhr.response)
} else {
console.error("COULD NOT GET ", url)
reject(url)
}
};
xhr.send();
})
}
private async Fetch(z: number, x: number, y: number): Promise<number[] | string[] | any> {
const path = `${z}.${x}.${y}.json`;
const url = this._host + "/" + path;
if (CountryCoder.runningFromConsole) {
return CountryCoder.FetchJsonNodeJS(url)
} else {
return CountryCoder.FetchJsonXhr(url)
}
}
private async FetchCached(z: number, x: number, y: number): Promise<number[] | string[] | any> {
const path = `${z}.${x}.${y}.json`;
let cached = CountryCoder.cache.get(path)
if (cached !== undefined) {
return cached
}
const promise = this.Fetch(z, x, y)
CountryCoder.cache.set(path, promise)
return promise;
}
private async GetCountryCodeForTile(lon: number, lat: number, z: number, x: number, y: number): Promise<string[]> {
const data = await this.FetchCached(z, x, y);
if (data === undefined) {
throw `Got undefined for ${z}, ${x}, ${y}`;
}
if (data.length === undefined) {
// We have reached an actual leaf tile with actual geometries
const geojson: any = data;
const countries = [];
for (const feature of geojson.features) {
const inPolygon = turf.inside(turf.point([lon, lat]), feature);
if (inPolygon) {
const country = feature.properties.country;
countries.push(country)
}
}
countries.sort();
return countries;
}
// This is an array either: either for numbers indicating the next zoom level to jump to or a list of country names
if (data.length === 1) {
// If there is a single element, we have found our country
return data;
}
// The appropriate subtile is determined by zoom level + 1
const dx = CountryCoder.lon2tile(lon, z + 1);
const dy = CountryCoder.lat2tile(lat, z + 1);
// Determine the quadrant
// We determine the difference with what 'x' and 'y' would be for the upper left quadrant
const index = (dx - x * 2) + 2 * (dy - y * 2);
const state = data[index];
if (state === 0) {
// No country defined, probably international waters
return []
}
const nextZoom = Number(state);
if (isNaN(nextZoom)) {
// We have found the country!
return [state];
}
// Last case: the next zoom level is given:
return this.GetCountryCodeForTile(lon, lat, nextZoom, CountryCoder.lon2tile(lon, nextZoom), CountryCoder.lat2tile(lat, nextZoom))
}
}

1
src/index.ts Normal file
View file

@ -0,0 +1 @@
export {CountryCoder} from './CountryCoder'

24
test.ts
View file

@ -1,27 +1,25 @@
import CountryCoder from "./client/countryCoder";
import exp = require("constants");
CountryCoder.runningFromConsole = true;
import {CountryCoder} from "./src/CountryCoder";
console.log("Testing...")
function pr(countries: string[]) {
console.log(">>>>>", countries.join(";"))
}
function expects(expected: string){
function expects(expected: string) {
return (countries) => {
if(countries.join(";") !== expected){
if (countries.join(";") !== expected) {
console.error("Unexpected country: got ", countries, "expected:", expected)
}else{
console.log("[OK] Got "+countries)
} else {
console.log("[OK] Got " + countries)
}
}
}
console.log("Hi world")
const coder = new CountryCoder("https://pietervdvn.github.io/latlon2country");
/*
coder.CountryCodeFor(3.2, 51.2, expects("BE"))
coder.CountryCodeFor(4.92119, 51.43995, expects("BE"))
coder.CountryCodeFor(4.93189, 51.43552, expects("NL"))
coder.CountryCodeFor(34.2581, 44.7536, expects("RU;UA"))//*/
coder.CountryCodeFor( -9.1330343,38.7351593, expects("PT"))
coder.GetCountryCodeFor(3.2, 51.2, expects("BE"))
coder.GetCountryCodeFor(4.92119, 51.43995, expects("BE"))
coder.GetCountryCodeFor(4.93189, 51.43552, expects("NL"))
coder.GetCountryCodeFor(34.2581, 44.7536, expects("RU;UA"))
coder.GetCountryCodeFor(-9.1330343, 38.7351593, expects("PT"))

30
tsconfig.json
View file

@ -1,19 +1,33 @@
{
"compilerOptions": {
"target": "ES5",
"module": "commonjs",
"target": "ES2015",
"module": "CommonJS",
"lib": [
"es2019",
"dom"
],
"declaration": true,
"outDir": "lib",
"rootDir": "client",
"types": ["node"],
"outDir": "./dist",
"rootDir": "src",
"types": [
"node"
],
"strict": false,
"esModuleInterop": true,
"resolveJsonModule": true
"resolveJsonModule": false,
"removeComments": true,
"noImplicitAny": false,
"preserveConstEnums": true,
"suppressImplicitAnyIndexErrors": true,
},
"include": ["client"],
"exclude": ["generator","data","lib","node_modules","tiles.zip"]
"include": [
"src/**/*"
],
"exclude": [
"generator",
"data",
"lib",
"node_modules",
"tiles.zip"
]
}