forked from MapComplete/MapComplete
123 lines
3.5 KiB
Markdown
123 lines
3.5 KiB
Markdown
# Setting up a synced OSM-server for quick layer access
|
|
|
|
## Requirements
|
|
|
|
Debian machine (e.g. Ubuntu)
|
|
|
|
`apt install osmium-tool docker.io osm2pgsql`
|
|
|
|
## Tuning the database
|
|
|
|
|
|
|
|
## Setting up the SQL-server (only once):
|
|
|
|
|
|
For performance, docker needs a specific config file. See https://osm2pgsql.org/doc/manual.html#tuning-the-postgresql-server
|
|
|
|
A suggested config file is:
|
|
|
|
(On HDD, 180GB RAM)
|
|
|
|
```
|
|
max_connections = 5000 # needed for OSM2PGSQL
|
|
shared_buffers = 2GB
|
|
work_mem = 200MB
|
|
maintenance_work_mem = 15GB
|
|
autovacuum_work_mem = 4GB
|
|
wal_level = minimal
|
|
checkpoint_timeout = 120min
|
|
max_wal_size = 20GB
|
|
checkpoint_completion_target = 0.9
|
|
max_wal_senders = 0
|
|
random_page_cost = 8.0
|
|
```
|
|
Save this config file somewhere as "postgresql.conf"
|
|
|
|
- Start the docker with: `docker run --name osm-cache-db -e POSTGRES_PASSWORD=password -e POSTGRES_USER=user -d -p 5444:5432 -v ~/data/pgsql/:/var/lib/postgresql/data -v postgresql.conf:/var/lib/postgresql/data/postgresql.conf postgis/postgis`
|
|
- Now, change the config file. This can easily be done as the config file is mirrored to the host system at:
|
|
`~/data/pgsql/postgresql.conf`. Append this file with configuration above, e.g. with `cat posgresql.conf >> ~/data/pgsql/postgresql.conf`
|
|
- Restart the db: `docker restart osm-cache-db`
|
|
|
|
# Seeding the data
|
|
|
|
> The following steps are also automated in .forgejo/workflows/update_database.yml
|
|
|
|
## Importing data
|
|
|
|
Install osm2pgsql (hint: compile from source is painless)
|
|
|
|
Download the latest with:
|
|
`nohup transmission-cli https://planet.osm.org/pbf/planet-latest.osm.pbf.torrent &>nohup_transmission.log &`
|
|
which will download the data to `~/Downloads`
|
|
|
|
To seed the database:
|
|
````
|
|
nohup osm2pgsql -O flex -S build_db.lua -s --flat-nodes=import-help-file -d postgresql://user:password@localhost:5444/osm-poi <file>.osm.pbf >> seeddb.log
|
|
|
|
# To see the progress
|
|
tail -f seeddb.log
|
|
````
|
|
Storing properties to table '"public"."osm2pgsql_properties" takes about 25 minutes with planet.osm
|
|
|
|
Belgium (~555mb) takes 15m
|
|
World (80GB) should take 15m*160 = 2400m = 40hr
|
|
|
|
73G Jan 23 00:22 planet-240115.osm.pbf: 2024-02-10 16:45:11 osm2pgsql took 871615s (242h 6m 55s; 10 days) overall on lain.local with RAID5 on 4 HDD disks, database is over 1Terrabyte (!)
|
|
|
|
Server specs
|
|
|
|
Lenovo thinkserver RD350, Intel Xeon E5-2600, 2Rx4 PC3
|
|
11 watt powered off, 73 watt idle, ~100 watt when importing
|
|
|
|
HP ProLiant DL360 G7 (1U): 2Rx4 DDR3-memory (PC3)
|
|
Intel Xeon X56**
|
|
|
|
|
|
## Updating data
|
|
|
|
|
|
`osm2pgsql-replication update -d postgresql://user:password@localhost:5444/osm-poi -- -O flex -S build_db.lua -s --flat-nodes=import-help-file`
|
|
|
|
|
|
## Deploying a tile server
|
|
|
|
pg_tileserv can be downloaded here: https://github.com/CrunchyData/pg_tileserv
|
|
|
|
In the directory where it is downloaded (e.g. `~/data`), run
|
|
|
|
First, look up the latest suitable database on https://mapcomplete.org/status
|
|
|
|
````
|
|
export DATABASE_URL=postgresql://user:password@localhost:5444/osm-poi.{date-of-suitable-database}
|
|
nohup ./pg_tileserv >> pg_tileserv.log &
|
|
````
|
|
|
|
Tiles are available at:
|
|
````
|
|
map.addSource("drinking_water", {
|
|
"type": "vector",
|
|
"tiles": ["http://127.0.0.2:7800/public.drinking_water/{z}/{x}/{y}.pbf"] // http://127.0.0.2:7800/public.drinking_water.json",
|
|
})
|
|
````
|
|
|
|
# Starting the summary server
|
|
|
|
`npm run summary-server` in the git repo
|
|
|
|
# Connecting to the database
|
|
|
|
Setup the SSH-tunnel:
|
|
|
|
`ssh folky.me
|
|
ssh -L 5444:127.0.0.1:5444 lain`
|
|
|
|
Open a second terminal:
|
|
|
|
`ssh -L 5444:127.0.0.1:5444 folky.me` `
|
|
|
|
Connect pgAdmin
|
|
|
|
# Rebooting:
|
|
|
|
-> Restart the docker container
|