Merge branch 'master' into master

Docs/Making_Your_Own_Theme.md

@@ -78,6 +78,7 @@ Every field is documented in the source code itself - you can find them here:
 - [The top level `LayoutConfig`](https://github.com/pietervdvn/MapComplete/blob/master/Customizations/JSON/LayoutConfigJson.ts)
 - [A layer object `LayerConfig`](https://github.com/pietervdvn/MapComplete/blob/master/Customizations/JSON/LayerConfigJson.ts)
 - [The `TagRendering`](https://github.com/pietervdvn/MapComplete/blob/master/Customizations/JSON/TagRenderingConfigJson.ts)
+- At last, the exact semantics of tags is documented [here](Docs/Tags_format.md)
 
 ## Some pitfalls
 

Docs/SpecialRenderings.md

@@ -0,0 +1,61 @@
+### Special tag renderings
+
+In a tagrendering, some special values are substituted by an advanced UI-element. This allows advanced features and visualizations to be reused by custom themes or even to query third-party API's.General usage is **{func\_name()}** or **{func\_name(arg, someotherarg)}**. Note that you _do not_ need to use quotes around your arguments, the comma is enough to seperate them. This also implies you cannot use a comma in your args
+
+### all\_tags
+
+Prints all key-value pairs of the object - used for debugging
+
+**Example usage:** {all\_tags()}
+
+### image\_carousel
+
+Creates an image carousel for the given sources. An attempt will be made to guess what source is used. Supported: Wikidata identifiers, Wikipedia pages, Wikimedia categories, IMGUR (with attribution, direct links)
+
+1.  **image key/prefix**: The keys given to the images, e.g. if image is given, the first picture URL will be added as image, the second as image:0, the third as image:1, etc... Default: image
+2.  **smart search**: Also include images given via 'Wikidata', 'wikimedia\_commons' and 'mapillary Default: true
+
+**Example usage:** {image\_carousel(image,true)}
+
+### image\_upload
+
+Creates a button where a user can upload an image to IMGUR
+
+1.  **image-key**: Image tag to add the URL to (or image-tag:0, image-tag:1 when multiple images are added) Default: image
+
+**Example usage:** {image\_upload(image)}
+
+### reviews
+
+Adds an overview of the mangrove-reviews of this object. Mangrove.Reviews needs - in order to identify the reviewed object - a coordinate and a name. By default, the name of the object is given, but this can be overwritten
+
+1.  **subjectKey**: The key to use to determine the subject. If specified, the subject will be **tags\[subjectKey\]** Default: name
+2.  **fallback**: The identifier to use, if _tags\[subjectKey\]_ as specified above is not available. This is effectively a fallback value
+
+**Example usage:** **{reviews()} **for a vanilla review, **{reviews(name, play\_forest)}** to review a play forest. If a name is known, the name will be used as identifier, otherwise 'play\_forest' is used****
+
+### ****opening\_hours\_table****
+
+****Creates an opening-hours table. Usage: {opening\_hours\_table(opening\_hours)} to create a table of the tag 'opening\_hours'.
+
+1.  **key**: The tagkey from which the table is constructed. Default: opening\_hours
+
+**Example usage:** {opening\_hours\_table(opening\_hours)}
+
+### live
+
+Downloads a JSON from the given URL, e.g. '{live(example.org/data.json, shorthand:x.y.z, other:a.b.c, shorthand)}' will download the given file, will create an object {shorthand: json\[x\]\[y\]\[z\], other: json\[a\]\[b\]\[c\] out of it and will return 'other' or 'json\[a\]\[b\]\[c\]. This is made to use in combination with tags, e.g. {live({url}, {url:format}, needed\_value)}
+
+1.  **Url**: The URL to load
+2.  **Shorthands**: A list of shorthands, of the format 'shorthandname:path.path.path'. Seperated by ;
+3.  **path**: The path (or shorthand) that should be returned
+
+**Example usage:** {live({url},{url:format},hour)} {live(https://data.mobility.brussels/bike/api/counts/?request=live&featureID=CB2105,hour:data.hour\_cnt;day:data.day\_cnt;year:data.year\_cnt,hour)}
+
+### share\_link
+
+Creates a link that (attempts to) open the native 'share'-screen
+
+1.  **url**: The url to share (default: current URL)
+
+**Example usage:** {share\_link()} to share the current page, {share\_link()} to share the given url****

Docs/Stats.md

@@ -0,0 +1,31 @@
+Statistics
+==========
+
+There are some fancy statistics available about MapComplete use. The most important once are listed below, some more graphs (and the scripts to generate them) are [in the tools directory](Tools/) 
+
+All Time usage
+--------------
+
+![](Tools/CumulativeContributors.png)
+![](Tools/Cumulative%20changesets%20per%20contributor.png)
+
+Note: in 2020, MapComplete would still make one changeset per answered question. This heavily skews the below graphs towards `buurtnatuur` and `cyclofìx`, two heavily used themes at the beginning.
+
+![](Tools/Cumulative%20changesets%20per%20theme.png)
+![](Tools/Theme%20distribution.png)
+
+2020
+----
+
+![](Tools/CumulativeContributors%20in%202020.png)
+![](Tools/Cumulative%20changesets%20per%20contributor%20in%202020.png)
+![](Tools/Cumulative%20changesets%20per%20theme%20in%202020.png)
+![](Tools/Theme%20distribution%20in%202020.png)
+
+2021
+----
+
+![](Tools/CumulativeContributors%20in%202021.png)
+![](Tools/Cumulative%20changesets%20per%20contributor%20in%202021.png)
+![](Tools/Cumulative%20changesets%20per%20theme%20in%202021.png)
+![](Tools/Theme%20distribution%20in%202021.png)

Docs/Tags_format.md

@@ -0,0 +1,37 @@
+ Tags format
+=============
+
+When creating the `json` file describing your layer or theme, you'll have to add a few tags to describe what you want. This document gives an overview of what every expression means and how it behaves in edge cases.
+
+Strict equality
+---------------
+
+Strict equality is denoted by `key=value`. This key matches __only if__ the keypair is present exactly as stated.
+
+**Only normal tags (eventually in an `and`) can be used in places where they are uploaded**. Normal tags are used in the `mappings` of a [TagRendering] (unless `hideInAnswer` is specified), they are used in `addExtraTags` of [Freeform] and are used in the `tags`-list of a preset.
+
+If a different kind of tag specification is given, your theme will fail to parse.
+
+### If key is not present
+
+If you want to check if a key is not present, use `key=` (pronounce as *key is empty*). A tag collection will match this if `key` is missing or if `key` is a literal empty value.
+
+### Removing a key
+
+If a key should be deleted in the OpenStreetMap-database, specify `key=` as well. This can be used e.g. to remove a fixme or value from another mapping if another field is filled out.
+
+Strict not equals
+-----------------
+
+To check if a key does _not_ equal a certain value, use `key!=value`. This is converted behind the scenes to `key!~^value$`
+
+### If key is present
+
+This implies that, to check if a key is present, `key!=` can be used. This will only match if the key is present and not empty.
+
+Regex equals
+------------
+
+A tag can also be tested against a regex with `key~regex`. Note that this regex __must match__ the entire value. If the value is allowed to appear anywhere as substring, use `key~.*regex.*`
+
+Equivalently, `key!~regex` can be used if you _don't_ want to match the regex in order to appear.

Docs/Tools/compileStats.sh

@@ -0,0 +1,6 @@
+#! /bin/bash
+
+./fetchStats.sh
+./csvPerChange.sh
+python csvGrapher.py 
+

Docs/Tools/csvGrapher.py

@@ -0,0 +1,318 @@
+import csv
+from datetime import datetime
+
+from matplotlib import pyplot
+
+
+def counts(lst):
+    counts = {}
+    for v in lst:
+        if not v in counts:
+            counts[v] = 0
+        counts[v] += 1
+    return counts
+
+
+class Hist:
+
+    def __init__(self, firstcolumn):
+        self.key = "\"" + firstcolumn + "\""
+        self.dictionary = {}
+        self.key = ""
+
+    def add(self, key, value):
+        if not key in self.dictionary:
+            self.dictionary[key] = []
+        self.dictionary[key].append(value)
+
+    def values(self):
+        allV = []
+        for v in self.dictionary.values():
+            allV += list(set(v))
+        return list(set(allV))
+
+    def keys(self):
+        return self.dictionary.keys()
+
+    def get(self, key):
+        if key in self.dictionary:
+            return self.dictionary[key]
+        return None
+
+    # Returns (keys, values.map(f)). To be used with e.g. pyplot.plot
+    def map(self, f):
+        vals = []
+        keys = self.keys()
+        for key in keys:
+            vals.append(f(self.get(key)))
+        return vals
+
+    def mapcumul(self, f, add, zero):
+        vals = []
+        running_value = zero
+        keys = self.keys()
+        for key in keys:
+            v = f(self.get(key))
+            running_value = add(running_value, v)
+            vals.append(running_value)
+        return vals
+
+    def csv(self):
+        csv = self.key + "," + ",".join(self.values())
+        header = self.values()
+        for k in self.dictionary.keys():
+            csv += k
+            values = counts(self.dictionary[k])
+            for head in header:
+                if head in values:
+                    csv += "," + str(values[head])
+                else:
+                    csv += ",0"
+            csv += "\n"
+        return csv
+
+    def __str__(self):
+        return str(self.dictionary)
+
+
+def build_hist(stats, keyIndex, valueIndex):
+    hist = Hist("date")
+    c = 0
+    for row in stats:
+        c += 1
+        hist.add(row[keyIndex], row[valueIndex])
+    return hist
+
+
+def as_date(str):
+    return datetime.strptime(str, "%Y-%m-%d")
+
+
+def cumulative_users(stats):
+    users_hist = build_hist(stats, 0, 1)
+    all_users_per_day = users_hist.mapcumul(
+        lambda users: set(users),
+        lambda a, b: a.union(b),
+        set([])
+    )
+    cumul_uniq = list(map(len, all_users_per_day))
+    unique_per_day = users_hist.map(lambda users: len(set(users)))
+    new_users = [0]
+    for i in range(len(cumul_uniq) - 1):
+        new_users.append(cumul_uniq[i + 1] - cumul_uniq[i])
+    dates = map(as_date, users_hist.keys())
+    return list(dates), cumul_uniq, list(unique_per_day), list(new_users)
+
+
+def pyplot_init():
+    pyplot.figure(figsize=(14, 8), dpi=200)
+    pyplot.xticks(rotation='vertical')
+    pyplot.tight_layout()
+
+
+def create_usercount_graphs(stats, extra_text=""):
+    print("Creating usercount graphs " + extra_text)
+    dates, cumul_uniq, unique_per_day, new_users = cumulative_users(stats)
+    total = cumul_uniq[-1]
+
+    pyplot_init()
+    pyplot.fill_between(dates, unique_per_day, label='Unique contributors')
+    pyplot.fill_between(dates, new_users, label='First time contributor via MapComplete')
+    pyplot.legend()
+    pyplot.title("Unique contributors" + extra_text + ' with MapComplete (' + str(total) + ' contributors)')
+    pyplot.ylabel("Number of unique contributors")
+    pyplot.xlabel("Date")
+    pyplot.savefig("Contributors" + extra_text + ".png", dpi=400, facecolor='w', edgecolor='w', bbox_inches='tight')
+
+    pyplot_init()
+    pyplot.plot(dates, cumul_uniq, label='Cumulative unique contributors')
+    pyplot.legend()
+    pyplot.title("Cumulative unique contributors" + extra_text + " with MapComplete - " + str(total) + " contributors")
+    pyplot.ylabel("Number of unique contributors")
+    pyplot.xlabel("Date")
+    pyplot.savefig("CumulativeContributors" + extra_text + ".png", dpi=400, facecolor='w', edgecolor='w',
+                   bbox_inches='tight')
+
+
+def create_theme_breakdown(stats, fileExtra="", cutoff=5):
+    print("Creating theme breakdown " + fileExtra)
+    themeCounts = {}
+    for row in stats:
+        theme = row[3].lower()
+        if theme in theme_remappings:
+            theme = theme_remappings[theme]
+        if theme in themeCounts:
+            themeCounts[theme] += 1
+        else:
+            themeCounts[theme] = 1
+    themes = list(themeCounts.items())
+    if len(themes) == 0:
+        print("No entries found for theme breakdown (extra: " + str(fileExtra) + ")")
+        return
+    themes.sort(key=lambda kv: kv[1], reverse=True)
+    other_count = sum([theme[1] for theme in themes if theme[1] < cutoff])
+    themes_filtered = [theme for theme in themes if theme[1] >= cutoff]
+    keys = list(map(lambda kv: kv[0] + " (" + str(kv[1]) + ")", themes_filtered))
+    values = list(map(lambda kv: kv[1], themes_filtered))
+    total = sum(map(lambda kv: kv[1], themes))
+    first_pct = themes[0][1] / total;
+    if other_count > 0:
+        keys.append("other")
+        values.append(other_count)
+    pyplot_init()
+    pyplot.pie(values, labels=keys, startangle=(90 - 360 * first_pct / 2))
+    pyplot.title("MapComplete changes per theme" + fileExtra + " - " + str(total) + " total changes")
+    pyplot.savefig("Theme distribution" + fileExtra + ".png", dpi=400, facecolor='w', edgecolor='w',
+                   bbox_inches='tight')
+    return themes
+
+
+def cumulative_changes_per(contents, index, subject, filenameextra="", cutoff=5, cumulative=True, sort=True):
+    print("Creating graph about " + subject + filenameextra)
+    themes = Hist("date")
+    dates_per_theme = Hist("theme")
+    all_themes = set()
+    for row in contents:
+        th = row[index]
+        all_themes.add(th)
+        themes.add(as_date(row[0]), th)
+        dates_per_theme.add(th, row[0])
+    per_theme_count = list(zip(dates_per_theme.keys(), dates_per_theme.map(len)))
+    # PerThemeCount gives the most popular theme first
+    if sort == True:
+        per_theme_count.sort(key=lambda kv: kv[1], reverse=False)
+    elif sort is not None:
+        per_theme_count.sort(key=sort)
+    values_to_show = []  # (theme name, value to fill between - this is stacked, with the first layer to print last)
+    running_totals = None
+    other_total = 0
+    other_theme_count = 0
+    other_cumul = None
+
+    for kv in per_theme_count:
+        theme = kv[0]
+        total_for_this_theme = kv[1]
+        if cumulative:
+            edits_per_day_cumul = themes.mapcumul(
+                lambda themes_for_date: len([x for x in themes_for_date if theme == x]),
+                lambda a, b: a + b, 0)
+        else:
+            edits_per_day_cumul = themes.map(lambda themes_for_date: len([x for x in themes_for_date if theme == x]))
+
+        if (not cumulative) or (running_totals is None):
+            running_totals = edits_per_day_cumul
+        else:
+            running_totals = list(map(lambda ab: ab[0] + ab[1], zip(running_totals, edits_per_day_cumul)))
+
+        if total_for_this_theme >= cutoff:
+            values_to_show.append((theme, running_totals))
+        else:
+            other_total += total_for_this_theme
+            other_theme_count += 1
+            if other_cumul is None:
+                other_cumul = edits_per_day_cumul
+            else:
+                other_cumul = list(map(lambda ab: ab[0] + ab[1], zip(other_cumul, edits_per_day_cumul)))
+
+    keys = list(themes.keys())
+    values_to_show.reverse()
+    values_to_show.append(("other", other_cumul))
+    totals = dict(per_theme_count)
+    total = sum(totals.values())
+    totals["other"] = other_total
+
+    pyplot_init()
+    for kv in values_to_show:
+        if kv[1] is None:
+            continue  # No 'other' graph
+        msg = kv[0] + " (" + str(totals[kv[0]]) + ")"
+        if kv[0] == "other":
+            msg = str(other_theme_count) + " small " + subject + "s (" + str(other_total) + " changes)"
+        if cumulative:
+            pyplot.fill_between(keys, kv[1], label=msg)
+        else:
+            pyplot.plot(keys, kv[1], label=msg)
+
+    if cumulative:
+        cumulative_txt = "Cumulative changesets"
+    else:
+        cumulative_txt = "Changesets"
+    pyplot.title(cumulative_txt + " per " + subject + filenameextra + " (" + str(total) + " changesets)")
+    pyplot.legend(loc="upper left", ncol=3)
+    pyplot.savefig(cumulative_txt + " per " + subject + filenameextra + ".png")
+
+
+def contents_where(contents, index, starts_with, invert=False):
+    for row in contents:
+        if row[index].startswith(starts_with) is not invert:
+            yield row
+
+
+def create_graphs(contents):
+    create_usercount_graphs(contents)
+    create_theme_breakdown(contents)
+    cumulative_changes_per(contents, 3, "theme", cutoff=10)
+    cumulative_changes_per(contents, 3, "theme", cutoff=10, cumulative=False)
+    cumulative_changes_per(contents, 1, "contributor", cutoff=15)
+    cumulative_changes_per(contents, 2, "language", cutoff=1)
+    cumulative_changes_per(contents, 4, "version number", cutoff=1, sort=lambda kv : kv[0])
+    cumulative_changes_per(contents, 8, "host", cutoff=1)
+
+    currentYear = datetime.now().year
+    for year in range(2020, currentYear + 1):
+        contents_filtered = list(contents_where(contents, 0, str(year)))
+        extratext = " in " + str(year)
+        create_usercount_graphs(contents_filtered, extratext)
+        create_theme_breakdown(contents_filtered, extratext)
+        cumulative_changes_per(contents_filtered, 3, "theme", extratext, cutoff=5)
+        cumulative_changes_per(contents_filtered, 3, "theme", extratext, cutoff=5, cumulative=False)
+        cumulative_changes_per(contents_filtered, 1, "contributor", extratext, cutoff=10)
+        cumulative_changes_per(contents_filtered, 2, "language", extratext, cutoff=1)
+        cumulative_changes_per(contents_filtered, 4, "version number", extratext, cutoff=1, cumulative=False, sort=lambda kv : kv[0])
+        cumulative_changes_per(contents_filtered, 4, "version number", extratext, cutoff=1, sort=lambda kv : kv[0])
+        cumulative_changes_per(contents_filtered, 8, "host", extratext, cutoff=1)
+
+
+theme_remappings = {
+    "metamap": "maps",
+    "groen": "buurtnatuur",
+    "wiki:mapcomplete/fritures": "fritures",
+    "wiki:MapComplete/Fritures": "fritures",
+    "lits": "lit",
+    "pomp": "cyclofix",
+    "wiki:user:joost_schouppe/campersite": "campersite",
+    "wiki-user-joost_schouppe-geveltuintjes": "geveltuintjes",
+    "wiki-user-joost_schouppe-campersite": "campersites",
+    "wiki-User-joost_schouppe-campersite": "campersites",
+    "wiki-User-joost_schouppe-geveltuintjes": "geveltuintjes",
+    "wiki:User:joost_schouppe/campersite": "campersites",
+    "https://raw.githubusercontent.com/osmbe/play/master/mapcomplete/geveltuinen/geveltuinen.json": "geveltuintjes"
+}
+
+
+def clean_input(contents):
+    for row in contents:
+        theme = row[3].strip().strip("\"").lower()
+        if theme == "null":
+            # The theme metadata has only been set later on - we fetch this from the comment
+            i = row[7].rfind("#")
+            theme = row[7][i + 1:-1].lower()
+        if theme in theme_remappings:
+            theme = theme_remappings[theme]
+        row[3] = theme
+        row[4] = row[4].strip().strip("\"")[0:len("MapComplete x.x.x")]
+        yield [data.strip().strip("\"") for data in row]
+
+
+def main():
+    print("Creating graphs...")
+    with open('stats.csv', newline='') as csvfile:
+        stats = list(clean_input(csv.reader(csvfile, delimiter=',', quotechar='"')))
+        print("Found " + str(len(stats)) + " changesets")
+        create_graphs(stats)
+    print("All done!")
+
+
+# pyplot.fill_between(range(0,5),  [1,2,3,3,2],)
+# pyplot.show()
+main()

Docs/Tools/csvPerChange.sh

@@ -0,0 +1,21 @@
+#! /bin/bash
+
+if [[ ! -e stats.1.json ]]
+then
+    echo "No stats found - not compiling"
+    exit
+fi
+
+rm stats.csv
+# echo "date, username, language, theme, editor, creations, changes" > stats.csv
+echo "" > tmp.csv
+
+for f in stats.*.json
+do
+    echo $f
+    jq ".features[].properties | [.date, .user, .metadata.language, .metadata.theme, .editor, .create, .modify, .comment, .metadata.host]" "$f" | tr -d "\n" | sed "s/]\[/\n/g" | tr -d "][" >> tmp.csv
+    echo "" >> tmp.csv
+done
+
+sed "/^$/d" tmp.csv | sed "s/^  //" | sed "s/  / /g" | sed "s/\"\(....-..-..\)T........./\"\1/" | sort >> stats.csv
+rm tmp.csv

Docs/Tools/fetchStats.sh

@@ -0,0 +1,18 @@
+DATE=$(date +"%Y-%m-%d%%20%H%%3A%M")
+COUNTER=1
+if [[ $1 != "" ]]
+then
+    echo "Starting at $1"
+    COUNTER="$1"
+fi
+
+NEXT_URL=$(echo "https://osmcha.org/api/v1/changesets/?date__gte=2020-07-05&date__lte=$DATE&editor=mapcomplete&page=$COUNTER&page_size=1000")
+rm stats.*.json
+while [[ "$NEXT_URL" != "" ]]
+do
+    echo "$COUNTER '$NEXT_URL'"
+    curl "$NEXT_URL" -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: https://osmcha.org/?filters=%7B%22date__gte%22%3A%5B%7B%22label%22%3A%222020-07-05%22%2C%22value%22%3A%222020-07-05%22%7D%5D%2C%22editor%22%3A%5B%7B%22label%22%3A%22mapcomplete%22%2C%22value%22%3A%22mapcomplete%22%7D%5D%7D' -H 'Content-Type: application/json' -H 'Authorization: Token 6e422e2afedb79ef66573982012000281f03dc91' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'TE: Trailers' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -o stats.$COUNTER.json
+    
+    NEXT_URL=$(jq ".next" stats.$COUNTER.json | sed "s/\"//g")
+    let COUNTER++
+done;