Steve Bennett blogs

…about maps, open data, Git, and other tech.

Tag Archives: system administration

A TileMill server with all the trimmings

Recently, I set up a server for a series of #datahack workshops. We used TileMill to make creative maps with OpenStreetMap and other available data.

The major pieces required are:

  • TileMill, which comes with its own installer, and is totally self-sufficient: web application server, Mapnik, etc.
  • Postgres, the database which will hold the OSM data
  • PostGIS, the extension which allow Postgres to do that
  • Nginx, a reverse proxy, so we can have some basic security (TileMill comes with none)
  • OSM2PGSQL, a tool for loading OSM data into PostGIS

I’ve captured all those bits, and their configuration in this script. You’ll probably want to change the password – search for “htpasswd”.

The script below is out of date, contains errors, and is not maintained. Go to:

http://github.com/stevage/saltymill

# This script installs TileMill, PostGIS, nginx, and does some basic configuration.
# The set up it creates has basic security: port 20009 can only be accessed through port 80, which has password auth.

# The Postgres database tuning assumes 32 Gb RAM.

# Author: Steve Bennett

wget https://github.com/downloads/mapbox/tilemill/install-tilemill.tar.gz
tar -xzvf install-tilemill.tar.gz

sudo apt-get install -y policykit-1

#As per https://github.com/gravitystorm/openstreetmap-carto

sudo bash install-tilemill.sh

#And hence here: http://www.postgis.org/documentation/manual-2.0/postgis_installation.html
#? 
sudo apt-get install -y postgresql libpq-dev postgis

# Install OSM2pgsql

sudo apt-get install -y software-properties-common git unzip
sudo add-apt-repository ppa:kakrueger/openstreetmap
sudo apt-get update
sudo apt-get install -y osm2pgsql

#(leave all defaults)

#Install TileMill

sudo add-apt-repository ppa:developmentseed/mapbox
sudo apt-get update

sudo apt-get install -y tilemill

# less /etc/tilemill/tilemill.config
# Verify that server: true

sudo start tilemill

# To tunnel to the machine, if needed:
# ssh -CA nectar-maps -L 21009:localhost:20009 -L 21008:localhost:20008
# Then access it at localhost:21009

# Configure Postgres

echo "CREATE ROLE ubuntu WITH LOGIN CREATEDB UNENCRYPTED PASSWORD 'ubuntu'" | sudo -su postgres psql
# sudo -su postgres bash -c 'createuser -d -a -P ubuntu'

#(password 'ubuntu') (blank doesn't work well...)

# === Unsecuring TileMill

export IP=`curl http://ifconfig.me`

cat > tilemill.config <<FOF
{
  "files": "/usr/share/mapbox",
  "coreUrl": "$IP:20009",
  "tileUrl": "$IP:20008",
  "listenHost": "0.0.0.0",
  "server": true
}
FOF
sudo cp tilemill.config /etc/tilemill/tilemill.config

# ======== Postgres performance tuning
sudo bash
cat >> /etc/postgresql/9.1/main/postgresql.conf <<FOF
# Steve's settings
shared_buffers = 8GB
autovaccuum = on
effective_cache_size = 8GB
work_mem = 128MB
maintenance_work_mem = 64MB
wal_buffers = 1MB

FOF
exit

# ==== Automatic start 
cat > rc.local <<FOF
#!/bin/sh -e
sysctl -w kernel.shmmax=8000000000
service postgresql start
start tilemill
service nginx start
exit 0
FOF

sudo cp rc.local /etc/rc.local

# === Securing with nginx
sudo apt-get -y install nginx

cd /etc/nginx
sudo bash
printf "maps:$(openssl passwd -crypt 'incorrect cow cell pin')\n" >> htpasswd
chown root:www-data htpasswd
chmod 640 htpasswd
exit

cat > sites-enabled-default <<FOF

server {
   listen 80;
   server_name localhost;
   location / {
        proxy_set_header Host \$http_host;
        proxy_pass http://127.0.0.1:20009;
        auth_basic "Restricted";
        auth_basic_user_file htpasswd;
    }
}

server {
   listen $IP:20008;
   server_name localhost;
   location / {
        proxy_set_header Host $http_host;
        proxy_pass http://127.0.0.1:20008;
        auth_basic "Restricted";
        auth_basic_user_file htpasswd;
    }
}

FOF

sudo cp sites-enabled-default /etc/nginx/sites-enabled/default
sudo service nginx restart

echo "Australia/Melbourne" | sudo tee /etc/timezone
sudo dpkg-reconfigure --frontend noninteractive tzdata

Improving on the “administration rights required” workflow

Consider an action like creating a new “space” in Confluence. Because of the visibility of this action, people want it restricted. Controlled. Managed. So Atlassian makes it only available to people with administrator rights. Which seems ok, until you realise that the workflow of the non-administrator ends up looking like this:

  1. …doing stuff…
  2. Decide to create new space
  3. Attempt to create space, discover that you need an administrator to do it
  4. Find out who the administrator is
  5. Ask them to do it
  6. Wait
And for the administrator it looks like this:
  1. …doing stuff…
  2. Receive request to make a new Confluence space.
  3. Confluence? What? I’m busy configuring a new VM here…oh, fine.
  4. Go log into Confluence, remember how to create a space
  5. Get back to work
And probably there will be some miscommunication about exactly what is required. Confluence spaces are a bit of a trivial example. In other instances, you need a lot of information to perform the administrative action, and if there’s a mistake, it again requires the administrator to fix it. Eventually the administrator gets sick of being bothered with such trivia, and the non-administrator gets sick of hassling them, finding an alternative, or living with a misconfigured thing.
For the user: frustration, blockage, helplessness
For the administrator: disruption, menial tasks

Alternative workflow #1: approval only

For the non-administrator:

  1. …doing stuff
  2. Decide to create a new Confluence space
  3. Enter the form, fill out all the details, press Ok.
  4. (Confluence sends a “Is this ok?” email to administrator)
  5. Wait
For the administrator:
  1. …doing stuff
  2. Receive approval request.
  3. Since it’s from a fairly trustworthy user, and seems to make sense, click the link.
  4. Get back to work.
It’s better. The administrator now doesn’t need to know anything about the request. This workflow exists in some kinds of systems (CMSes particularly), but could be a lot more widespread. Ironically, although Jira is a workflow tool, it doesn’t actually have any workflows built in for administration.
For the user: blockage
For the administrator: disruption

Alternative workflow #2: private pending approval

Since creating a space has very limited potential for destruction if it’s hidden, how about this:

For the administrator:

  1. …doing stuff
  2. Decide to create Confluence space
  3. Fill out form
  4. Press Ok
  5. (Confluence sends email to administrator, creates space in “private” mode)
  6. Start working in new space. Do almost anything except collaborate with others in this space.
The benefit is clear: there is now no “waiting” step.
For the administrator, it’s the same as before:
  1. …doing stuff
  2. Receive approval request.
  3. Since it’s from a fairly trustworthy user, and seems to make sense, click the link.
    1. If more information is required, look at what the user has done in the new space, for a bit of context.
  4. Get back to work.
There is a mild benefit here, too: the administrator is no longer under as much pressure to immediately approve the thing, and can even see what has been done with the space.
For the user: flow maintained
For the administrator: very mild disruption

Alternative workflow #3: public until reverted

Most users aren’t destructive. And especially in professional environments, virtually all users can be trusted to act in good faith. (Managers strangely predict “chaos” if many users are authorised). So, let them do it:

  1. Decide to create Confluence space
  2. Create Confluence space
  3. Work in new space
For the administrator:
  1. Receive notification that a space has been created
  2. If it looks wrong, strange, inappropriate etc, discuss with the user, and possibly remove it.
  3. Otherwise, keep working
For the user: productivity, responsibility
For the administrator: productivity, trust