…about maps, open data, Git, and other tech.
National Map is a pretty awesome place to find geospatial open data from all levels of Australian government. (Disclaimer: I work on it at NICTA). But thanks to some not-so-obvious features in TerriaJS, the software that drives it, you can actually create and share your own private version with your own map layers – without programming, and without deploying any code.
What you get:
Want to create your own spatial layer – polygons, lines and points? Use geojson.io and choose Save > Gist to save the result to Github Gist. (Gist is just a convenient service that stores text on the web for free).
How about a layer of data about suburbs by postcode? Create a Google Sheet that follows the csv-au-geo specification (it’s easy!), download as CSV, paste it into a fresh Gist.
Using the new TerriaJS Data Source Editor (I made this!), create your new catalog. You’re basically writing a JSON file but using a web form (thanks json-editor!) to do it.
To add one of your datasources on Gist, make sure you link to the Raw view of the page:
Don’t forget to select the type for each file: GeoJSON, CSV, etc.
You might want to bring in some other data sources that you found on National Map. This can be a little tricky – there’s a lot of complexity in accessing data sources that National Map hides for you.
But here’s roughly how to go about it for a WMS (web map service) data source.
In the layer’s info window, grab the WMS URL
You’ll need to put “http://nationalmap.gov.au/proxy/” in front of a some layers, because their WMS servers don’t support CORS.
You’ll also need the value of the “Layer name” field. (For Esri layers you need to dig a bit further.)
(Yes, this layer is called “2”)
Add a WMS layer, and add “Layers Names” as an additional property. So it looks like this:
You can add extra properties to layers to fine tune their appearance. For example, for our CSV dataset:
You might want to set “Is Enabled” and “Is Shown” on every layer so they display automatically.
And finally, you might want to set an initial camera and base map, so the view doesn’t start off the west coast of Africa with a satellite view.
As you make changes, click “Save to Gist” to save your configuration file to a secret location on Gist. You can then click “Preview your changes in National Map”.
Make a note of the Gist link so you can keep working on it in the future. You can’t modify an existing configuration, but you can load from there and save a new copy.
Now you have a long URL like this: http://nationalmap.research.nicta.com/#clean&https%3A%2F%2Fgist.githubusercontent.com%2Fanonymous%2Fc3f181ca742b9ed94fe4%2Fraw%2F10853f7d8bb33610e4f2ce26947eaf6882192957%2Fdatasource.json
So, use tinyurl.com or another URL shortening service to get something more useful:
Everyone loves hackathons. And almost as much, everyone loves asking “but what happens to the projects afterwards?” There’s more than one route to follow. I’d like to propose four standard recipes we can use to describe the prospects of each project.
The creators of the hack could form a business. The developers work very hard to polish up what they’ve written until it’s a viable product ready for the marketplace, and then try to build a start-up around it while probably looking for external funding.
Snap Send Solve – hackathon to start-up success story
This kind of result is very desirable for hackathon organisers because there is such a clear story of benefits and outcomes: “a few thousand dollars of sponsorship paid for a weekend hackathon which led to this $50 million start-up which makes the app your grandma uses, which is great for the economy”.
Ingredients required: Start-up mentors, entrepreneurs, a business focus from the get-go
OpenBinMap.org – a government app in waiting?
If you make an interesting and useful app with a government body’s data, then maybe they’d like to take it on board. They might use the code base, but it’s probably better to use the concept and vision and write the code from scratch. Imagination isn’t a government strong suit, but once they see something they like, they’re pretty good at saying “we need one of those”.
This also doesn’t seem to happen very often, but can we try harder? We should follow GovHack up with serious discussions between hack developers and the government bodies that sponsored them. Following my cheeky “CanIBoatHere.com” category winner last year, I did meet with Transport Safety Victoria, but didn’t really have the time or motivation to pursue it. But they were very keen, so why couldn’t we have made it work? Similarly, there was potentially money available from the Victorian Technology Innovation Fund to support GovHack projects, but no clear process meant that months of fumbling through paperwork might eventually lead to nothing. Not so appealing to developers.
Ingredients needed: A solid process, government/developer wranglers, pre-commitment to funding.
Eventable would make a great community project.
If a hack is interesting and important enough to other developers, could it become a self-sustaining open source project? The idea seems plausible, but I don’t know if I’ve seen it happen. The major blockers are the hackish quality of the code itself which typically would require a major rewrite, and the sense that the weekend was fun, and this would be a lot of work. Hacks are a kind of showy facade. Once developers sit down to talk seriously about onward development, all kinds of serious difficulties start to emerge. And between the end of the weekend and the announcement of prizes a lot of momentum gets lost which can be hard to start up again.
Ingredients needed: Post-hackathon events to explore projects and establish communities.
Living, Breathing Melbourne – still just a story.
And finally, let’s acknowledge that the most important part of many hacks is their potential as an interesting story in their own right. Anthony Mockler’s GovHack 2012 entry “Is your Pollie Smarter than a Fith Grader” isn’t a failure because it didn’t lead to a start up – it was a great story that captured a lot of attention. My team’s 2014 entry “Living, Breathing Melbourne” has been frequently referred to as a model for actual open data dashboards, even though we didn’t develop it further. We should try to extract as much value as possible from these stories, and preserve their essence, even if only in screenshots and blog posts.
Ingredients needed: Story tellers, blog posts, active engagement with journalists
Let’s think of these different paths early on when discussing projects: “This would make a great community project“, “I don’t see this going anywhere, but let’s get the story out”, “It would be a shame if the department doesn’t take this on as a government app“. And don’t write off a hack just because it didn’t fit into the mould you were thinking of.
As a data guru in residence, I’m helping government bodies prioritise which datasets to release as open data. Sometimes people say “No one would ever find this data interesting, so why bother releasing it?” I think there are several distinct reasons why a given dataset might be worth releasing. Some datasets are valuable for several reasons simultaneously. Some aren’t valuable at all.
When a public servant comments that a potential dataset isn’t interesting or useful, ask: “are there other reasons to release it”?
But if a dataset fails to meet any of these criteria? You have my permission not to release it.
Census Explorer, by Yuri Feldman, allows easy exploration of part of the 2011 Australian Census.
Datasets like public transport timetables, public bike share station status, or parking space availability are obvious candidates for third party developers to use to build an app. Unfortunately, these examples also require near-realtime feeds in order to be useful.
Even if a dataset isn’t interesting or useful enough to warrant an app in its own right, it could add value to another website or app if it’s easy to use. I’ve come across many of these:
Which way would you cycle to Port Albert?
If a dataset is big, rich, detailed and high quality, then there’s a pretty good chance it’s worth of some kind of analysis. If it’s unique enough, then it might even interest a researcher in starting a research project just to look at this dataset.
Examples: building permits database, public transport timetables (for urban planning).
Much more common than such a rich dataset is small datasets that researchers find useful to solve particular problems, add context, or strengthen an analysis. Local Government Area boundaries aren’t inherently interesting, but they’re one of the geospatial datasets that researchers request the most often. The ATO’s Standard Business Rules taxonomy sounds incredibly dry to me, but is of potential use to lots of people trying to glue different kinds of data and applications together.
Lots of organisations need government data to develop internal strategies or policies to be shared with the public – or even to influence government. Typically they get the data either by transcribing tables from official reports, or by developing direct relationships with the government body in question. Publishing data directly to an open data portal allows a wider range of groups to make use of it, without the overhead of having to ask whether the data is available. Data that is collected regularly, in the same format is a particularly likely to be useful.
If the data relates to how government decisions are made, it may be worth releasing to demonstrate transparency – regardless of how much the dataset is even used. For example, releasing annual budget data as an easy to use spreadsheet makes a big political statement about willingness to be scrutinised. Even if no citizen takes up the opportunity to crunch the numbers, they may still appreciate having that option.
Examples: annual budgets, revenue sources (parking meters, speeding fines), parliamentary voting records.
If you’re really lucky, the dataset you publish may help another part of government do something useful. I think good things happen when people can access data without having to ask anyone for it, and the some goes for governments themselves. You can’t really expect insights, but if it happens – great.
Steve Bennett blogs 
Recent Comments