GeoBase

Page last modified 18:22, 29 Oct 2009 by matt

1. Hacking geobase.ca
2. Ordering Data
3. Downloading
1. 3.1. Post Download
2. 3.2. without ArcInfo or TIN
4. The End Results

Hip Hip Hooray! http://geobase.ca/ is live and online, ready and waiting to serve up delicious free canadian geospatial data. Now for our regularily scheduled "Matt criticizes everything" broadcast

They've cleaned up a lot of things in the last six months. Performance still sucks, but maybe that's because this is their big launch day. Everything (that I've looked at) is tiled. That means to download the geographical names for 105D you need to ask for 105D01, 105D02 etc. (and you must use the mouse for each tile. no cutting and pasting or typing allowed).

Upon submitting you are asked for an email address to which, after a time, the ftp instructions are sent. The length of time varies according to the number of tiles requested.

Looking at the delivery instructions, it is clear that they are using a secure transaction process and now we understand why the server is easily overloaded. You are not directed to a public ftp directory but to unique numbered location like this ftp://ftp.cits.rncan.gc.ca/pub/geobase/024343_1069262100484/ where each file also has a unique session id like 105A01_1069262100484.zip. Why a computationally expensive procedure like this is being used for free unrestricted-use data is a good question.

If you hack the form and use wildcards, delivery instructions are sent but there is nothing in the identified folder to pickup. Looking at the URLs I'm pretty sure a script could be thrown together with relatively little effort to bypass all the grunt work and send a "deliver me everything" request.

One wonders whether anybody will do this before the end of their big launch day. And whether the server will come down in flames or just slow like ever thickening molasses as the temperature drops (brrr! it's cold outside today!). Please people, be kind. Just because something can easily be done doesn't mean it should.

All in all it's a very beautiful looking site but to actually use it for more than a few bite size pieces of data at a time is painful. The good news here is that the data is available and free. There is nothing in the license agreement which prevents another agency from grabbing the data and hosting it in a more usable fashion. NRCAN and CCOG should be congratulated for their efforts and thanked effusively for liberated this sizable chunk of canadian geospatial data.

(P.S. if you are using a popup blocker you won't get treated to the interactive web-mapping interface. To launch the web map interface use http://www.geobase.ca/cwc2/cwc2.php for english and http://www.geobase.ca/cwc2/cwc2.php?TEMPLATE=geobase_fr.html for french.)

-- MattWilkie, sometime in November 2003

Hacking geobase.ca

or: how to make geobase.ca more useful

Hacking geobase.ca

Easy: Ask someone else who has already gone through all the garbage to give, or sell, the data to you! (see end of page)

Medium: see below

Hard: use http://geobase.ca/ as designed

Ordering Data

Go to Data Extraction, select the data layer you are interested in, zoom in until you get to the form which has a text area box for selecting individual tiles, then:

bookmark or drag the URL to a folder (the title will be like "Geobase / Select")

use the undisable bookmarklet to activate the form so you can type in it

type or paste a comma separated list of the tiles you want (the total number of characters has to be less than 1850) and press the Submit button

Choose the format and language options and Save Options. Bookmark the resultant URL ("Geobase / Confirmation"). Double check the list of selected tiles.

Slap Delivery, you will be presented with license terms, presuming you accept them punch I Agree

Enter an email address. Don't use a fake one as delivery instructions are sent to this address. Bookmark this URL before tapping Delivery

After a lengthy wait, perhaps 5 minutes, you should get a confirmation page with an order number and the list of tiles you selected. The title is "Geobase / Order Completed". Double check the list of selected tiles.

Wait for up to several hours to recieve an email with ftp instructions for collecting your data.

If, as is likely, you want more data than can scrunch into the 1850 character limit, use the bookmarks to shorten the process:

Bookmark 1: Undisable > Paste > Submit
After page loads use Bookmark 2 to save your format and language options
After page loads use Bookmark 3 to skip license page; slap delivery
wait for confirmation number

We need to use bookmarks because 'sessionid' (and a temporary cookie I think) is used to remember your selections. However, thankfully, it appears that once an order is completed the same sessionid can be used over and over again.

URLS

http://geobase.ca/

Geobase Data Extraction

'undisable' read only forms bookmarklet

Downloading

You will want an ftp client capable of accepting a long list of URLs to download. If you don't have a favourite try wget or curl (part of Cygwin) for the command line and File Zilla for gui-ness.

Post Download

Okay, so now you have a thousand or so files with a bunch of extra garbage in their names to work with. Windows users can grab a utility like wildcard renamer and shorten 107N03_1069375429328.zip to 107N03_dem.zip in a few seconds.

I leave it up to you to append or mosaic all those tiles into larger mealsized chunks, but here a few snippets to get you started (assumes a windows machine with cygwin installed):

Command to unpack distribution zips all at once:

D:\geobase\data\>for %%a in (..\distrib\cded\*.zip) do unzip %%a

Now use wildcard renamer to strip all the garbage from the names. If you are like me and now have almost no disk space left, gzip everything with:

D:\geobase\data\> gzip *.dem

Now we are ready to bring into ArcInfo (assumes TIN license; if you don't have Arc or TIN see below for other options):

D:\geobase\data\> for %a in (*.gz) do (
   gzip --decompress %a
   rename %~na %~na.dem
   arc demlattice %~na.dem %~na usgs # int
   gzip %~na.dem
   )

And now we mosaic all the teeny morsels into real meals:

/* short ArcInfo AML to create 1:250k quads out of 32 50k half pieces
/* e.g. 094a01_e, 094a01_w, 094a02_e ...
GRID
&do quad &list a b c d e f g h i j k l m n o p
   &do tile &list 094 104 114    /* <-- change to suit
      &if [exist %tile%%quad%* -grid] &then &do
         &ty ...%tile%%quad%
         dem_%tile%%quad% = mosaic([listfile %tile%%quad%* -grid])
         /* if you are really brave, uncomment the following
         /* line to delete the source tiles:
         /* kill (![listfile %tile%%quad%* - grid]!) all
         &end
   &end
&end
QUIT

without ArcInfo or TIN

VirtualTerrainProject VTBuilder supports drag'n'drop. Just grab a bunch of .zip files (no need to unzip first!) and drop them onto the main window. (Also works for Arc grids by the way: recursive search for hdr.adf and drop those instead). Set your Extract Area to Extents, or draw a box, and then export as .bt file (no other export options available. yet).

If you have OpenEV installed open a bunch of the .dem files (no drag'n'drop yet I'm afraid) and File > Export to the format of your choice. For a more automated route, use gdal_translate from the OpenEV/bin directory and substite gdal_translate for the arc demlattice line in the example above.

If you have the complete gdal suite installed with python enabled you can use the gdal_merge.py script to duplicate the mosaicking step outlined above.

The End Results

As some of you know MattWilkie downloaded, projected and mosaicked the 1:50,000 digital elevation models for Northern BC (NTS 094*, 104*, 114*) available on http://geobase.ca/ .These are now online for whoever wants 'em at ftp://ftp.geomaticsyukon.ca/DEMs/NorthBC/ -- thank you Geomatics Yukon for setting aside ~2gb!

Update, Sept 2007: The rest of Canada available now too, see http://sydney.freeearthfoundation.com/mattwilkie/draft/canada_50k_dem. Thank you Adam Nowaki and the Free Earth Foundation for hosting it.

There is no support for this data, it's up to you to figure out how to load and use it. If you're stuck you can still email Matt but it may take some time before any response is forthcoming.

The quality of the 50k DEMs is widely variable. Some are just awesome, clearly having much more detail than is possible from 50k data alone -- I'm pretty sure they are derived from the 20k TRIM. Others are, well, still clearly derived from TRIM but also clearly wrong: significant north-south smearing patterns and occasional sunken countours. Use the shaded relief images to get a general indication of which areas are problematic.

Also downloaded are the 'geographic placenames' and 'primary vertical benchmarks' for YT but nothing been done with them yet. Maybe in January but Matthew a big project starting then so it could take awhile to process. You don't have to wait for him though, you have just read the outline of a process to make Geobase more useful, so get going!

I	Attachment	Action	Size	Date	Who	Comment
cmd	gen50k-list.cmd	manage	0.4 K	2003-11-26 - 16:07	TWikiGuest	generate list of 50k tiles
txt	list-all-50k.txt	manage	21.0 K	2003-11-26 - 16:06	TWikiGuest	all YK 50k NTS tiles, in lines of less 1800 char

Retrieved from "http://yukongis.ca/oldsite/GeoBase"