Difference between revisions of "Processing GeoData for CEDEUS"

Latest revision as of 17:44, 30 June 2016

>> return to Cedeus IDE

This page provides info and processing instructions for CEDEUS data.

GDAL cheat sheet

GDAL can be used for raster data manipulation. A cheat sheet can be found here.

Example: cropping the CIGAR SRTM DEM for the americas to Chile, with a shapefile that contains a bounding box polygon for Chile:

gdalwarp -cutline polygon_bbox_chile_wgs84.shp -crop_to_cutline SRTM_W_250m.tif SRTM_W_250m_chile.tif

Processing Aerial Images from SECTRA / CEDIZ

Pre-processing of ECW image use with GeoServer

• downlading zip files with ecw images from:
  • 2007: http://www.sectra.gob.cl/datos_e_informacion_espacial/gran_santiago/ortofoto_2007.html
  • 2012: http://www.sectra.gob.cl/datos_e_informacion_espacial/gran_santiago/ortofoto_2012.html
• unpacking the zip files with winrar on Windows to have finally the ecw files (I did this with winrar, as it allows to select multiple zips easily and giving one output dir)
• converting the ECW files to GeoTIFF:
  • use QGIS under Windows (installed with OSGeo installer) to convert the files. Important, QGIS for Windows comes with ecw support, while on Ubuntu it has to be installed first... so its much easier
  • I used the QGIS function: Raster > Conversion > Translate (Convert Format) which utilized gdal_translate (of course ;)
  • apply batch processing
  • for the 16cm/px data it makes sense, I think, to use jpg compression - but see next note below (can't get JPG to work with GeoServer). This returns the smallest files, in comparing to other compression modes: e.g. for file 166.ecw : Packbits: 1.6 GB, Deflate: 527 MB, JPG: 214 MB . Using jpg compression should be ok, as the ecw files are compressed lossy anyway (using the wavelet transform), and we have images = non-category data.
  • it looks like javax.imageio used by GeoServer can't read the jpg compressed TIFF, so I converted the jpg Tiff file to Tiff with "Deflate" compression using QGIS. The gdal command for that looks something like this:

    gdal_translate -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=9 -of GTiff ./tiff_jpgcompress/101.tif ./tiff_deflate/101d.tiff

• not sure about the need for this step (except for better visualisation?): generate an index file:
  • the idea is to create a shapefile containing an image index, to be uploaded to GeoNode
  • in QGIS I used the function Raster > Miscellaneous > Tile Index... which utilizes gdaltindex

Correcting Declared Image SRS

I got one image which geotiff header said it is UTM19s but it was actually UTM18s. This could be seen when overlaying the image with OSM data (using the OpenLayersPlugin in QGIS). Gdal allows to see the GeoTiff header info this way:

gdalinfo Villarrica_utm18s.tif

I was able to overwrite the header with the following command

gdal_translate -a_srs EPSG:32718 Villarrica_utm18s.tif Villarrica_utm18s_corrected.tif 

=> means, I am not using gdal_warp which would reproject the file, which is not what I wanted. Note that I had to compress the file afterwards again (did this in QGIS using the high-compression option which uses DEFLATE)

Adding the images to GeoServer

The set of images should be added as ImageMosaic. On how to do that I used this guide, however, there is also an official doc:

• http://docs.geoserver.org/stable/en/user/tutorials/image_mosaic_plugin/imagemosaic.html

In short the process is like this:

• copy the files to the GeoNode VM
  • I created a new folder /var/www/geoserver to not have the images files in the regular GeoNode backup, since the image folder size may be pretty huge (e.g. for Sectra stgo 2012 at 16cm/px I have 202 files with a total size of 135 GB!)
  • make sure the new ./geoserver folder and its folder below are owned by tomcat7 (using chown)... because I read about this issue. When I tried adding 2 Orbview3 images from Iquique as ImageMosaic, I also got the error message until made tomcat7 owner of the ./geoserver/orbview3/iquique folder.
• create a new workspace.

  Note 1, its possible to chose any URI/domain. It does not need to be an URI that is existing/accessible. So in my case I called it: "Sectra_Stgo_2012_img_test" and gave the URI as "cedeusimages.geosteiniger.cl"

  Note 2, don't change the name of the workspace later on in the workspace info dialog... After I did this, I could neither access nor delete the store properly.

• create a new Store.
  • Here I chose that this store is of type "ImageMosaic".
  • Make sure that this store is part of the newly created workspace.
  • I called the store "Sectra_stgo2012_img_test", so like the workspace before, but one can have many stores in a workspace.
  • In the URL field I used the browse button to select the folder with the tiff images.

    => I could not figure yet if I can chose an http folder else where, so I can store the images on CedeusDB (outside of the Geonode1204 VM) or so...

  • Note: if this folder does not have a shape file with the image bounds, the ImageMosaic plugin/module will generate such shapefile (and other stuff?)
• now create a new Layer (may go to this step automagically, but if something goes wrong and one delete the layer on can do this separatly)
  • important here under the "Data" tab: check if the coordinate system is set (and bounds are set)
  • under the "Publishing" tab: go to the styles and select "raster" in the style list and move it to the "Selected Styles" field.
• check if the Layer is displayed using the "Layer Preview" function

Import the image data into GeoNode

• use "geonode updatelayers --ignore-errors --workspace Sectra_Stgo_2012_img_test"

Converting Census data from INE

see http://stackoverflow.com/questions/156279/how-to-import-a-sql-server-bak-file-into-mysql

• downloaded "SQL Server 2014 Management Studio Express 64 Bit" from Microsoft webpage (Sa...u) : http://www.microsoft.com/en-us/server-cloud/products/sql-server-editions/sql-server-express.aspx
• downloaded also SQL Server 2014 Express with Tools 64 Bit : file name "SQLEXPRWT_x64_ENU" (or SQLEXPRADV_x64_ENU.exe as outlined [here http://stackoverflow.com/questions/28127673/microsoft-sql-server-2012-no-database-engines]) as the above contains only the Studio, not the Server
• then I tried following this tutorial: https://msdn.microsoft.com/en-us/library/ms345332.aspx
• and then I followed this tutorial for restoring the DB - choosing "Device" in the Restore Dialog to select the *.bak file: https://msdn.microsoft.com/en-us/library/ms177429%28v=sql.120%29.aspx
• how to export the single DB tables to csv is described here.

=> Thanks to Chile having only 17 Million people, I have been able to export all 7 Censo 2012 tables to csv files! *yay* The "poblacion" table was the biggest with 16.x million records and about 3.2 GB size.