Difference between revisions of "Processing GeoData for CEDEUS"

Revision as of 16:41, 3 July 2015

>> return to Cedeus IDE

This pager provides logging info and processing instructions for CEDEUS data.

Processing Aerial Images from SECTRA

Pre-processing of ECW image use with GeoServer

• downlading zip files with ecw images from:
  • 2007: http://www.sectra.gob.cl/datos_e_informacion_espacial/gran_santiago/ortofoto_2007.html
  • 2012: http://www.sectra.gob.cl/datos_e_informacion_espacial/gran_santiago/ortofoto_2012.html
• unpacking the zip files with winrar on Windows to have finally the ecw files (I did this with winrar, as it allows to select multiple zips easily and giving one output dir)
• converting the ECW files to GeoTIFF:
  • use QGIS under Windows (installed with OSGeo installer) to convert the files. Important, QGIS for Windows comes with ecw support, while on Ubuntu it has to be installed first... so its much easier
  • I used the QGIS function: Raster > Conversion > Translate (Convert Format) which utilized gdal_translate (of course ;)
  • apply batch processing
  • for the 16cm/px data it makes sense, I think, to use jpg compression - but see next note below (can't get JPG to work with GeoServer). This returns the smallest files, in comparing to other compression modes: e.g. for file 166.ecw : Packbits: 1.6 GB, Deflate: 527 MB, JPG: 214 MB . Using jpg compression should be ok, as the ecw files are compressed lossy anyway (using the wavelet transform), and we have images = non-category data.
  • it looks like javax.imageio used by GeoServer can't read the jpg compressed TIFF, so I converted the jpg Tiff file to Tiff with "Deflate" compression using QGIS. The gdal command for that looks something like this:

    gdal_translate -co COMPRESS=DEFLATE -co PREDICTOR=2 -co ZLEVEL=9 -of GTiff ./tiff_jpgcompress/101.tif ./tiff_deflate/101d.tiff

• not sure about the need for this step (except for better visualisation?): generate an index file:
  • the idea is to create a shapefile containing an image index, to be uploaded to GeoNode
  • in QGIS I used the function Raster > Miscellaneous > Tile Index... which utilizes gdaltindex

Adding the images to GeoServer

The set of images should be added as ImageMosaic. On how to do that I used this guide, however, there is also an official doc:

• http://docs.geoserver.org/stable/en/user/tutorials/image_mosaic_plugin/imagemosaic.html

In short the process is like this:

• copy the files to the GeoNode VM
  • I created a new folder /var/www/geoserver to not have the images files in the regular GeoNode backup, since the image folder size may be pretty huge (e.g. for Sectra stgo 2012 at 16cm/px I have 202 files with a total size of 135 GB!)
  • make sure the new ./geoserver folder and its folder below are owned by tomcat7 (using chown)... because I read about this issue. When I tried adding 2 Orbview3 images from Iquique as ImageMosaic, I also got the error message until made tomcat7 owner of the ./geoserver/orbview3/iquique folder.
• create a new workspace.

  Note 1, its possible to chose any URI/domain. It does not need to be an URI that is existing/accessible. So in my case I called it: "Sectra_Stgo_2012_img_test" and gave the URI as "cedeusimages.geosteiniger.cl"

  Note 2, don't change the name of the workspace later on in the workspace info dialog... After I did this, I could neither access nor delete the store properly.

• create a new Store.
  • Here I chose that this store is of type "ImageMosaic".
  • Make sure that this store is part of the newly created workspace.
  • I called the store "Sectra_stgo2012_img_test", so like the workspace before, but one can have many stores in a workspace.
  • In the URL field I used the browse button to select the folder with the tiff images.

    => I could not figure yet if I can chose an http folder else where, so I can store the images on CedeusDB (outside of the Geonode1204 VM) or so...

  • Note: if this folder does not have a shape file with the image bounds, the ImageMosaic plugin/module will generate such shapefile (and other stuff?)
• now create a new Layer (may go to this step automagically, but if something goes wrong and one delete the layer on can do this separatly)
  • important here under the "Data" tab: check if the coordinate system is set (and bounds are set)
  • under the "Publishing" tab: go to the styles and select "raster" in the style list and move it to the "Selected Styles" field.
• check if the Layer is displayed using the "Layer Preview" function

Import the image data into GeoNode

• use "geonode updatelayers --ignore-errors --workspace Sectra_Stgo_2012_img_test"

Converting Census data from INE

see http://stackoverflow.com/questions/156279/how-to-import-a-sql-server-bak-file-into-mysql

• downloaded "SQL Server 2014 Management Studio Express 64 Bit" from Microsoft webpage (Sa...u) : http://www.microsoft.com/en-us/server-cloud/products/sql-server-editions/sql-server-express.aspx
• downloaded also SQL Server 2014 Express with Tools 64 Bit : file name "SQLEXPRWT_x64_ENU" (or SQLEXPRADV_x64_ENU.exe as outlined [here http://stackoverflow.com/questions/28127673/microsoft-sql-server-2012-no-database-engines]) as the above contains only the Studio, not the Server
• then I tried following this tutorial: https://msdn.microsoft.com/en-us/library/ms345332.aspx
• and then I followed this tutorial for restoring the DB - choosing "Device" in the Restore Dialog to select the *.bak file: https://msdn.microsoft.com/en-us/library/ms177429%28v=sql.120%29.aspx