Difference between revisions of "Nominatim for Traffic Accident Database"
(→Download (optional) data) |
(→First Nominatim Installation) |
||
Line 131: | Line 131: | ||
(=> requires to be in the Nominatim folder). ''wikipedia_article.sql.bin'' had a size of 1.2GB and redirect 240MB. Combined the 2 files are around 1.5GB and add around 30GB to the install size of nominatim. They also increase the install time by an hour or so. | (=> requires to be in the Nominatim folder). ''wikipedia_article.sql.bin'' had a size of 1.2GB and redirect 240MB. Combined the 2 files are around 1.5GB and add around 30GB to the install size of nominatim. They also increase the install time by an hour or so. | ||
+ | |||
+ | ==== Creating postgres accounts ==== | ||
+ | The import needs to be done with a postgres superuser with the same name as the account doing the import. | ||
+ | sudo -u postgres createuser -s <your user name> |
Revision as of 16:34, 4 March 2015
>> return Cedeus_IDE
Objective
Develop a script for automated geo-referencing of Chiles transit accident database.
Contents
Tools
- Nominatim
- MS-Access / LibreOffice : need to load the MS Access DB from Carabineros de Chile in OpenOffice/LibreOffice: http://askubuntu.com/questions/187389/is-it-possible-to-open-an-access-2010-database-file-without-using-wine-or-virtua/519571#519571
- iPython + Notebook
Installing Nominatim
- Nominatim installation manual: https://wiki.openstreetmap.org/wiki/Nominatim/Installation
Setup of Nominatim VM
- copying basicubunutu1404.vdi file and renaming to nominatim.vdi
- creating the nominatim VM on CedeusDB (ip.18):
-
VBoxManage createvm --name nominatim --ostype Ubuntu_64 --register
-
VBoxManage modifyvm nominatim --memory 4096
-
VBoxManage modifyvm nominatim --cpus 4
-
VBoxManage modifyvm nominatim --nic1 nat
-
VBoxManage storagectl nominatim --name "SATA Controller" --add sata --controller IntelAhci
- assign the (old) disk image a new uuid
-
VBoxManage internalcommands sethduuid nominatim.vdi
-
- attach the (old) disk image:
-
VBoxManage storageattach "nominatim" --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium nominatim.vdi
-
-
VBoxManage storagectl nominatim --name "IDE Controller" --add ide --controller PIIX4
- set the nat rules (ports):
-
VBoxManage modifyvm nominatim --natpf1 "ssh,tcp,,20022,,22"
-
VBoxManage modifyvm nominatim --natpf1 "apache,tcp,,20080,,80"
-
-
- optional - set VRDE port:
-
VBoxManage modifyvm nominatim --vrdeport 7761
(Note: 3389 is the default port anyway)
-
- check all settings with
VBoxManage showvminfo nominatim
- start the VM
-
VBoxHeadless -s nominatim --vrde on &
- the VM will listen on port 7761 - connect with GTKvncViewer to check
- check if ssh connection works
-
- change the computers name in /etc/hosts and /etc/hostname
- restarting Apache gives the error message: "apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 ..."
- => solve this by adding ServerName nominatim as the last line in /etc/apache2/apache2.conf file
- shut the VM down to resize the HDD if necessary
- check the HDD size:
VBoxManage showhdinfo nominatim.vdi
returns:
- check the HDD size:
UUID: e148d758-00ac-4b60-a067-d5c97316069d Parent UUID: base</br> State: created Type: normal (base) Location: /home/ssteinig/nominatim.vdi Storage format: VDI Format variant: dynamic default Capacity: 100000 MBytes Size on disk: 6224 MBytes In use by VMs: nominatim (UUID: be795392-3786-4f3b-9714-a9445f91855d)
- so I resized the disk to 200GB as, according to Nominatim infos the disk size for a full planet is 700GB
-
VBoxManage modifyhd nominatim.vdi --resize 200000
-
- so I resized the disk to 200GB as, according to Nominatim infos the disk size for a full planet is 700GB
Setup of Nominatim itself
Install needed libraries
=> see https://wiki.openstreetmap.org/wiki/Nominatim/Installation#Software
sudo apt-get install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev libtool automake libproj-dev sudo apt-get install gcc proj-bin libgeos-c1 git osmosis libgeos++-dev sudo apt-get install php5 php-pear php5-pgsql php5-json php-db instead of: >> sudo apt-get install postgresql postgis postgresql-contrib postgresql-9.1-postgis postgresql-server-dev-9.1 use: >> sudo apt-get install postgresql postgis postgresql-contrib postgresql-9.3-postgis-2.1 postgresql-server-dev-9.3 sudo apt-get install libprotobuf-c0-dev protobuf-c-compiler
and perhaps
sudo apt-get install git autoconf-archive
Edit PostreSQL settings
perhaps edit Postgres settings (before data import) sudo nano /etc/postgresql/9.3/main/postgresql.conf
=> see https://wiki.openstreetmap.org/wiki/Nominatim/Installation#Software
My machine has only 4GB, so I changed:
- shared_buffers = 128MB => 512MB
- work_mem = 50MB (previously uncommented, and set to 1MB)
- maintenance_work_mem = 2GB (previously commented, and set to 16MB) //=> This value should be reduced again afterwards to avoid swapping when autovacuum runs.
- effective_cache_size = 2GB (previously commented, and set to 128MB)
- synchronous_commit = off (previously commented, and set to on)
- checkpoint_segments = 100 (previously commented, and set to 3)
- checkpoint_timeout = 10min (previously commented, and set to 5min)
- checkpoint_completion_target = 0.9 (previously commented, and set to 0.5)
and for initial import only:
- fsync = off (previously commented, and set to 'on')
- full_page_writes = off (previously commented, and set to 'on')
- => switch both values back to 'on' after import, to avoid database corruption.
Note, I did a restart of the postgresql service after to have the settings applied.
First Nominatim Installation
Get the source code
I used the last stable release 2.3.1:
wget http://www.nominatim.org/release/Nominatim-2.3.1.tar.bz2 tar xvf Nominatim-2.3.1.tar.bz2
and rename the folder
mv Nominatim-2.3.1 Nominatim
Compiling
now compile the software
cd Nominatim ./configure make
Note, the warning about missing lua libraries can be ignored. Nominatim does not make use of osm2pgsql's lua extension.
Configuration
create the configuration file local.php in the sub folder settings with the following content:
<?php // Paths @define('CONST_Postgresql_Version', '9.3'); @define('CONST_Postgis_Version', '2.1'); // Website settings //@define('CONST_Website_BaseURL', 'http://mysite/nominatim/'); @define('CONST_Website_BaseURL', 'http://146.155.17.18:20080/nominatim/');
for further params see the file settings/settings.php.
Download (optional) data
Wikipedia rankings (downloading took me 20+ mins @PUC)
Wikipedia can be used as an optional auxiliary data source to help indicate the importance of osm features. Nominatim will work without this information but it will improve the quality of the results if this is installed. This data is available as a binary download.
wget --output-document=data/wikipedia_article.sql.bin http://www.nominatim.org/data/wikipedia_article.sql.bin wget --output-document=data/wikipedia_redirect.sql.bin http://www.nominatim.org/data/wikipedia_redirect.sql.bin
(=> requires to be in the Nominatim folder). wikipedia_article.sql.bin had a size of 1.2GB and redirect 240MB. Combined the 2 files are around 1.5GB and add around 30GB to the install size of nominatim. They also increase the install time by an hour or so.
Creating postgres accounts
The import needs to be done with a postgres superuser with the same name as the account doing the import.
sudo -u postgres createuser -s <your user name>