Difference between revisions of "Installing CartoDB for CEDEUS"

Revision as of 22:11, 28 October 2015

>> return to Cedeus IDE

Installing CartoDB

Setup of CartoDB VM

Setup of VM based on Ubuntu 1204 VM

1. copying tilestream.vdi file (we need Ubuntu 12.04!) and renaming to cartodb.vdi
2. creating the cartodb VM on CedeusDB (ip.18):
   • VBoxManage createvm --name cartodb --ostype Ubuntu_64 --register
   • VBoxManage modifyvm cartodb --memory 8096
   • VBoxManage modifyvm cartodb --cpus 6
   • VBoxManage modifyvm cartodb --nic1 nat
   • VBoxManage storagectl cartodb --name "SATA Controller" --add sata --controller IntelAhci
   • assign the (old) disk image a new uuid

     VBoxManage internalcommands sethduuid cartodb.vdi

   • attach the (old) disk image:

     VBoxManage storageattach "cartodb" --storagectl "SATA Controller" --port 0 --device 0 --type hdd --medium cartodb.vdi

   • VBoxManage storagectl cartodb --name "IDE Controller" --add ide --controller PIIX4
   • set the nat rules (ports):

     VBoxManage modifyvm cartodb --natpf1 "ssh,tcp,,19022,,22"

     VBoxManage modifyvm cartodb --natpf1 "apache,tcp,,19080,,80"

     VBoxManage modifyvm cartodb --natpf1 "pgsql,tcp,,19432,,5432"

3. optional - set VRDE port:
   • VBoxManage modifyvm cartodb --vrdeport 7763 (Note: 3389 is the default port anyway)
4. check all settings with VBoxManage showvminfo cartodb
5. start the VM
   • VBoxHeadless -s cartodb --vrde on &

   the VM will listen on port 7763 - connect with GTKvncViewer to check

   • open /etc/udev/rules.d/70-persistent-net.rules and comment out existing network adapters
   • reboot
   • check if ssh connection works
6. change the computers name in /etc/hosts and /etc/hostname
7. restarting Apache gives the error message: "apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 ..."

   => solve this by adding ServerName cartodb as the last line in /etc/apache2/apache2.conf file

Resize the CartoDB VM

SINCE CartoDB seems to require Ubuntu 12.04 I did NOT do the following with the tilestream.vdi based disk:

• shut the VM down to resize the HDD if necessary
• check the HDD size: VBoxManage showhdinfo cartodb.vdi returns:

UUID:           e148d758-00ac-4b60-a067-d5c97316069d
Parent UUID:    base</br>
State:          created
Type:           normal (base)
Location:       /home/ssteinig/cartodb.vdi
Storage format: VDI
Format variant: dynamic default
Capacity:       100000 MBytes
Size on disk:   6224 MBytes
In use by VMs:  nominatim (UUID: be795392-3786-4f3b-9714-a9445f91855d)

• so I resized the disk to 200GB as, according to Nominatim infos the disk size for a full planet is 700GB

  VBoxManage modifyhd cartodb.vdi --resize 200000

• to make this resizing effective I actually have to change the partitions:
  • create a zip-copy of the vdi file,
  • download & start the VM with Ubuntu Desktop(!) 14.04 iso

  VBoxManage storageattach cartodb --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium ubuntu-14.04.2-desktop-amd64.iso

  • choose the "Try Ubuntu" mode
  • use the program "gparted" to adjust/extend the partition's size
  • to be able to enlarge "extended" partition, on needs to click on the key symbol and chose "deactivate", as described for instance here: http://www.howtogeek.com/114503/how-to-resize-your-ubuntu-partitions/
  • on 12.04 when extending GeoNode1204 VM: one may also need to extend /dev/sda2 (cyan color) first before extending /dev/sda5 (brownish color)
  • remove the "iso" and restart the Vm; check size if the disks is really 200GB now

  VBoxManage storageattach cartodb --storagectl "IDE Controller" --port 1 --device 0 --type dvddrive --medium none

  • => Well it turned out that I used LVM (Logical Volume Manager) so the new space is not taken into account. Some more stuff needs to be adjusted!
    • after a bit of reading here: http://www.rootusers.com/how-to-increase-the-size-of-a-linux-lvm-by-expanding-the-virtual-machine-disk/ and here: http://www.thegeekstuff.com/2010/08/how-to-create-lvm/ and checking the status with commands such as pvscan, pvdisplay and lvdisplay; I figured that I simply need to extend the size of the logical volume (see also http://serverfault.com/questions/501895/used-vgextend-lvextend-to-add-addtional-8gb-space-but-it-is-not-reflected-in-df ) This is done with lvextend - so I used:
    • sudo lvextend -l +100%FREE /dev/basicubuntu1404-vg/root
    • sudo resize2fs /dev/mapper/basicubuntu1404--vg-root

    => this extended the disk to 187GB (see with sudo lvdisplay). The second command is needed to have this really available (see with df -h or sudo fdisk -l)

Setup of CartoDB itself

• General install instructions for 12.04 are here: https://github.com/CartoDB/cartodb
• launchpad (precise): http://ppa.launchpad.net/cartodb/base/ubuntu/dists/
• Google email Group: https://groups.google.com/forum/#!forum/cartodb

Had the following errors when executing

• sudo PGUSER=postgres make installcheck

The following tests are failing: test_cdb_querytables_returns_schema_and_table_name

• on pip install --no-use-wheel -r python_requirements.txt

/usr/local/lib/python2.7/dist-packages/pip-7.1.0-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: 
InsecurePlatformWarning: A true  SSLContext object is not available.
This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see  
https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
 InsecurePlatformWarning
Could not open requirements file: [Errno 2] No such file or directory: 'python_requirements.txt'

• on installing Windshaft and doing npm install

No package 'pangocairo' found
gyp: Call to 'pkg-config pangocairo --libs' returned exit status 1. while trying to load binding.gyp
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:343:16)

....and later:

Success: "/usr/include/CartoDB-SQL-API/Windshaft-cartodb/node_modules/windshaft/node_modules/grainstore/node_modules/millstone/node_modules/srs/lib /binding/node-v11-linux-x64/srs.node" is installed via remote
npm ERR! canvas@1.2.1 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the canvas@1.2.1 install script.
npm ERR! This is most likely a problem with the canvas package,
npm ERR! not with npm itself.

Trabajo Cristian

Trabajo basado en el tutorial de arjendk en https://gist.github.com/arjendk/6080388 y en https://github.com/gusmacaulay/CartodbUbuntu Fue posible ejecutar una máquina virtual con Ubuntu 14.04.3x64 con las configuraciones básicas de cartoDB siguiendo los siguientes pasos de instalación:

• Instalar git y vim

 sudo apt-get update
 sudo apt-get install -y git-core vim

• Clonar proyecto de CartoDB en la máquina

 git clone --recursive https://github.com/CartoDB/cartodb.git

• Tomar café ...

• Instalar dependencias de python, unp y zip

 sudo apt-get install -y python-software-properties software-properties-common unp zip

• Instalar GDAL

 sudo apt-get install -y gdal-bin libgdal1-dev

• Instalar GEOS, puede que ya aparezca instalado

 sudo apt-get install -y libgeos-c1 libgeos-dev

• Instalar Json

 sudo apt-get install -y libjson0 python-simplejson libjson0-dev

• Instalar Proj-4

 sudo apt-get install -y proj-bin proj-data libproj-dev

• Instalar Postgresql (*Acá ocasionalmente causa problemas, no debería pasar nada si es que se instala de los repositorios oficiales de ubuntu)

 sudo apt-get install -y postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3 postgresql-server-dev-9.3

• Instalar plp-python

 sudo apt-get install -y postgresql-plpython-9.3

• Instalar PostGis, es posible que Ubuntu ya lo tenga instalado

 sudo apt-get install postgis postgresql-9.3-postgis-2.1

• Crear un template para la configuración básica de postgis para cartodb, para ello crear un archivo en /var/lib/postgresql/config_db.sh, yo lo hice de la siguiente forma

 sudo -u postgres vim /var/lib/postgresql/config_db.sh

Luego agregar las siguientes líneas al archivo

 #!/usr/bin/env bash
 POSTGIS_SQL_PATH=/usr/share/postgresql/9.3/contrib/postgis-2.1
 createdb -E UTF8 template_postgis
 createlang -d template_postgis plpgsql
 psql -d postgres -c \ "UPDATE pg_database SET datistemplate='true' WHERE datname='template_postgis'"
 psql -d template_postgis -f $POSTGIS_SQL_PATH/postgis.sql
 psql -d template_postgis -f $POSTGIS_SQL_PATH/spatial_ref_sys.sql
 psql -d template_postgis -f $POSTGIS_SQL_PATH/legacy.sql
 psql -d template_postgis -f $POSTGIS_SQL_PATH/rtpostgis.sql
 psql -d template_postgis -f $POSTGIS_SQL_PATH/topology.sql
 psql -d template_postgis -c "GRANT ALL ON geometry_columns TO PUBLIC;"
 psql -d template_postgis -c "GRANT ALL ON spatial_ref_sys TO PUBLIC;"

Finalmente ejecutarlo como usuario postgres

 sudo -u postgres sh /var/lib/postgresql/config_db.sh

Para comprobar que todo esté correcto, se puede ejecutar:

 sudo -u postgres psql --command="\l"

• Instalar Ruby 1.9.3

 \curl -L https://get.rvm.io | bash
 source /home/<usuario>/.rvm/scripts/rvm
 rvm install 1.9.3

• Instalar Node, Redis, ImageMagick

 sudo apt-get install -y nodejs npm redis-server imagemagick

• Instalar als dependencias de python de cartodb

 sudo apt-get install -y python-setuptools python-dev
 sudo easy_install pip

entrar al archivo cartodb/python_requirements.txt y borrar la línea de Gdal (ya se instaló previamente)

 sudo pip install -r cartodb/python_requirements.txt
 sudo pip install -e git+https://github.com/RealGeeks/python-varnish.git@0971d6024fbb2614350853a5e0f8736ba3fb1f0d#egg=python-varnish

• Instalar Mapnik (un dolor de cabeza)

En caso de que, por alguna razón se agregó el ppa de mapnik, eliminarlo.

 sudo apt-add-repository --remove ppa:cartodb/mapnik

Luego

 sudo apt-get install -y libmapnik-dev python-mapnik mapnik-utils

• Instalar la API SQL de cartoDB (otro dolor de cabeza) Observar si causa algún error en comando npm install

 sudo apt-get install -y nodejs-legacy
 git clone git://github.com/CartoDB/CartoDB-SQL-API.git
 cd CartoDB-SQL-API
 git checkout master 
 npm install
 mv config/environments/test.js.example config/environments/test.js
 mv config/environments/development.js.example config/environments/development.js
 cd ..

• Instalar Windshaft-cartodb (este me causó más de una jaqueca) Observar si causa algún error en comando npm install

 sudo apt-get install -y libcairo2-dev libpango1.0-dev libjpeg8-dev libgif-dev
 git clone git://github.com/CartoDB/Windshaft-cartodb.git
 cd Windshaft-cartodb
 git checkout master
 npm install
 mv config/environments/development.js.example config/environments/development.js
 cd ..

• Configurar conexiones locales con postgres

Editar el archivo /etc/postgresql/9.3/main/pg-hba.conf, cambiar todo tipo de método de autentificación a "trust"

 local   all             postgres                                trust
 host    all             all             127.0.0.1/32            trust

Luego reiniciar el servicio de postgres

 sudo service postgresql restart

• Utilidades extras

 git clone https://github.com/CartoDB/pg_schema_triggers.git && cd pg_schema_triggers && sudo make all install && sudo sed -i "/#shared_preload/a shared_preload_libraries = 'schema_triggers.so'" /etc/postgresql/9.3/main/postgresql.conf && cd ..
 git clone --branch 0.5.2 https://github.com/CartoDB/cartodb-postgresql && cd cartodb-postgresql && PGUSER=postgres sudo make install && cd ..

• Configuración inicial de CartoDB antes de su ejecución

Aquí se coloca el nombre del subdominio de cartodb

 export SUBDOMAIN=carto
 cd cartodb
 rvm use 1.9.3@cartodb --create && bundle install

• Tomar café, nuevamente

 mv config/app_config.yml.sample config/app_config.yml

Editar en el mismo archivo la línea:

 developers_host:    'http://development.localhost.lan:3000'

Por el subdominio que se haya ingresado

 developers_host:    'http://carto.localhost.lan:3000'

Luego ejecutar

 mv config/database.yml.sample config/database.yml

Y editar en caso de que el usuario postgres se le haya cambiado su contraseña. (De todas formas no es necesario)

 echo "127.0.0.1 ${SUBDOMAIN}.localhost.lan" | sudo tee -a /etc/hosts
 echo "127.0.0.1 admin.localhost.lan" | sudo tee -a /etc/hosts
 echo "127.0.0.1 development.localhost.lan" | sudo tee -a /etc/hosts

• Crear usuario para cartodb

 redis-server &
 sh script/create_dev_user ${SUBDOMAIN}

Resar para que todo salga bien

• Iniciar todos los servicios

 bundle exec script/resque
 bundle exec rails s -p 3000
 cd ~
 cd cartodb-sql-api && node app.js &
 cd ..
 cd windshaft-cartodb && node app.js &

• FINALMENTE

Ingresar a carto.localhost.lan:3000 para visualizar la maravilla