Difference between revisions of "Accessing server infra"

From stgo
Jump to: navigation, search
(other stuff)
(using VirtualBox)
 
(28 intermediate revisions by one user not shown)
Line 8: Line 8:
 
** the version of ssl can be obtained using <code>sudo openssl version -a</code>
 
** the version of ssl can be obtained using <code>sudo openssl version -a</code>
 
* upgrade <code>apt-get upgrade PackageName</code> or: <code>sudo apt-get install libssl1.0.0</code>
 
* upgrade <code>apt-get upgrade PackageName</code> or: <code>sudo apt-get install libssl1.0.0</code>
 +
* if I get the message: ''"n packages can be updated"'', then [http://askubuntu.com/questions/196768/how-to-install-updates-via-command-line do the following]:
 +
** sudo apt-get update        # Fetches the list of available updates
 +
** sudo apt-get upgrade      # Strictly upgrades the current packages
 +
** sudo apt-get dist-upgrade  # Installs updates (new ones)
 +
** perhaps reboot
  
 
== git stuff ==
 
== git stuff ==
Line 72: Line 77:
 
* show list of VMs: <code>VBoxManage list vms</code>
 
* show list of VMs: <code>VBoxManage list vms</code>
 
* powerdown a VM: <code>VBoxManage controlvm opensuse132 poweroff</code>
 
* powerdown a VM: <code>VBoxManage controlvm opensuse132 poweroff</code>
* emergency power-off a VM that does not properly shutdown and when restarting there is a message such as:
+
** emergency power-off a VM that does not properly shutdown and when restarting there is a message such as:
 
  VBoxHeadless: error: The machine 'opensuse132' is already locked for a session (or being unlocked)
 
  VBoxHeadless: error: The machine 'opensuse132' is already locked for a session (or being unlocked)
 
  VBoxHeadless: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component Machine, interface IMachine, callee nsISupports
 
  VBoxHeadless: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component Machine, interface IMachine, callee nsISupports
 
  VBoxHeadless: error: Context: "LockMachine(session, LockType_VM)" at line 960 of file VBoxHeadless.cpp
 
  VBoxHeadless: error: Context: "LockMachine(session, LockType_VM)" at line 960 of file VBoxHeadless.cpp
 
: => solve with: <code>VBoxManage startvm opensuse132 --type emergencystop</code>
 
: => solve with: <code>VBoxManage startvm opensuse132 --type emergencystop</code>
 +
* start VM: <code>VBoxHeadless -s opensuse132 --vrde on &</code>
 +
* add user to VBoxuser group to be able to install a VM: <ode>sudo usermod -a -G vboxusers usernane</code>
 +
: => to check in what groups the user is: <code>groups username</code>
  
== OpenTripPlanner ==
+
== OpenTripPlanner (for older version: 0.9, not 0.20/1.0) ==
 
* start with "sh start-server.sh"; use "&" at the end to keep the process running
 
* start with "sh start-server.sh"; use "&" at the end to keep the process running
 
* make sure that winstone.jar is executable, with chmod 755
 
* make sure that winstone.jar is executable, with chmod 755
Line 87: Line 95:
 
== Unix / Linux / Ubuntu commands ==
 
== Unix / Linux / Ubuntu commands ==
 
http://www.computerhope.com/issues/ch000623.htm
 
http://www.computerhope.com/issues/ch000623.htm
 +
 +
'''assign static IP'''
 +
* for Ubuntu see see http://www.howtogeek.com/howto/ubuntu/change-ubuntu-server-from-dhcp-to-a-static-ip-address/
  
 
'''connect to server'''
 
'''connect to server'''
Line 129: Line 140:
 
'''other useful commands'''
 
'''other useful commands'''
 
* used ports: <code>netstat -a</code>
 
* used ports: <code>netstat -a</code>
 +
* monitor network traffic: iftop
 
* mac addresses: <code>ifconfig -a</code>
 
* mac addresses: <code>ifconfig -a</code>
 
* creating an ISO file from a DVD: mount the DVD. If mounted under /dev/sr0 use: <code>sudo cat /dev/sr0 > /home/ssteinig/example.iso</code>
 
* creating an ISO file from a DVD: mount the DVD. If mounted under /dev/sr0 use: <code>sudo cat /dev/sr0 > /home/ssteinig/example.iso</code>
 
* what to do if I get ''"There are stopped jobs"''? see http://unix.stackexchange.com/questions/116959/there-are-stopped-jobs-on-bash-exit
 
* what to do if I get ''"There are stopped jobs"''? see http://unix.stackexchange.com/questions/116959/there-are-stopped-jobs-on-bash-exit
 +
** use <code>jobs</code> to check what jobs are still around
 +
** use <code>fg</code> + job# to get the job to be able to terminate it later
 +
 +
'''powerdown (sleep) and wakeup'''
 +
* <code>sudo rtcwake -m disk -s 120 &</code> ...this worked with Camila and put her 120 secs to sleep
 +
* the stuff below did not work as expected ("rtcwake" not found msg)
 +
** for a scheduled shutdown of the servers I put the command in the sudo crontab, like this:
 +
*: <code>MAILTO=ssteiniger --- at --- uc.cl</code>   
 +
*: <code>0 22 12 2 * rtcwake -m disk -s 108000 &</code>
 +
:: => this shall send an email to me, and executes the rtcwake command at 12. Feb. 22:00 (each year, I guess)
 +
* for ''rtcwake'' details see: http://www.howtogeek.com/121241/how-to-make-your-linux-pc-wake-from-sleep-automatically/
 +
* for crontab details see: http://www.thegeekstuff.com/2009/06/15-practical-crontab-examples/
 +
 +
'''sending email from server'''
 +
* install ''mailutils'' (apt-get)
 +
* test: <code>mail -s "test email at: `date +%m.%d.%Y.%H.%M`" myemail@email.de</code>
 +
: => I used this to send me an email on reboot from cedeusgeonode
  
 
== Installing lightweight XFCE Desktop GUI on Ubuntu Server ==
 
== Installing lightweight XFCE Desktop GUI on Ubuntu Server ==
Line 147: Line 176:
 
* connect via Windows Remote Desktop using the IP etc.  
 
* connect via Windows Remote Desktop using the IP etc.  
 
* if not needed, switch off the remote desktop server using: <code>sudo service xrdp stop</code>
 
* if not needed, switch off the remote desktop server using: <code>sudo service xrdp stop</code>
 +
 +
== Apache Access Log Analysis ==
 +
* access to the servers is logged by Apache in the following 2 files (apart from error log):
 +
** <code>/var/log/apache2/access.log</code> This is for access to the server. --> for seeing it live use: <code>tail -f /var/log/apache2/access.log</code>
 +
** <code>/var/log/apache2/other_vhosts_access.log</code> This helps to see access to my VMs for instance. --> for seeing it live use: <code>tail -f /var/log/apache2/other_vhosts_access.log</code>
 +
* I also installed on CedeusGeoNode a ''Top'' like open source log analyzer named [http://goaccess.io/ GoAccess]:
 +
** to run it simply type: <code>goaccess -f /var/log/apache2/access.log -a</code> (see http://goaccess.io/faq#howto-run)
 +
** and for the "other vhosts" <code>goaccess -f /var/log/apache2/other_vhosts_access.log -a</code>
 +
 +
== Setting Up a CEDEUS Subdomain ==
 +
Its possible to create a CEDEUS subdomain using two methods, provided by Dreamhost
 +
* under ''Domain'' entry create a (sub)domain redirect to the VM directly, i.e. inlcuding IP:port number
 +
* under '''Domain''' clicking on '''DNS''' under the ''cedeus.cl'' entry, one can add an "A Record" specifying the the subdomain name (i.e. pubs.cedeus.cl) and the server IP, i.e. 146.155.17.18. Then on the server, the apache default sites file needs to be modified and a VirtualHost + ProxyPass entry added
 +
sudo nano /etc/apache2/sites-available/000-default.conf
 +
and then add a complete new entry (not modifying the main one)
 +
#Omeka / Pubs.CEDEUS.cl
 +
<VirtualHost *:80>
 +
    ServerName pubs.cedeus.cl
 +
    ProxyRequests Off
 +
    ProxyPreserveHost On
 +
    ProxyPass / http://146.155.17.18:20080/
 +
    ProxyPassReverse / http://146.155.17.18:20080/
 +
</VirtualHost>
 +
Afterwards on should(???) also modify for the VM the Apache Site setting and define the ServerName (or ServerAlias), i.e. ''pubs.cedeus.cl'' for the example.
  
 
== other stuff ==
 
== other stuff ==
Line 154: Line 207:
 
** files by using: <code>zip newzipfile.zip fileToZip.obj</code>
 
** files by using: <code>zip newzipfile.zip fileToZip.obj</code>
 
** whole folders by using: <code>tar -cvzf tarfilename.tgz /folder/to/compress</code>
 
** whole folders by using: <code>tar -cvzf tarfilename.tgz /folder/to/compress</code>
 +
** using 7z
 +
*** install 7zip: <code>sudo apt-get install p7zip-full</code>
 +
*** on how to use 7zip: see [http://www.dotnetperls.com/7-zip-examples here] and [http://www.ibm.com/developerworks/community/blogs/6e6f6d1b-95c3-46df-8a26-b7efd8ee4b57/entry/how_to_use_7zip_on_linux_command_line144?lang=en here]
 +
*** e.g. to pack: <code>7z a archivefile.7z archivetocompress.vdi</code>
 
* '''uncompress''':
 
* '''uncompress''':
 
** zip files, using: <code>unzip filename.zip</code>,  
 
** zip files, using: <code>unzip filename.zip</code>,  
Line 161: Line 218:
 
* start PhpStorm on my machine: <code>sh ./dev/PhpStorm-133.1777/bin/phpstorm.sh</code>
 
* start PhpStorm on my machine: <code>sh ./dev/PhpStorm-133.1777/bin/phpstorm.sh</code>
 
* [[Installing New Harddisk in Ubuntu]]
 
* [[Installing New Harddisk in Ubuntu]]
* install 7zip: <code>sudo apt-get install p7zip-full</code>
+
* [[Solving Ubuntu GRUB issue]]
** on how to use 7zip: see [https://www.ibm.com/developerworks/community/blogs/6e6f6d1b-95c3-46df-8a26-b7efd8ee4b57/entry/how_to_use_7zip_on_linux_command_line144?lang=en here]
+
* check network bandwidth use: use ''iftop'' (<code>sudo apt-get install iftop</code>)
** e.g. to pack: <code>7z a archivefile.7z archivetocompress.vdi</code>
+
 
 +
=== resolve nginx 502 FIC ===
 +
resolve error 502 for FIC web front
 +
* <code>cd /var/www/fic/</code>
 +
* <code>pm2 start npm --name "fic" -- start</code> (yes, with space before start)
 +
* reload webpage
 +
 
 +
=== resolve nginx 502 Indicadores or Stride ===
 +
resolve error 502 for Indicadores web front
 +
* <code>cd /home/csfuente/Indicadores</code>  => correcion, debe ser "<code>indicadores_REST-API</code>"
 +
* <code>source bin/activate</code>
 +
* <code>gunicorn --bind 0.0.0.0:8000 indicadores.wsgi</code>
 +
* reload webpage
 +
 
 +
for Stride it would be instead:
 +
* <code>cd /home/csfuente/stride</code>
 +
* <code>source bin/activate</code>
 +
* <code>gunicorn --bind 0.0.0.0:8000 stride.wsgi</code>
 +
* reload webpage

Latest revision as of 16:18, 11 April 2022

>> return to Cedeus IDE


apt-get

an intro can be found here

  • see what is installed dpkg -l
  • info on a particular package apt-cache show PackageName
    • the version of ssl can be obtained using sudo openssl version -a
  • upgrade apt-get upgrade PackageName or: sudo apt-get install libssl1.0.0
  • if I get the message: "n packages can be updated", then do the following:
    • sudo apt-get update # Fetches the list of available updates
    • sudo apt-get upgrade # Strictly upgrades the current packages
    • sudo apt-get dist-upgrade # Installs updates (new ones)
    • perhaps reboot

git stuff

  • git diff
  • git pull
  • git status
  • git reset --hard HEAD
  • git reset HEAD~1
  • git reset HEAD opentripplanner-api-webapp/src/main/java/org/opentripplanner/api/ws/analyst/IsoChrone.java
  • git log -p
  • git add opentripplanner-api-webapp/src/main/java/org/opentripplanner/api/ws/analyst/IsoChrone.java
  • git commit -m 'added debug code to be able to figure why some edges are traversed to fast (aside fromom u-shaped roads)'
  • gitk
  • git push origin master
  • git show-ref
  • git stash list
  • git log origin/master..master
  • git fetch upstream
  • git remote add upstream https://github.com/openplans/OpenTripPlanner.git
  • git remote show origin
  • git remote set-url origin git://github.com/mentaer/OpenTripPlanner.git
  • git remote set-url origin git@github.com:mentaer/OpenTripPlanner.git
  • git diff > /Volumes/WINDOOF/walkingslopeeffects.patch
  • patch -p1 -R < IsoChrone2.patch
  • git apply --check /Volumes/WINDOOF/IsoChrone.patch

maven stuff

  • mvn package -DskipTests

postgres stuff

  • echo 'export PATH=/Library/PostgreSQL/9.2/bin:$PATH' >> ~/.bash_profile
  • psql
  • sudo mkdir /Library/PostgreSQL/9.2/data/
  • sudo chown postgres /Library/PostgreSQL/9.2/data/
  • sudo -u postgres initdb -D /Library/PostgreSQL/9.2/data/
  • service postgresql start
  • sudo -u postgres pg_ctl -D /Library/PostgreSQL/9.2/data -l logfile start
  • sudo -u postgres postgres -D /Library/PostgreSQL/9.2/data &
  • sudo -u postgres createdb santiagotest
  • sudo -u postgres createuser -P -s -e mentaer
  • psql santiagotest
  • to access via terminal use for instance psql -U sstein -d geonode -h localhost

using VirtualBox

  • a VM contains 3 files: .img + .vbox (an editable xml) + .vbox-prev
  • " use ssh -X sstein..@xxx to enable a visual client
  • vboxgtk (start visual client)

create and start a VM from comandline:

  • VBoxManage createvm --name ubuntu --register
  • VBoxManage modifyvm ubuntu --ostype ubuntu_64 --cpus 4
  • VBoxManage modifyvm ubuntu --memory 8000
  • VBoxManage startvm ubuntu

other:

  • options of ip-addressing/web access: NAT (Network Address Translation) or Bridge (see also this blog entry)
  • downloading .iso archives/vm from OSGeo (use commands curl or wget -c), e.g.:
wget -c "http://downloads.sourceforge.net/project/osgeo-live/7.0/osgeo-live-7.0.iso?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fosgeo-live%2Ffiles%2F%2F7.0%2Fosgeo-live-7.0.iso%2Fdownload&ts=1384444616&use_mirror=ufpr"
  • showing list of running VMs: VBoxManage list runningvms
  • show list of VMs: VBoxManage list vms
  • powerdown a VM: VBoxManage controlvm opensuse132 poweroff
    • emergency power-off a VM that does not properly shutdown and when restarting there is a message such as:
VBoxHeadless: error: The machine 'opensuse132' is already locked for a session (or being unlocked)
VBoxHeadless: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component Machine, interface IMachine, callee nsISupports
VBoxHeadless: error: Context: "LockMachine(session, LockType_VM)" at line 960 of file VBoxHeadless.cpp
=> solve with: VBoxManage startvm opensuse132 --type emergencystop
  • start VM: VBoxHeadless -s opensuse132 --vrde on &
  • add user to VBoxuser group to be able to install a VM: <ode>sudo usermod -a -G vboxusers usernane</code>
=> to check in what groups the user is: groups username

OpenTripPlanner (for older version: 0.9, not 0.20/1.0)

  • start with "sh start-server.sh"; use "&" at the end to keep the process running
  • make sure that winstone.jar is executable, with chmod 755
  • define in the start-server.sh file also where the webapps folder is located (physical location), so winstone knows that
  • define in the WEB-INF/classes/data-sources.xml file where the graph is stored (physical location) - the files is within /otp/webapps/opentripplanner-api-webapp.war (or: the deflated war folder)
  • kill winstone server (or OTP) process: use
    ps aux | grep winstone
    to find the associated PID, and use kill pid

Unix / Linux / Ubuntu commands

http://www.computerhope.com/issues/ch000623.htm

assign static IP

connect to server

  • ssh stefan@136.159.xxx.xxx
  • ssh sstein@146.155.xxx.xxx.
  • transfer a file to a remote computer: scp /home/username/ftw.txt user@example.com:/home/username/

see processes and find stuff

user management commands

memory related commands

other useful commands

  • used ports: netstat -a
  • monitor network traffic: iftop
  • mac addresses: ifconfig -a
  • creating an ISO file from a DVD: mount the DVD. If mounted under /dev/sr0 use: sudo cat /dev/sr0 > /home/ssteinig/example.iso
  • what to do if I get "There are stopped jobs"? see http://unix.stackexchange.com/questions/116959/there-are-stopped-jobs-on-bash-exit
    • use jobs to check what jobs are still around
    • use fg + job# to get the job to be able to terminate it later

powerdown (sleep) and wakeup

  • sudo rtcwake -m disk -s 120 & ...this worked with Camila and put her 120 secs to sleep
  • the stuff below did not work as expected ("rtcwake" not found msg)
    • for a scheduled shutdown of the servers I put the command in the sudo crontab, like this:
    MAILTO=ssteiniger --- at --- uc.cl
    0 22 12 2 * rtcwake -m disk -s 108000 &
=> this shall send an email to me, and executes the rtcwake command at 12. Feb. 22:00 (each year, I guess)

sending email from server

  • install mailutils (apt-get)
  • test: mail -s "test email at: `date +%m.%d.%Y.%H.%M`" myemail@email.de
=> I used this to send me an email on reboot from cedeusgeonode

Installing lightweight XFCE Desktop GUI on Ubuntu Server

  • use sudo apt-get install xubuntu-desktop to install XFCE client (see here).
  • this will also install xauth, which allows remote window clients
  • disable start of GUI-based login on system startup with Grub (taken from here):
    • Change (edit) in /etc/default/grub file: GRUB_CMDLINE_LINUX_DEFAULT="text"
    • then do sudo update-grub
  • start the desktop GUI session using startx
  • end the desktop GUI session using "logout" button

Enabling Remote Desktop for Windows

got the info from here

  • install XRDP: sudo apt-get install xrdp
  • connect via Windows Remote Desktop using the IP etc.
  • if not needed, switch off the remote desktop server using: sudo service xrdp stop

Apache Access Log Analysis

  • access to the servers is logged by Apache in the following 2 files (apart from error log):
    • /var/log/apache2/access.log This is for access to the server. --> for seeing it live use: tail -f /var/log/apache2/access.log
    • /var/log/apache2/other_vhosts_access.log This helps to see access to my VMs for instance. --> for seeing it live use: tail -f /var/log/apache2/other_vhosts_access.log
  • I also installed on CedeusGeoNode a Top like open source log analyzer named GoAccess:
    • to run it simply type: goaccess -f /var/log/apache2/access.log -a (see http://goaccess.io/faq#howto-run)
    • and for the "other vhosts" goaccess -f /var/log/apache2/other_vhosts_access.log -a

Setting Up a CEDEUS Subdomain

Its possible to create a CEDEUS subdomain using two methods, provided by Dreamhost

  • under Domain entry create a (sub)domain redirect to the VM directly, i.e. inlcuding IP:port number
  • under Domain clicking on DNS under the cedeus.cl entry, one can add an "A Record" specifying the the subdomain name (i.e. pubs.cedeus.cl) and the server IP, i.e. 146.155.17.18. Then on the server, the apache default sites file needs to be modified and a VirtualHost + ProxyPass entry added
sudo nano /etc/apache2/sites-available/000-default.conf

and then add a complete new entry (not modifying the main one)

#Omeka / Pubs.CEDEUS.cl
<VirtualHost *:80>
   ServerName pubs.cedeus.cl
   ProxyRequests Off
   ProxyPreserveHost On
   ProxyPass / http://146.155.17.18:20080/
   ProxyPassReverse / http://146.155.17.18:20080/
</VirtualHost>

Afterwards on should(???) also modify for the VM the Apache Site setting and define the ServerName (or ServerAlias), i.e. pubs.cedeus.cl for the example.

other stuff

  • gdalinfo cgydem_fromgrid.tif
  • if computer crashs: boot from a LiveDVD and the run fsck (see http://www.thegeekstuff.com/2012/08/fsck-command-examples/)
  • compress:
    • files by using: zip newzipfile.zip fileToZip.obj
    • whole folders by using: tar -cvzf tarfilename.tgz /folder/to/compress
    • using 7z
      • install 7zip: sudo apt-get install p7zip-full
      • on how to use 7zip: see here and here
      • e.g. to pack: 7z a archivefile.7z archivetocompress.vdi
  • uncompress:
    • zip files, using: unzip filename.zip,
    • bz2 files with: bunzip2 filename.bz2
    • tar files: tar -C / -xvzf tarfilename.tgz . Attention !!! This will restore the data exactly in the same place on the computer, and overwrite existing folders and files.
  • VBoxManage Windows 8 - 64bit createvm code : "Windows8_64"
  • start PhpStorm on my machine: sh ./dev/PhpStorm-133.1777/bin/phpstorm.sh
  • Installing New Harddisk in Ubuntu
  • Solving Ubuntu GRUB issue
  • check network bandwidth use: use iftop (sudo apt-get install iftop)

resolve nginx 502 FIC

resolve error 502 for FIC web front

  • cd /var/www/fic/
  • pm2 start npm --name "fic" -- start (yes, with space before start)
  • reload webpage

resolve nginx 502 Indicadores or Stride

resolve error 502 for Indicadores web front

  • cd /home/csfuente/Indicadores => correcion, debe ser "indicadores_REST-API"
  • source bin/activate
  • gunicorn --bind 0.0.0.0:8000 indicadores.wsgi
  • reload webpage

for Stride it would be instead:

  • cd /home/csfuente/stride
  • source bin/activate
  • gunicorn --bind 0.0.0.0:8000 stride.wsgi
  • reload webpage