watonomous.github.io

[ Software Division : Global Mapping ]

Created by [ Rowan Dempster], last modified by [ John Phillips] on Mar 19, 2020

Setting up the big SF database

The global mapping uses the HERE Maps data and PostgreSQL database.

Download and setup the PostgreSQL database

This installs the database:

Installing Database

sudo apt-get update
sudo apt-get install postgresql postgresql-contrib

You can access the database by switching to the postgres account.

Setup the Database

# This logs you into your postgres@<user_machine>, type exit to exit 
sudo -i -u postgres
# Create the database
createdb '1_1_CA_SF'
# Enter the SQL environment \q to exit
psql
# set password as "root" when prompted
\password

This will prompt for a new password. Set this to 'root'

After all that, the database should be set up like this:

user:postgres
database name:1_1_CA_SF
password:root

Set up your database to a spatial database
We need our database to be a spatial database to store the geometry information.
We use PostGIS to do this.

Install Postgis

# Make sure to not be in the postgres environment
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt xenial-pgdg main" >> /etc/apt/sources.list'
wget --quiet -O - http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -
sudo apt update
sudo apt-get install postgis

After installing, run this command inside your database (after \c 1_1_CA_SF):

From outside of the postgres environment:

Creating Postgis extension

sudo -i -u postgres
psql
# Database from earlier
\c 1_1_CA_SF
# Note this should give some sort of feedback on success
CREATE EXTENSION postgis;

Note: if the above fails with:
`ERROR:  could not open extension control file "/usr/share/postgresql/9.5/extension/postgis.control": No such file or directory`
try:
`sudo apt-get install postgresql-9.5-postgis-scripts`

Download the data
The map data is stored in the WATonomous Google Drive under Software/HERE maps/1_1_CA_SF: Download here: Google Drive
The 1_1_CA_SF folder contains the data for one section of the US.
The rest of North America data is stored inside the WATonomous Embedded Laptop.
The manual for this data is in User-Guide.zip. Global Mapping uses the NAVSTREETS data.

Store the data in the PostgreSQL database
The data we want is stored in the .shp files.
PostGIS has a command that allows us to turn .shp files into PostgreSQL tables.

// After unzipping the 1_1_CA_SF folder, we can access the user info on your machine from the postgres env.
cd /home/<username>/Downloads/1_1_CA_SF

Run these commands while on the postgres account and in the folder where all the .shp files are stored:

This will take a while

shp2pgsql -I -s 4326 -W "latin1" Streets.shp streets | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" AltStreets.shp altstreets | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" Zlevels.shp zlevels | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" PointAddress.shp pointaddress | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" AutoSvc.shp autosvc | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" Business.shp business | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" CommSvc.shp commsvc | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" EduInsts.shp eduinsts | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" Entertn.shp entertn | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" FinInsts.shp fininsts | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" Hospital.shp hospital | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" NamedPlc.shp namedplc | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" Restrnts.shp restrnts | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" Shopping.shp shopping | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" TransHubs.shp transhubs | psql -d 1_1_CA_SF -U postgres
shp2pgsql -I -s 4326 -W "latin1" TravDest.shp travdest | psql -d 1_1_CA_SF -U postgres

This will take a while.

Index the id
Index the node_id, link_id, and address in the database to make the query much faster.

Index the Id

# First go into your database within the database:
psql
\c 1_1_CA_SF
# Database commands for Indexes:
CREATE INDEX node_id_index ON zlevels(node_id);
CREATE INDEX st_link_id_index ON streets(link_id);
CREATE INDEX pa_link_id_index ON pointaddress(link_id);
CREATE INDEX address_index ON pointaddress(address);

# The only zlevels we will use are the intersection ones.
# We can delete all the non-intersection zlevels to speed up the routing:

DELETE FROM zlevels WHERE intrsect = 'N';

Installing QGIS 3

We use QGIS 3 to display our data.
\

QGIS3 Installation

# In your user (outside of postgres env)
sudo sh -c 'echo "deb https://qgis.org/ubuntugis-ltr xenial main" >> /etc/apt/sources.list'
sudo sh -c 'echo "deb-src https://qgis.org/ubuntugis-ltr xenial main" >> /etc/apt/sources.list'
sudo gpg  --keyserver hkp://keyserver.ubuntu.com:80 --recv 51F523511C7028C3
sudo gpg --export --armor 51F523511C7028C3 | sudo apt-key add -
sudo sh -c 'echo "deb https://qgis.org/ubuntugis  xenial main" >> /etc/apt/sources.list'
sudo sh -c 'echo "deb-src https://qgis.org/ubuntugis xenial main" >> /etc/apt/sources.list'
sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
sudo apt-get update
sudo apt-get install qgis

If the last step of the above doesn't work, try: (OUTDATED, KEEPING HERE FOR SOURCE)
--QGIS 3 Ubuntu 16.04

sudo sh -c 'echo "deb https://qgis.org/ubuntugis  xenial main" >> /etc/apt/sources.list'
sudo sh -c 'echo "deb-src https://qgis.org/ubuntugis xenial main" >> /etc/apt/sources.list'
sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable

// These lines might not work, refer to latest instructions
wget -O - https://qgis.org/downloads/qgis-2017.gpg.key | gpg --import
gpg --fingerprint CAEB3DC3BDF7FB45
gpg --export --armor CAEB3DC3BDF7FB45 | sudo apt-key add -

sudo apt-get update
sudo apt-get update && sudo apt-get install qgis python-qgis qgis-plugin-grass

Source: https://gis.stackexchange.com/questions/276539/installing-qgis3-on-ubuntu-16-04-xenial-dependency-problem

Running the Code

There are some python libraries you need.
If you don't already have these libraries, install them with these commands:

Extra Libraries

sudo apt-get install python3-psycopg2
sudo apt-get install python3-geopy
sudo apt-get install python3-pyqt5

Code is located in the global_mapping_2019 branch of path planning repository. Routing code is located in the global_mapping folder. Now you should be able to run the code by simply doing:

python3 launch.py

Setting up the WATO test track database

Follow similar instructions to the above instructions for the SF database with a couple of changes:

The map files to download and use are stored at: https://drive.google.com/drive/u/1/folders/1LUZAjKHVStQEB4LdlBUjHQ5M5QrzOWUE
If you've already gone through the SF database setup instructions, you won't need to install all the tools like pyqt5 again this time around, because you already have them installed
Since you're setting up a different database, you need to give it a different name. The name we use for the WATO test track database is WATO_TEST_TRACK, so everywhere there is a 1_1_CA_SF in the SF database setup instructions needs to be replaced with WATO_TEST_TRACK
There are some shp2pgsql instructions that you won't need to run, because there are fewer .shp files for the WATO test track map than for the SF map. Similarly, there are some CREATE INDEX instructions that you won't need to run; don't worry about determining which ones you don't need to run, just run all of them and some of them will fail, and you can continue passed the failures

Document generated by Confluence on Dec 10, 2021 04:01

Atlassian