Why This Matters for Data Engineers

Data engineering work is rarely confined to a single machine. Pipelines pull from remote databases, push to cloud storage, and run on servers accessible only through a terminal. Networking knowledge is therefore empowering, helping you debug issues with these varied services more independently.

NAT: Network Address Translation

NAT, or Network Address Translation, is the mechanism that allows devices on a private network to communicate with external networks — and with each other.

In VirtualBox (the open-source hypervisor software allowing me to run VMs), a VM is given a NAT adapter by default. This means the VM can reach the internet through the host machine, and the host can reach the VM through a NAT. What it does not allow by default is VM-to-VM communication. For that, you need to configure a NAT Network — a shared virtual network that multiple VMs can join. This can be done through the VirtualBox settings UI.

Network Interfaces: ip a and ifconfig

A network interface is a point of connection between a device and a network. This can be wifi, ethernet, loopback/localhost, etc.

Before setting up SSH or port forwarding, it helps to understand what your network interfaces look like. Two commands are commonly used for this:

• ip a — the modern standard on Linux
• ifconfig — the older Unix/macOS equivalent, part of the net-tools package

Both display information about your active network interfaces, including interface names (e.g., eth0, lo), IPv4 and IPv6 addresses, MAC addresses, and interface status. They usually show the private ip address of devices on network.

Note: if you want to use ifconfig on a Linux machine where it isn’t preinstalled, run sudo apt install net-tools and then add it to the path. To add it to the path, edit your ~/.bashrc file and append export PATH="$PATH:/sbin:/usr/sbin" to it. Then, restart your terminal or reload the bashrc file with source ~/.bashrc

What addresses will you see?

The addresses you see fall into a few categories:

Private (RFC 1918) addresses — routable only within a local network:

• 10.0.0.0/8
• 172.16.0.0/12
• 192.168.0.0/16

Other common addresses:

• 127.0.0.1 / ::1 — loopback, also known as localhost. Traffic sent here is intercepted by the OS and looped back to the local machine — it never touches a NIC or a router.
• 169.254.x.x — link-local, OR APIPA. APIPA is typically assigned when DHCP fails. DHCP is the Dynamic Host Configuration Protocol. It’s the system that automatically assigns IP addresses to devices when they join a network, so you don’t have to set one. APIPA stands for Automatic Private IP Addressing. It’s a fallback mechanism some OS’s use when a device can’t reach a DHCP server. In practice, seeing a 169.254.x.x address is usually a sign that something is wrong — your machine couldn’t find the DHCP server, meaning the router is down, the network cable is unplugged, or there’s some other connectivity issue. Conversely, link-local refers to addresses that are only valid on a single network segment — they can’t be routed beyond the directly connected link (cable, Wi-Fi, etc.). Link-local does not necessarily mean that there is a connectivity issue.
• fd00::/8 or fe80::/10 — IPv6 private and link-local ranges

On a typical home or office machine, ip a will show only private IPs. On a cloud VM (AWS, GCP, etc.), you will often see a private IP on the interface even if the machine has a public IP — the public IP is handled by the cloud provider’s NAT layer and won’t appear in ip a at all.

To find your public IP, query an external service:

curl ifconfig.me      # returns whichever IP your request arrived from (may be either IPv4 or IPv6)
curl -4 ifconfig.me   # show IPv4
curl -6 ifconfig.me   # show IPv6

SSH: Secure Shell

SSH (Secure Shell) lets you control a remote system from the command line over an encrypted connection. If openssh-server is not already installed on your target machine, you can add it with:

sudo apt install openssh-server
sudo systemctl enable ssh

The default port for SSH (or other secured connections) on any machine is 22.

To connect:

ssh <username>@<ip-address>

By default, there is no direct route from the host machine to the VM’s private IP in VirtualBox. Port forwarding is the solution.

Port Forwarding

Port forwarding redirects traffic arriving at one address and port to a different address and port. When data arrives at a machine on a specific port, instead of that machine handling it directly, the traffic is forwarded elsewhere.

The Home Router Example

Your router has a single public IP. Behind it are many devices with private IPs, none of which are reachable directly from the internet. Port forwarding solves this:

1. Someone connects to your public IP on port 8080
2. Your router sees the incoming traffic and forwards it to a private IP on port 80
3. The server on that machine responds, and the router sends the reply back

The outside world only ever sees the public IP — the internal machine stays hidden.

Other common uses include:

• Hosting a game server at home
• SSH-ing into a home machine from outside your network
• Exposing a local web server to the internet

Port Forwarding with VirtualBox

To SSH from your host machine into a VirtualBox VM, configure port forwarding in the VM’s network settings:

Figure 1

Then SSH into the VM from your host:

ssh -p 22022 <username>@127.0.0.1

The -p flag specifies the host port you configured. The traffic hits your localhost on port 22022, gets forwarded by VirtualBox to the VM on port 22, and you have a shell.

Conclusion

Networking is one of those topics that feels irrelevant until you find yourself unable to explain why a pipeline can’t reach a database you know is running. The concepts here — NAT, private vs. public IPs, SSH, and port forwarding — come up often enough in data engineering work to make learning these foundations worthwhile, even when they aren’t the focus of what you’re building. I hope this serves as a useful starting point. And as always, happy coding!

In this post, I’ll walk through how to set up an Ubuntu VM in UTM and configure it so you can SSH into it from your macOS host. If you only need terminal access, SSH is recommended over using the UTM GUI for a smoother and more efficient experience.

1. Install UTM
2. Download your desired Ubuntu instance (use ARM compatible ISO’s for M-series Macs).
3. Follow all the steps found here: https://www.youtube.com/watch?v=1PL-0-5BNXs
4. Go to the UTM network settings for your new VM
5. Change the “Network Mode” to “Emulated VLAN”
6. Go to port forwarding and select these options:

Figure 1

1. Start your Ubuntu instance and log in
2. Ensure you can allow for other machines to ssh into the Ubuntu guest machine by running
   1. sudo apt install openssh-server
   2. sudo systemctl enable ssh
3. In your host terminal, run ssh -p 22022 <username>@localhost

Ugly Code Reduces Team Efficiency

Imagine a scenario where your colleague was just asked to enhance some code you built. As part of that work, new modules, classes, and functions must be created that will be reused elsewhere.

You have a different coding style than your coworker though, and while reviewing the diffs, you see a pull request marred with conflicting indentation patterns, varied casing styles, and in-line comments describing the functions inplace of docstrings (or worse, no documentation at all). Hopefully their code is not approved and merged by another engineer before you have a chance to complete your review.

This scenario is a reality on many engineering teams, and is not ~just~ an annoyance for developers that appreciate pretty code; rather, this reality can cause a multitude of issues including:

1. visual clutter that distracts from the actual code logic being written
2. non-standard style practices that cause IDEs to light up with lint warnings everywhere
3. reduced efficacy of IDE auto-complete functionality when searching for the correct variable, class, function or method name
4. negated functionality of IDEs ability to preview docstring descriptors for modules, classes, functions and methods
5. increased time spent deciphering code that could have been spent building features, fixing bugs, or doing (quite literally) anything else.

Overall, such discrepancies create a hinderance a team’s productivity and will only become more problematic as projects grow. Ugly code (and undocumented code- thank goodness for PEP-257) almost certaintly means longer feature-to-production timetables; it can even lead to bugs from misinterpretted code. Team-leads at a minimum should encourage adaptation of some guideline to adhere to. Preferably, we should attach linters and autoformatters to our pull requests and CI/CD pipelines.

…Enter the Black Formatter

Autoformatters and linters are tools designed to automatically format or check code according to a set of predefined style guidelines or rules; they can also help make code that was written by ten developers and a GPT look like it was written by just one. One of the most popular options for Python is the Black Formatter, which helps format your Python code according to PEP-8 industry standards, the most famous of the PEP guidelines.

In the words of its documentation,

      “By using Black, you agree to cede control over minutiae of hand-formatting. In       return, Black gives you speed, determinism, and freedom from pycodestyle       nagging about formatting. You will save time and mental energy for more important       matters.

      Black makes code review faster by producing the smallest diffs possible.       Blackened code looks the same regardless of the project you’re reading.       Formatting becomes transparent after a while and you can focus on the content       instead.”

If you use VS Code, you can install the Black Formatter as an extension: https://marketplace.visualstudio.com/items?itemName=ms-python.black-formatter.

You can also install black formatter as a CLI utility: pip install black.

Other Useful Linters and Extensions:

You can use other linters as either an alternative or in addition to Black. Some recommended ones include:

• flake8, flakeheaven, flake8-docstrings
• pylint
• ruff

Each has their own advantages.

Linter Usage and Enforcement:

There are many strategies for how to best enforce linting and formatting on software engineering teams. I will suggest an approach that works best for me, and show how this can be done using CI/CD.

Requirements:

To follow this tutorial in its entirety, you must:

1. Have VS Code installed
2. Have Python installed, and a basic understanding of python syntax
3. Install these VS Code Extensions:
   • Black formatter extension
   • Python extension
4. Have a GitHub account and public repository to hold examples
5. A basic understanding of git concepts, shell scripting, CI/CD, and preferably experience using GitHub Actions or Azure Pipelines

It will also be helpful- but not explicitly required- to use these tools locally as you work through this tutorial:

• black formatter CLI utility: https://github.com/psf/black
• act utility for running github actions locally: https://github.com/nektos/act

Workflow:

The easiest methods I have seen has been to first set up your IDE to autoformat your python code automatically upon saving.

After installing the Black Formatter VS Code Extension, I go into my VS Code user settings. On MacOS, these can usually be found at: /Users/<user>/Library/Application Support/Code/User/settings.json.

Add this to the json:

    "editor.formatOnSave": false,
    "[python]": {
        "editor.formatOnType": true,
        "editor.defaultFormatter": "ms-python.black-formatter",
        "editor.formatOnSave": true,
    },
    "black-formatter.showNotifications": "always"

With these settings, saving a python file will automatically be autoformatted according to Black style guides. If you keep formatOnSave set to false, you could alternatively right-click the editor, and select “format document” for the formatter to run.

On a team though, it can be hard to enforce contributor IDE settings. A CI/CD rule to check that code is styled according to team guidelines is a necessary second-check.

To properly test that this next piece works as expected, make sure to set the formatOnSave from the previous step to false. Then, put two Python files in your project directory (a src subdirectory is fine, as is the root- just keep it out of the .github folder). Keep one properly formatted, and the second improperly formatted. You can use these basic examples:

no_issues.py

# Properly formatted python code:
x = {"hello": "world"}

lint_this.py

# Improperly formatted python code, with added whitespace before the ending curly brace
x = {"hello": "world" }

Then, commit these changes to a branch that is based off of main, but is not main.

To check ONLY those Python files that have been modified against the main branch, first add this shell script to your .github folder under a “scripts” subdirectory:

modified_files.sh

# echos modified python files, but excludes those deleted
changed_files=$(git diff --diff-filter=d --name-only $(git merge-base HEAD remotes/origin/main) HEAD | grep .py)
echo ${changed_files}

Then, add these steps to your Actions workflow:

      - name: install black formatter
        run: pip install black

      - name: Save modified files to environmental variables
        id: get_modified_files
        run: echo MODIFIED_FILES=$(./.github/scripts/modified_files.sh) >> $GITHUB_ENV
      
      - name: return modified python files
        run: echo $(./.github/scripts/modified_files.sh)

      - name: Lint modified Python files with Black
        run: |
          if [ -z "$MODIFIED_FILES" ]; then
            echo "No Python files modified."
          else
            black --check --diff --color $MODIFIED_FILES
          fi

If there was any improperly formatted Python code, the deployment would fail. You can check this on GitHub Actions in your personal repository. Otherwise, to test this locally, first make sure you have the nektos/act CLI tool and run the act command in your terminal, with any options if desired. For example, I usually will run act <github_event_name> --container-architecture linux/amd64. For all options using this utility, please look into its documentation. For my own complete, working example on this workflow, you visit my Github repository: https://github.com/FreyGeospatial/github_actions.

Overall, this is a pretty simple example of what you can do to enforce a cleaner codebase on your team. I encourage you to explore other linters, including Flake8, or even ruff, the latter of which has gained popularity recently for being very computationally performant.

I hope you have found this helpful. If you have any comments, questions, or feedback- please reach out! And as always, happy coding!

Since my package was private, it could only be accessed by those with specific credentials- credentials which could authenticate that a user has correct access. One of the services requiring use of this package ran inside a continuously-running AWS ECS task. The ECS task ran from a docker image. In the Dockerfile used to create the image, we have code telling Docker to install some software packages. This is where I would run my pip install command to pull my package from Azure Artifacts.

Usually when you install a package from Pypi, you simply run pip install <package name>. If you want to specify a repository other than the default, you can use pip install <package name> --extra-index-url http://xyz123.com

From Azure Artifacts, you need to specificy exactly this, but add credentials to the beginning of the link:

pip install <package name> --extra-index-url http://<user>:<token>@xyz123.com

Remember, you can find the link to download your package by clicking the “Connect to Feed” option in Azure Artifacts. (For those unfamiliar, a Feed is a container for packages in Azure DevOps; it provides version and access control for project dependencies).

However, it is bad practice (think security flaw) to hardcode credentials directly into the software; it should be passed another way, and it made sense to use Azure’s library feature to store the credentials. Library is a service that is used to store key-value pairs, and only those with proper permissions can access the items. These variables can be passed into any pipeline within the Azure ecosystem. This also makes it possible to work with Azure DevOps even if you want to run the code in another cloud environment like AWS; you would just need to have Azure push the pre-built image to ECR, Amazon’s container repostory service, for instance.

So, we are left with the option to load the credentials into the image at build-time. And don’t worry about the token showing up in the pipeline log messages- the password is obfuscated if set as “secret”:

Azure Library

But take note: if you are going to store the image in Azure’s container registry and are only looking for a secure way of downloading packages, this workflow might isn’t totally relevant and might be a little circuitous (but, still good to know and might be tangentially useful). You could simply use a Docker Registry Service Connection to to more directly download packages and run them inside Docker. However, there could be other reasons to pass key-values into Docker. So, Let’s keep on keeping on. In this tutorial, we will simply print out the username stored in the library.

After creating the library group and variable, create your dockerfile. Then, upload it to your Azure repository.

For our purposes, I am skipping any package creation as discussed above. No need to get complex.

FROM python:3.11-slim-bullseye

ARG username # here, we are telling Docker to expect an argument passed during the build process

ENV APP_ENV=${username} # here, we are passing the build argument to an environmental variable.
CMD echo ${APP_ENV} # We are going to print out the variable value to the command line.

In Azure pipelines, we can create a task to build the docker image. There is a specific task to do this, called “Build an Image” in the traditional UI for building azure pipelines. In YAML, the task would be called Docker@0. You could also use a Bash command instead to accomplish the same thing:

docker build --build-arg username=$(username) -f $(Build.SourcesDirectory)/Dockerfile

You might need to change or add additional arguments, including to specify the location of the dockerfile. I put down the default location for this tutorial.

However, I won’t be using the bash command directly in the pipeline, but it is good to know what is happening behind the scenes.

But whatever method you choose to use, you need to connect the pipeline the Azure Library group. In the traditional UI, this is where you can make the connection:

Connecting to Library Variable Group in traditional pipeline UI

In YAML, you would use a snippet like this:

variables:
    - group: Variables-Dev  # this is where we connect the variable group to the YAML pipeline

In the traditional UI, we can reference the build arguments. We specify the variable by using this format: $(variable_name).

Building and pushing a Docker image in the traditional UI

In YAML, the whole thing would look like this (I’ve removed sensitive information):

pool:
  vmImage: ubuntu-latest

variables:
    - group: Variables-Dev  # this is where we connect the variable group to the YAML pipeline

steps:
- task: Docker@0
  displayName: 'Build an image'
  inputs:
    azureSubscription: 'Azure subscription 1 (########################)'
    azureContainerRegistry: '{"loginServer":"##############.azurecr.io", "id" : "/subscriptions/################/resourceGroups/##########/providers/Microsoft.ContainerRegistry/registries/##########"}'
    buildArguments: 'username=$(username)'
    additionalImageTags: 'my_image'
- task: Docker@0
  displayName: 'Push an image'
  inputs:
    azureSubscription: 'Azure subscription 1 (###############)'
    azureContainerRegistry: '{"loginServer":"################.azurecr.io", "id" : "/subscriptions/##################/resourceGroups/#############/providers/Microsoft.ContainerRegistry/registries/#############"}'
    action: 'Push an image'
    additionalImageTags: 'my_image'

- task: Docker@2
  displayName: 'login to ACR'
  inputs:
    containerRegistry: testServiceConnection
    command: login

- script: 'docker run ################.azurecr.io/testproject:my_image'
  displayName: 'Command Line Script'

And you can see the username populating after running the dockerfile:

Final output. See how we print the username

I am creating four non-distinct, overlapping areas that I think anyone interested in GIS, spatial data science, and engineering should study- because that is what I did. They are:

Computer Science

Computer science is a broad field, and can be used in a large number of disciplines. It is the study of computers and computational systems. It has many components, but my own experience tells me that its concepts in data structures and algorithms are most fundamental and important. Knowledge of computer science can be extremely beneficial for working with data efficiently.

Software Engineering

Software engineering is the systematic design, development, testing, and maintenance of software. It is not to be forgotten that software engineering also encompasses project management and quality assurance. It goes well beyond ad hoc scripting- but if you can learn to code, you can learn to be a software engineer. It just takes dedication, like anything else. Knowledge of software engineering can help bring geospatial data and information solutions to your company, the government, and the masses.

Data Engineering

Data engineering is just a niche within software engineering, focused on data collection and processing. Data engineers build data pipelines- that is, software that helps move data from one place to another. Data pipelines should be built in a reproducible, versioned, automated, scalable, and robust way- minimizing the potential for data loss or data duplication; systems do fail and bugs are inevitably introduced in the software development process, so engineers attempt to mitigate this as much as possible. Usually, data engineers work to consolidate data from a wide variety of sources into a single, standardized data warehouse. The data that is collected is made available to data analysts and scientists for use in creating models, visualizations, dashbaords, and reports. Data engineering is my discipline at the time of writing this post.

Spatial Data Science, GIS, and GIScience

Data science is a multidisciplinary field that combines statistics, artificial intelligence and machine learning, computer science, and industry or scientific knowledge to gain insights from data and predict future outcomes. There can be some overlaps with data engineering when it comes to data collection and munging, but data science specializes in the creation of models to make predictions based on historical data. Often times, the models data scientists create are used as integral parts of software offerings. Other times, the findings are purely academic or ad hoc in nature.

Spatial data science is all of this, but focuses on data with a geographic component, which is an added layer of complexity. Spatial data science is very similar to geographic information science (GIS or GIScience), which is the basis for (and is also semantically similar to) geographic information systems (also GIS). While geographic information systems are the tools primarily concerned with the management, visualization and analysis of geographic data, geographic information science is the theory underpinning geographic information systems. Spatial data science, meanwhile, is more practical in nature than geographic information science by attempting to answer real world questions and often places an emphasis on predictive analyses.

TLDR: you can perform spatial data science using geographic information systems, but you cannot perform spatial data science without knowledge of geographic information science.

If you’ve never used the Power BI REST API previously with PowerShell previously, you first need to download the module onto your system. Enter the following into the PowerShell console:

# A REST API is simply an interface for interacting with a server. Full details on
# the API functionality can be found at: https://docs.microsoft.com/en-us/rest/api/power-bi/

Install-Module -Name MicrosoftPowerBIMgmt # please type this line into your PowerShell console

Next, I will demonstrate how to quickly interact with the API and your published content. If you haven’t yet, open up PowerShell ISE, an IDE for PowerShell Scripting (though, I am sure VS Code would work just fine, too). I encourage you to run each line of code independently (not all at once) and to examine the structure and contents of each variable. And of course, don’t forget to save your script when finished.

Here, you will be able to access metadata on your reports:

Login-Power BI # This initial line will open up a login screen for the Power BI service

<# 
Ensure you find the group ID for your content. This is located in the URL of your workspace
E.g., app.powerbi.com/groups/<YOUR-GROUP-ID>/list
Also note how in PowerShell, variables are initialized and called using the dollar-sign 
#>
$result = Invoke-PowerBIRestMethod - URL"https://api.powerbi.com/v1.0/myorg/groups/<YOUR-GROUP-ID>/datasets" -Method Get

<# 
Unlike in R where piping is performed with %>% when using the Tidyverse
and |> in base R, PowerShell uses an actual pipe character to transfer the
contents of $result to the function 'ConvertFrom-Json'
#>
$workspaceContents = $result | ConvertFrom-Json 

# For testing purposes, let's try to extract the contents of the first report listed.
# type this into the console and then call $firstWorkspace
$firstWorkspace = $workspaceContents.value[0]

# Of course, it's bad practice to specify an object by referencing a direct location.
# What if that changes? Let's set up the basis of a for-loop by getting the number
# of reports stored in our workspace:
$m = $workspaceContents.value | measure

<# 
Now we can create our for-loop. Note the similarities of a PowerShell 
for-loop structure with Java syntax. '-lt' is a less-than operator here,
specifying to iterate i (beginning at 0) with increments of 1 until it 
reaches less than the value of $m.Count
#>
for ($i=0; $i -lt $m.Count; $i++){
    if ($workspaceContents.value[$i].name -eq "My Geospatial Report"){ # '-eq' is the equals operator
        $desired_report = $workspaceContents.value[$i]
        Write-Host "The report was found!" # this is a print-to-console operation
        break # this stops the loop at the current iteration
    }
}

# call the report metadata:
$desired_report

I may post more tutorials and code snippets on Power BI API implementations in the future. Until then, I would strongly recommend watching Guy in a Cube tutorials on YouTube. The channel has some great content for all things Power BI, including implementation of the REST API. Thanks for stopping by, and happy coding!

In this post, I will demonstrate how to: 1) Set up an Amazon Web Services (AWS) S3 Bucket 3) Connect S3 to an R script 4) Use this connection in an R Shiny dashboard.

Setting up an Amazon S3 Bucket

Create an AWS Account. You will be asked for an email, password, and credit card information. As long as you stay within Free Tier restrictions, you will not be charged. Only my use of the AWS Key Management Service (KMS) has incurred charges from the workflow described in this tutorial, but currently I’ve only been charged $0.12.

Create an S3 Bucket. The name for your bucket must be unique- no other AWS user must have created a bucket with the same name.

Choose an applicable region. The “cloud” just refers to data centers that Amazon maintains. These data centers are located in various regions. I chose the region closest to me as of the writing of this tutorial. Ensure that you remember the region you picked; transferring data between regions can incur costs, so if you have other AWS services that you anticipate needing in the future, having them located in one region will reduce spending.

Block all public access to this bucket. Amazon recommends this security feature be activated to prevent unrestricted access to the contents of your bucket.

Encrypt your data. I can’t stress the importance of data security enough. I recommend you enable server-side encryption. This will encrypt all the data at the object level that enters the bucket. You have the option of using either SSE-S3 encryption of SSE-KMS (Key Management Service) encryption. It is up to you, however, to decide what is more suitable for your needs. I personally chose to use SSE-KMS, with symmetric encryption. I then assigned a specific IAM user to have access to the key. (For details on setting up an IAM user, see the next step; this can be done concurrently).

When you use SSE-KMS, you create an Access Key ID, Secret Access Key, and user for said key. Ensure you keep these credentials safe- you will need these for APIs and other services (e.g, R Shiny).

Set-up IAM users: IAM stands for Identity and Access Management. This allows you to enable users to access certain AWS services without needing access to the root account. This improves security and reduces risk of unauthorized access to data and cloud services.

​Complete the set-up of your bucket and upload files. Feel free to upload some test files. I decided to upload an RDS file for ease of use with R. This RDS file contains a simple Leaflet map I created previously that I wanted to be able to showcase easily on Shinyapps.io. I ensured all my read/write permissions for the file were set to ‘private’.

​Optional: Set up budget alerts. In the management console under billing, you have the option to set up email and SNS alerts if you begin to go over budget. I have mine set to $10/month, and will receive notifications if I approach 80% of that amount.

Connect R to your S3 Bucket

Now comes the easy part! Seriously- connecting R to S3 is a piece of cake, if you are already an R programmer. First, store your S3 credentials in a file named “Renviron” in your working directory. While it is not good practice to hard code credentials, this should get your code up and running for at least a proof of concept and demo. Just don’t forget to pass these credentials in using a safer method before putting it into production!

The format for saving your credentials in the Renviron file should be as follows:

AWS_ACCESS_KEY_ID = "XXXXXXXXXXX"
AWS_SECRET_ACCESS_KEY = "XXXXXXXXX"
AWS_DEFAULT_REGION = "XXXXXXX"

R will know where to find these credentials. I’ve also hidden the file names and paths within text files located in my working directory for the same reason. Run my code below (albeit with the correct folder paths):

library(aws.s3)

s3BucketName <- scan("bucketname.txt", what = "txt")
s3File <- scan("filepath.txt", what = "txt")

# These files are located on the bucket
file_names <- get_bucket_df(s3BucketName)

# This loads my desired file - in my case, a Leaflet map stored in an RDS file.
myMap <- s3readRDS(object = file_names[file_names$Key == s3File, "Key"], bucket = s3BucketName)

myMap

Implementing this into a ​Shiny dashboard is relatively straightforward once you’ve completed the previous task. We simply need to wrap all this into a new script called app.R.

library(shiny)
library(shinydashboard)
library(leaflet)
library(aws.s3)

s3BucketName <- scan("bucketname.txt", what = "txt")
s3File <- scan("filepath.txt", what = "txt")

# files in bucket
file_names <- get_bucket_df(s3BucketName)
myMap <- s3readRDS(object = file_names[file_names$Key == s3File, "Key"], bucket = s3BucketName)

ui <- dashboardPage(
    dashboardHeader(),
    dashboardSidebar(),
    dashboardBody(
        leafletOutput("mymap", height = "92vh") #this text is an ID that must match `output$var_name` below***
    )
)

server <- function(input, output, session) {
    
    output$mymap <- renderLeaflet({ #****
        myMap
    })
}

shinyApp(ui, server)

This is a pretty bare-bones dashboard- merely a structure for you to build off of. But I hope you’ve found this tutorial helpful for merging these technologies together. And once you build out the the dashboard, deploying it online is pretty simple using Shinyapps.io.

And again, don’t forget that after your demo and POC, you will want to pull in your AWS credentials from a secrets store, whether that is with GitHub Secrets, AWS Parameter Store, or AWS Secrets Manager.

If you decide to host your application with AWS EC2 to host the application, you might decide to use instance profiles as a method of authenticating your application against your cloud provider.

Thanks for reading, and as always, happy coding!

Study Area

Our study area includes part of the New-York Metropolitan Area, including New York City, Long Island, counties in Upstate New York, and large portions of New Jersey. These areas were determined by the New York Core Based Statistical Area (NY CBSA).

Data

PM2.5 data has been provided by the Environmental Protection Agency (EPA), and can be found through the open data portal: https://www.epa.gov/outdoor-air-quality-data/download-daily-data.

The CBSA boundary shapefile can be obtained from the United States Census Bureau.

Software Packages and Libraries

To begin the analysis, first load the packages we will be using for this project. Please type in install.packages(“PACKAGE_NAME_HERE”) for each package you do not already have locally installed (all those used in this project are listed below). Then, run the code chunk below:

library(sf) # offers an exceptional data structure for storing GIS vector data 
library(tidyverse) # compilation of packages curated to make data cleaning easy 
library(raster) # data structure for storing raster data and preprocessing functions 
library(gstat) # package for interpolating data
library(lubridate) # provides functionality for working with dates and times 
library(plyr) # provides extra data cleaning functions
library(tmap) # interactive visualization functionality for geospatial data 
library(rasterVis) # provides enhanced plotting functions for raster data 
library(RColorBrewer) # enhanced and customized palettes for visualizing data 
library(reshape) # provides extra functionality for data cleaning 
library(Kendall) # provides acccess to Mann-Kendall modeling 
library(EcoGenetics) # provides access to Theil-Sen modeling
library(imager) # easy plotting of non-spatial image files, like .BMP or .PNG

Data Pre-Processing

The raw data used in this analysis spans a 10-year period, from January 1st, 2010 to April 12th, 2020. Air quality data was sampled at each location at least once every few days, making it a near-daily data set. To focus on the yearly winter season, the data was filtered based on the December - March months. The mean PM2.5 value was obtained for each time index (season). Users of this tutorial are encouraged to test view the output of each intermediate step to better understand how the workflow operates.

# Reads all csv files in this directory
data = list.files(path="data", pattern="*.csv", full.names=TRUE)
# Merges all csv files into a single dataframe for analysis
data_all = ldply(data, read_csv)
# replace spaces in column names with '_'
names(data_all) <- gsub(" ", "_", names(data_all))
# rename latitude & longitude columns
names(data_all)[names(data_all) == "SITE_LONGITUDE"] <- "x"
names(data_all)[names(data_all) == "SITE_LATITUDE"] <- "y"
# Convert to date format for filtering
data_all$Date <- data_all$Date %>% as.Date("%m/%d/%Y")
# extract year from Date as a new column for yearly interpolation
data_all[, "year"] <- format(data_all[,"Date"], "%Y")
# filter for winter months
df_winter <- data_all %>% 
  filter(strftime(data_all$Date, "%m") %in% c('12','01','02','03'))

Spatial Conversion

The shapefile for the CBSA is first loaded into the R working environment. Then, sampling locations were extracted from the .csv file of PM2.5, and converted into a spatial object.

# read in cbsa shapefile
nycbsa <- st_read("data/ny_cbsa.shp")

# convert sites to an sf object using projection from cbsa
epa_sites <- df_winter %>% 
  distinct(Site_ID, x, y) %>% 
  st_as_sf(coords = c("x", "y"), crs = st_crs(nycbsa))

Explore Metropolitan Area & Air Pollution Monitoring Sites

This chunk of code visualizes the sampling locations and the overarching area of interest. Monitoring site that falls outside the CBSA boundary will be removed from the analysis.

tmap_mode("view")

tm_shape(nycbsa, name = "NY Metropolitan Area", is.master = TRUE) + 
  tm_borders(col= "coral") +
  tm_layout(title = "New York Metro Area Air Pollution Monitoring Sites") +
  tm_shape(epa_sites, name = "EPA Air Monitoring Sites") +
  tm_dots(col="red")

Let’s take a look at our inputs:

Rasterize

The NYCBSA data object is currently in the form of a vector file, but needs to be converted into a raster image to create an interpolation surface. This code chunk uses the rasterize function from the raster library for this conversion, and then plots our area of interest.

# create a raster file with resolution of 0.01 for ny metro area
target <- raster(x = extent(nycbsa), crs = crs(nycbsa), res = 0.01)
# assign value of 1 to each pixel
values(target) <- 1
# crop raster surface to NY metro region
nycbsar <- nycbsa %>% rasterize(x = ., y = target, field = "GEOID")
# plot for visual check
par(mar = c(1, 0, 0, 4))
plot(nycbsar, axes = FALSE, box = FALSE, legend = FALSE)

Interpolation

There needs to be a few more preprocessing steps to ready the dataset for analysis, and then use the gstat function to perform an inverse distance weighting (IDW) interpolation.

# assign year to winter months
# if jan-march, assign to previous year, else assign same year
df_winter$winter <- ifelse(format(df_winter$Date, "%m") == "12",
                           df_winter$year,as.numeric(df_winter$year)-1)
# create a list of all winter years
years <- unique(df_winter$winter)

# Interpolate for each winter season
# Calculate mean for every site for each winter season
# int_df.list: List of dataframes with mean PM2.5 for each winter
# invdist.list List of interpolated surfaces for each winter
int_df.list =list()
invdist.list = list()
for (each_year in years){
  # calculate mean for every site for each winter season
  int_df <- df_winter %>% 
  filter(winter == each_year) %>% 
  group_by(Site_ID, winter, x, y) %>%
  summarise_at(vars(Daily_Mean_PM2.5_Concentration),
               list(PM_Mean = mean))
  # add dataframe to list
  int_df.list[[each_year]] = int_df
  
  # inverse distance interpolation for each year
  # create a gstat object
  invdist <- gstat(id = "PM_Mean",
                   formula = PM_Mean ~ 1,
                   locations = ~x + y,
                   data = int_df)
  # generate interpolated layer
  invdistr <- interpolate(object = nycbsar, model = invdist)
  # mask to metro area
  invdistrmsk <- mask(x = invdistr, mask = nycbsa)
  # add interpolated layer to list
  invdist.list[[each_year]] = invdistrmsk
}

Plotting the Interpolated Layer

# create a stack for all interpolated layers for plotting
s <- raster::stack(invdist.list)
# edit name for each layer
names(s) <-  gsub(x = names(s), pattern = "X", replacement = "Winter")
# set up theme for plot
mapTheme <- rasterTheme(region=brewer.pal(8,"Blues"), rev=TRUE)
# plot interpolated layers
levelplot(s, pretty = TRUE,  main="NY Metropolitan Area PM2.5 Mean", par.settings=mapTheme)

Saving Interpolated Raster Images as .RST

for (i in 1:length(s@layers)){
  writeRaster(s[[i]], filename = paste0("terrset/", names(s[[i]]), ".rst"),
              overwrite = F)
}

Histogram

Histograms are a great way to view how PM2.5 measurements change year-to-year within the study area. Each bar represents a sampling location and the amount of PM2.5 measured there. You can see some sampling locations with increasing PM2.5 values over the 10-year time span, while others have decreasing PM2.5 values.

# raster stack to dataframe for each x, y
# remove NA values
stack_df <- as.data.frame(s, xy = TRUE, na.rm = TRUE)%>%
  melt(id.vars = c('x','y'))

# histogram for interpolated layers
par(mfrow=c(4,2))
ggplot(stack_df) +
  geom_histogram(aes(value), binwidth = .5) +
    facet_wrap(~variable)

Mann-Kendall Trend Test: Tau & P-Value

A Mann-Kendall model is a non-parametric test similar to a pearson correlation analysis. Ranging from +1 to -1, a positive tau value indicates an increasing trend while a negative tau value indicates a decreasing trend. The higher the absolute tau value, the more consistent that trend is. This helps us answer the question, “where are PM2.5 measurements falling, and where are they rising?”

# function for Mann-Kendall trend test
kendall = function(x){return(unlist(MannKendall(x)))}
# apply function to raster stack. RUN THIS CHUNK IN CONSOLE ONLY
# takes too long in markdown and ignore "WARNING: Error exit, tauk2. IFAULT =  
# 10" warning. Output has been validated in TerrSet, and this message appears 
# to be a bug in the package. For simplicity, an R object (.rds file) has been 
# saved to the working directory, which contains the output for the Mann-Kendall
kendall_output = calc(s,kendall)

# Save output to a .rds file
#saveRDS(kendall_output, file = "kendall_output.rds")

Because of some bugs running the above code chunk in an R-Markdown file, we recommend running the above chunk in the R console specifically. If you continue to have trouble running the above code (e.g, the code infinitely runs without stopping after a few minutes), we have provided the output for you as a .rds file which can be easily read into RStudio.

These file types are useful in R; they can be read in directly into your global environment using the same data structure it was created in, without running any further conversions or data processing steps, saving time and preventing potential bugs in your code. To save this file, we have have used the saveRDS() function seen above. If you are familiar with Python programming, saveRDS() is analogous to pickle.dump() from the pickle Python package.

In any respect, we recommend trying to run the code from the above chunk first. If you have trouble using that code, you can instead load in the provided file using this code here:

kendall_output <- readRDS("output/kendall_output.rds")

Plotting Mann-Kendall Output and saving to .RST

The levelplot() function for raster images can be considered to be an upgrade to the generic plot() function. It provides enhanced functionality and better visualizations.

Notice that the kendall_output data structure is that of a RasterBrick. RasterBricks are similar to RasterStacks. In both, you can store multiple raster images within the object, allowing for easier file organization and analysis. To access raster images within a brick, type the name of the object, followed by a dollar sign and the name of the raster you are searching for.

# plot Kendall's tau statistic
levelplot(kendall_output$tau, main = "Kendall's tau statistic")

# plot p-value
levelplot(kendall_output$sl, main = "p-value")

# write to rst
writeRaster(kendall_output$tau, filename = "output/tau.rst", overwrite=TRUE)
writeRaster(kendall_output$sl, filename ="output/mk_p-value.rst", overwrite=TRUE)

You may notice from these plots that at approximately 40.5N, 74.7W PM2.5 tends to increase consistently over the 10-year time frame, while other area see more consistently downward trends. Remember, however, that these estimates are very dependent on the interpolation method used. IDW was chosen for its simplicity for this tutorial, though other methods may yield more accurate results.

TerrSet Validation: Kendall Tau & P-Value

To ensure R is returning the values we expect (and that the package was not poorly written, or that there was some methodological flaw), it is good to compare results with those in software you may be more familiar with, such as TerrSet. After performing a Mann-Kendall test in the TerrSet Earth Trends Modeler, we have found the results between these two programs to be virtually identical.

ktau_img <- load.image("terrset/TERRSET_MK_TAU.BMP")
plot(ktau_img, axes = F)

mkp_img <- load.image("terrset/TERRSET_MK_P.BMP")
plot(mkp_img, axes = F)

Theil-Sen Median Trend Test

Knowing simply whether trends are increasing or decreasing is not necessarily enough. It is also useful to know the matgnitude, or rate at which trends are increasing. The Theil-Sen median trend analysis returns a slope for each pixel, which can tell us rates of change in our time series.

# saves two images automatically to working directory: 
# pvalue, and slope of theil-sen.
# Note that this function takes at least 10 minutes of processing time 
# to complete, so be prepared to wait.
eco.theilsen(s, dates = c(seq(from = 2009, by = 1, to = 2019)))
file.rename(from = "pvalue.tif", to = "theilsen_pvalue.tif")
file.rename(from = "slope.tif", to = "theilsen_slope.tif")

A Theil-Sen median trend image shows us the slope, or rate of change in a time series analysis. You can see that there is a positive rate of change located at approximately 40.5N, 74.7W, with negative rates of change elsewhere.

theilsen_slope <- raster("output/theilsen_slope.tif")
plot(theilsen_slope, main = "output/Theil-Sen Median Slope")

# write slope image to IDRISI raster format for easy comparison
writeRaster(theilsen_slope, filename = "output/theilsen_slope.rst", overwrite=T)

TerrSet Validation: Theil-Sen Median Trend

Similarly, we can validate our results in TerrSet, and we found the results from these two programs to be identical.

theilsen_slope_img <- load.image("terrset/TERRSET_THEILSEN_SLOPE.BMP")
plot(theilsen_slope_img)

One of the ways in which I initially learned to use PostGIS functions (my new RDMS of choice) was to redo some homework assignments from a previous introductory course in GIS- but using SQL functions instead of ArcMap geoprocessing tools. This particular homework assignment sought to illustrate a use case for buffering, dissolving, and intersecting. By transliterating the workflow from ArcMap into SQL code, I was able to create a process that was reproducible, easy to understand, and more efficient to run. Plus, there was no file bloat of intermediary files- a common data management problem when geoprocessing data in ArcGIS.

The assignment simulates a problem easy to envision occurring in the natural resources sector: “Forest management staff have asked you to develop information to assist with the application of pesticides… Management intends to apply pesticides to areas within 200 meters of each infested point.” The goals of the project include:

• Finding total area of each forest cover type within 200 meters of an infested point
• Finding out whether streams pass through the application areas
• Finding the length of each stream segment that passes through an application area, and what type of forest cover it passes through

*You can download the data for this tutorial on my github, here.

We are given 4 different set of geometries, stored as shapefiles:

• Park boundary (polygon)
• Infestation points (points)
• Streams (lines)
• Forest cover type (polygon)

First, I uploaded each shapefile into its own table in the database with a specified schema. I used ogr2ogr through the command prompt to accomplish this.

ogr2ogr -f "PostgreSQL" PG:"host=localhost user=postgres dbname=postgres password=******* port=5432" D:\FILE_PATH\boundary.shp -overwrite -lco precision=NO -lco GEOMETRY_NAME=geom -nln "staging.kemo_bound"
ogr2ogr -f "PostgreSQL" PG:"host=localhost user=postgres dbname=postgres password=******* port=5432" D:\FILE_PATH\infest_point.shp -overwrite -lco precision=NO -lco GEOMETRY_NAME=geom -nln "staging.kemo_infest_points"
ogr2ogr -f "PostgreSQL" PG:"host=localhost user=postgres dbname=postgres password=******* port=5432" D:\FILE_PATH\streams.shp -overwrite -lco precision=NO -lco GEOMETRY_NAME=geom -nln "staging.kemo_streams"
ogr2ogr -f "PostgreSQL" PG:"host=localhost user=postgres dbname=postgres password=******* port=5432" D:\FILE_PATH\vegetation.shp -overwrite -lco precision=NO -lco GEOMETRY_NAME=geom -nln "staging.kemo_vegetation"

*ogr2ogr comes with GDAL. Install that if you have not already.

Let’s take a look at our inputs:

Boundary

Points

Streams

Vegetation

The first task was to create a buffer around the infestation points, and to dissolve the boundaries. You can do this in a couple ways. The first is the create a nested query, like so:

--create buffers and dissolve boundaries
select (st_dump(st_union(blah.st_buffer))).geom from
(select st_buffer(staging.kemo_infest_points.geom, 200)
from staging.kemo_infest_points) as blah

You can think of this method as querying the results of another query. Using nested queries when I first learned database management helped me understand more intuitively how databases functioned. Additionally, it was crucial for helping me to understand the WITH statement (also called a common table expression, or CTE), which is very important for making large queries easier to read and understand. Nested queries by themselves can often be messy, lack conciseness, and can become overly verbose. Though important to learn, it is often best to avoid using them if possible. A better method would be the following:

SELECT (st_dump(st_union(st_buffer(a.geom, 200)))).geom 
from staging.kemo_infest_points as a;

Both result in the same output. See below:

Dissolved Buffer

Let’s break down this process. First, we use the ST_Buffer function, on a.geom (with ‘a’ being the table, infest_points, as it has been given an alias), and give it a buffer of 200 meters. Geom is the column that holds the geometry of infest_points. Remember that we set this name originally in our ogr2ogr command. Using ST_Buffer alone (without the other functions) would produce the following result:

SELECT st_buffer(a.geom, 200)
from staging.kemo_infest_points as a;

Buffer Only

As you can see, we still need a dissolve. Which leads us to adding the function, st_union().

SELECT st_union(st_buffer(a.geom, 200))
from staging.kemo_infest_points as a;

Single Record from Union and Buffer Only

The problem with this, though, is that no longer maintain the individuality of each buffered point. Each buffer has now been merged into a single record, which is not what we want. To avoid this, we use ST_Dump(). According to the PostGIS documentation, ST_Dump can be thought of as the opposite of a GROUP BY clause. Remember to add .geom at the end of your select statement, otherwise you will be returned an improperly formatted column formatted geometry column without the desired visuals.

Knowing this general methodology, let’s skip ahead to my near-final product. You will now find that at this stage in my coding, I reverted back to nested loops! Not the most pleasant sight. It’s hard to believe, but this hard-to-read chunk identifies the forest cover within each buffered location, and those sections that are within the park boundary. But how does it work? What are those inner joins doing? Not easy to tell with the way this code is formatted…

select veg_intersect_t.ogc_fid, veg_intersect_t.class2, st_intersection(st_intersection, geom) from
(select ogc_fid, class2, st_intersection(dissolve_geom, geom) FROM --select FID, vegetation class, and only geometry that is within buffer zone
(SELECT dissolve.dissolve_geom, b.* FROM
(SELECT (st_dump(st_union(st_buffer(a.geom, 200)))).geom as dissolve_geom --buffer and dissolve points
FROM staging.kemo_infest_points as a) AS dissolve
INNER JOIN staging.kemo_vegetation as b ON st_intersects(dissolve.dissolve_geom, b.geom)) as joined) as veg_intersect_t
INNER JOIN staging.kemo_bound as c on st_intersects(veg_intersect_t.st_intersection, c.geom)

Vegetation within Buffered Zones

What I am about to show will start to make the above more readable. I will introduce the WITH statement, or CTE query, which as I mentioned previously provides an easier way to write large SQL queries. See the documentation for that here. This code chunk below is equivalent to the above, and a bit easier on the eyes:

WITH buffer as (select (st_dump(st_union(st_buffer(a.geom, 200)))).geom
from staging.kemo_infest_points as a),

vegetation_intersect as (select ogc_fid, class2, st_intersection(b.geom, buffer.geom) 
from buffer
inner join staging.kemo_vegetation as b on st_intersects(b.geom, buffer.geom))

select vegetation_intersect.ogc_fid, vegetation_intersect.class2, st_intersection(vegetation_intersect.st_intersection, blah.geom) as good_geom
from vegetation_intersect
inner join staging.kemo_bound as blah on st_intersects(vegetation_intersect.st_intersection, blah.geom)

The WITH statement is akin to a nested query, but allows users to separate their code into chunks, piping one query into the next quite easily, even on an ad hoc basis. See how we can easily add an extra line at the end of the above code to return the area of each forest type that will be impacted by the pesticide application:

WITH buffer as (select (st_dump(st_union(st_buffer(a.geom, 200)))).geom
from staging.kemo_infest_points as a),

vegetation_intersect as (select ogc_fid, class2, st_intersection(b.geom, buffer.geom) 
from buffer
inner join staging.kemo_vegetation as b on st_intersects(b.geom, buffer.geom)),

hygge as (select vegetation_intersect.ogc_fid, vegetation_intersect.class2, st_intersection(vegetation_intersect.st_intersection, blah.geom) as good_geom
from vegetation_intersect
inner join staging.kemo_bound as blah on st_intersects(vegetation_intersect.st_intersection, blah.geom)),

woohoo as (select *, st_area(hygge.good_geom) from hygge)

select class2, sum(st_area) * random()
from woohoo
group by class2

*Note that I added the random() function to the code just as an assurance that no future student will copy my results.

The WITH statement turns a convoluted chunk of code into something easily reproducible, and makes streamlining the workflow a relatively quick process. It is probably one of the most important concepts I have learned this past semester.

I may write a future blog to investigate in more detail how intersections using ST_Intersects and INNER JOIN work in PostGIS. That tutorial will also seek to answer those two remaining questions: Do any streams pass through the application areas? What is the length of each stream segment that passes through an application area?

Our professor noted to us, however, that working on a locally hosted database has become outmoded in the face of cloud computing. He recommended that we learn to set up PostGIS on AWS and learning how to secure it against unauthorized access. I had never used AWS prior to this semester, and didn’t know too much about what was involved. As it turns out, use of these cloud services often necessitates knowledge of the terminal/command prompt and its associated functions, which I had little prior experience with. By the end of the semester, though, I became much more proficient in its use and how to improve data security.

The Project

We set out to host geosptial data on a PostgreSQL database. To visualize the data, we used Leaflet.js in combination with GeoServer. We found that GeoServer improved interoperability between our front and backend solutions, and even added an additional layer of security to our project.

The Data

For the purposes of this tutorial, I am changing the data from the original project. You may use the sample data, here

Virtual Machine Creation

1. Create an AWS Account if you do not already have one

   • If you are a student, you can open an AWS Educate account, which gives you $40 of free credits per month as of December 2019

   • You can also open a free-tier account. You may still need to enter in credit card information. However, the charges for this tutorial should be minimal (in the single dollars)

   • Please, please, please do not use the root user to create or manage resources for your account. This is a major security flaw and just bad practice. Instead, use a separate IAM user to do this- and as always, enable MFA. I will demonstrate how in another tutorial. Until then, please search for information on how to do this if you are unsure of the methodology.

2. Search for EC2 in the management console search bar, and go to the EC2 page.

3. Generate a Key Pair
   • This option will be under the “Network and Security Subsection”

   
   

   

   
   

   

   1. Choose a name for your key pair that is easy to remember
   2. If you are using Linux, MacOS, or a modern Windows OS (v10 or higher), use .pem file format

   3. Download the key pair and store it in a safe place. Do not share it!

      

4. Click Launch Instance button, and then choose an AMI (Amazon Machine Image)

   1. We chose an Ubuntu instance, which is Free Tier Eligible. But, Amazon Linux is also free-tier eligible as well.

   2. The t2.micro instance is the only one available for Free Tier, which will be fine for getting start
      • Each instance type is optimized for different purposes. Choose one at your discretion
   3. Select the key pair you just created.
   4. Select your network settings
      • For our purposes, we left this as the default option. You may wish to change these going forward, however.
      • Using the default VPC can allow you to have your resource available to the public internet. If you do not want or require this, choose a custom option.
        • Do not worry about your EC2 instance being public even if choosing the default option; you still need the key-pair .pem file you generated earlier. So, again, do not lose it and keep it safe.

      

   5. Configure your storage
      • We left this as the default

      

5. Click Review and Launch
6. Instance should now be running. Check for it in the dashboard.
   • Remember to stop the instance when you are not using it; AWS charges per minute of EC2 usage.
   • A stopped instance will still incur charges for use of storage. To completely stop charges, the instance must be deleted.

VM Connection

1. Under your available EC2 instances, click on the one you just created and then click the “connect” button
2. Click the “SSH Client” tab
   • This will give you instructions on how to use an SSH client to log into your instance remotely.
   • MacOS, Linux, and modern Windows OS should have an SSH client pre-installed. On MacOS, this is the terminal.
3. Open your SSH Client and cd into the directory containing your .pem file.
4. Change the permissions on the .pem file.
   • chmod 400 pem_filename.pem
5. Connect to the instance using its public dns: ssh -i "filename.pem" <public_DNS_address>

Setting up new users on an Ubuntu Instance

1. Log into the instance from your SSH Client
2. Check the currently logged-in user by entering who into the command line
   • you can get a list of all users on the instance by typing cut -d: -f1 /etc/passwd
3. Run the command sudo adduser <username> and follow the prompts
   • remember to save the credentials somewhere safe
4. Give the user sudo permissions: sudo usermod -aG sudo <username>
5. Switch the active user to the one you just created: su - <username>
   • Note: it is bad practice to use the root user

Installing PostgreSQL on EC2 Ubuntu Instance

1. sudo apt update
   • This command is used to update package information
2. sudo apt upgrade
   • This command upgrades the packages based on the information downloaded in the previous update command
   • If you are prompted to reboot the instance, do so. Then, log back into the instance with the user you have created
3. sudo apt install postgresql postgresql-contrib
   • This command installs postgres on the instance
4. sudo apt-get install postgis
   • This command installs the required files to run the PostGIS extension for Postgres
5. sudo systemctl start postgresql.service
   • This command starts running Postgres. If you stop your EC2 instance, you will need to run this line again.

Setting up the postgres database

1. Open an interactive shell session as the user postgres with elevated privileges. Let’s use: sudo -i -u postgres
   • this is slightly different from using the su command as thisopens a new shell session and environment
2. Create a new user: createuser --interactive (I named mine dbadmin)
   • remember to make this a superuser
   • Note that you can only create new superuser roles if you are creating them while logged in from a role that is already a superuser. By default, the postgres role is a superuser, which is what you should be logged in as.
3. start using the postgres command line interface by typing psql into your ssh shell
   • You can always exit psql by entering \q into the command line
4. Check that the new user has been created by entering \du into the psql console
5. Switch to the new role you just created:

    -- remember, you can always use SQL syntax in the psql command line
    SET ROLE <role_name>;
   
    -- Check that you are switched to the new role:
    SELECT current_role;
   

6. Create a new database with the desired name by typing in standard sql code:

    CREATE DATABASE my_postgis_db;
   

7. Make sure that you have created the database asnd it is owned by the new role you have created by typing \l into the psql console

    -- This SQL code will also give you the same information as `\l`
    SELECT * FROM pg_database; -- to exit the console menu if stuck, click the `q` key on your keyboard.
   

8. Enable PostGIS capabilities

    CREATE EXTENSION PostGIS;
   
    -- check to make sure it is now enabled:
    SELECT extname FROM pg_extension where extname = 'postgis';
   

Notes:

For any role used to log into Postgres, that role will attempt to connect to a database of the same name by default. You can create the appropriate database with the createdb command and adjust what database to connect to.

Enabling GUI on EC2 (not recommended for this tutorial)

Important consideration for GUI use: As of writing this, CPU credits are limited to about 144 credits for the free-tier eligible micro EC2 instance. These credits are regenerated every 24 hours (6 credits per hour). GUI use seems to monopolize CPU usage and was not necessary for our work. It will also incur performance costs. More info about CPU and credit usage here.

However, to remote in with a GUI, we took these steps:

1. Edited the security group settings in the AWS Management Console for the instance for RDP import access port
2. To remote in from a Windows local machine, open Remote Desktop Connection and enter “ec2-x-xx-xxx-xx.us-east-1.compute.amazonaws.com”. For Linux, use the VNC viewer.

Postgres Data Import

Copy local files onto the AWS EC2 Ubuntu Virtual Machine. Make sure you have the .pem file handy.

• scp -i path\to\pem\file -r C:\Users\username\directory\of\files user@ec2-xx-xxx-xx-xx.us-east-2.compute.amazonaws.com:~/path/to/save/location ​- To upload a single file instead of a directory, remove the -r (recursive) option.

You could also use the PostGIS Shapefile Import/Export Manager as an alternative. But that is a GUI application, is less reproducible and cannot be automated.

Next Steps:

Using ogr2ogr

Use the ogr2ogr command to upload vector spatial data into the database. This can be used for a multitude of file types, but the general structure of the code stays the same (see that the code chunks below are almost identical). Note that the option -lco precision=NO and argument are often necessary when importing shapefiles, which contain field size limits. If uploading data to to postgres that has been copied to the virtual machine, you must now connect to EC2 instance prior to following the next steps. Instructions on connecting to EC2 are found earlier in this markdown.

For uploading Shapefiles and GPX files, respectively:

ogr2ogr -f "PostgreSQL" PG:"host=localhost user=postgres dbname=postgres password=PASSWORD port=5432" FILE_PATH.shp -overwrite -lco precision=NO -lco GEOMETRY_NAME=geom -nln "NAME_OF_NEW_TABLE"

ogr2ogr -f "PostgreSQL" PG:"host=localhost user=postgres dbname=postgres password=PASSWORD port=5432" FILE_PATH.gpx -overwrite -lco GEOMETRY_NAME=geom -nln "NAME_OF_NEW_TABLE"

If you encounter an error, you might need to download GDAL from https://gdal.org/download.html and paste the folder into your postgres program folder.

Importing CSVs with psql

First, create the table you wish to upload data to:

CREATE TABLE schema.table_name (
column_name1 text,
lat double precision,
lon double precision
);

Then, use the COPY command to load the CSV to the table:

copy ch01.restaurants_staging
FROM 'CSV_file_path.csv' 
DELIMITER as ',';

Securing the Virtual Machine and PostgreSQL Database

https://www.digitalocean.com/community/tutorials/how-to-secure-postgresql-against-automated-attacks contains useful information on securing the database. It helped us set up a firewall on our EC2 instance while only allowing our specific IP addresses through them.

To do this we used the following commands:

• sudo ufw status - To check the status the firewall
• sudo ufw enable - To enable the firewall
• sudo ufw allow from <client-ip> to any port 5432 - restricts database access to specific IP addresses

Accessing the EC2 Postgres Database on PgAdmin

• This guide was very helpful in the second step of getting IP addresses specifically set up for use of the database with Postgres.
• To keep security strong, we elected to add only the IP addresses we will be using as the allowable ip addresses to access the database.

We had to augment the pg_hba.conf file to change the allowable ip addresses.

Note: We had an error where the database was not starting up after changing the pg_hba.conf file. The error arose because we did not include a /xx notation at the end of an ip address and the invalid IP was preventing the server from starting up.

Data Visualization

To visualize our data stored in the Postgres database, we chose to use a combination of Geoserver and Leaflet

Geoserver Set-up

1. Geoserver was set-up on our local machines, following this guide .
2. Connected to our Postgres database by creating a new “Store”, then adding our EC2 and database parameters.
3. Styling layers

Leaflet Visualization

After setting up GeoServer, we pulled data into Leaflet.js to visualize it as a web map. The code for our web map can be found here on hackmd.io, along with additional details regarding EC2 security/firewall implementation.