Software Tools for Data Analysis
STA 9750
Michael Weylandt
Week 10 – Tuesday 2026-11-10, Thursday 2026-11-05
Last Updated: 2026-11-05

STA 9750 Week 10

Today: Lecture #08: Data Import and File System Usage; APIs

These slides can be found online at:

https://michael-weylandt.com/STA9750/slides/slides10.html

In-class activities can be found at:

https://michael-weylandt.com/STA9750/labs/lab08.html

Upcoming TODO

Upcoming student responsibilities - Tuesday Section:

Date Time Student Responsibility
2026-11-16 11:59pm ET Mini-Project Peer Feedback #02 Due
2026-11-17 6:00pm ET Pre-Assignment #09 Due
2026-11-24 6:00pm ET Pre-Assignment #10 Due
2026-11-27 11:59pm ET Mini-Project #03 Due
2026-12-01 6:00pm ET Final Project Presentation Slides Due
2026-12-07 11:59pm ET Mini-Project Peer Feedback #03 Due
2026-12-08 6:00pm ET Pre-Assignment #11 Due

Upcoming TODO

Upcoming student responsibilities - Thursday Section:

Date Time Student Responsibility
2026-11-06 11:59pm ET Mini-Project #02 Due
2026-11-12 6:00pm ET Pre-Assignment #09 Due
2026-11-16 11:59pm ET Mini-Project Peer Feedback #02 Due
2026-11-19 6:00pm ET Pre-Assignment #10 Due
2026-11-26
Thanksgiving
2026-11-27 11:59pm ET Mini-Project #03 Due
2026-12-03 6:00pm ET Final Project Presentation Slides Due

Other Important Dates

Other Important Dates:

Date Instructor Action
2026-11-05 Mini-Project Released #03
2026-11-08 Estimated Return of Meta-Review Comments #01
2026-11-09 Mini-Project Peer Feedback Assigned #02
2026-11-19 Mini-Project Released #04
2026-11-20 Estimated Return of Peer Comments #02
2026-11-24 Estimated Return of Meta-Review Comments #02
2026-11-26 Thanksgiving #01
2026-11-30 Mini-Project Peer Feedback Assigned #03

STA 9750 Week 10

Today: Lecture #08: Data Import and File System Usage; APIs

  • Communicating Results (quarto) ✅
  • R Basics ✅
  • Data Manipulation in R ✅
  • Data Visualization in R ✅
  • Getting Data into R ⬅️
    • Files and APIs ⬅️
    • Web Scraping
    • Cleaning and Processing Text
  • Statistical Modeling in R

STA 9750 Week 10

Today: Lecture #08: Data Import and File System Usage; APIs

  • Course Administration
  • Warm-Up Exercises
  • Using the File System
  • How The Web Actually Works
  • JSON and APIs
  • Wrap-Up

Today

Today

  • Course Administration
  • Warm-Up Exercise
  • New Material
    • Files and the File System
    • How the Web Actually Works
    • HTTP and Web Access
    • JSON API Usage

Course Administration

Mini-Project #02

MP#02 - Identifying Environmentally Responsible US Public Transit Systems

Due 2026-11-06 at 11:59pm ET

  • GitHub post (used for peer feedback) AND Brightspace
  • Start early to avoid Git issues

Pay attention to the rubric

  • Writing and presentation are about 50% of your grade
  • You now receive a grade (no automatic 10/10 on data visualization)
  • Evaluated on rigor and thoughtfulness
  • Use what you learned from MP#01

Mini-Project #02

Occasional issues downloading EIA (Energy) and NTD (Transit) data:

  • EIA usually fixed by retrying a little bit later
  • Copy of NTD posted to teams: you can pull from there and save locally

Only need to download once; then script will use cached copy

Post on Teams with issues

Mini-Project #03

MP#03 - Where My People At?

Due 2026-11-27 at 11:59pm ET

Analysis of Population Density

  • Identifying Areas of High Population Density
  • Demographic Breakdowns of Major Cities

Mini-Project #03

Topics Covered:

  • Using API Wrapper Packages (tidycensus)
  • Downloading and Parsing Static Files
  • Using APIs Directly
  • Spatial Data Visualization
  • Interactive Data Visualization

Future Mini-Projects

  • MP#04: Tentatively visualizing 2026 election results
    • Deadline: 2026-12-11 at 11:59pm ET
    • Web Parsing (Wikipedia) to get election results

Will announce when posted

Course Support

  • Synchronous - MW Office Hours 2x / week:
    • Tuesdays 5pm: Zoom
    • Thursdays 5pm: Zoom
  • Asynchronous: Teams

See Week 02 Slides for advice on asking good questions - Good questions get faster answers

Ask early for help with MPs - fixing problems is slow and iterative

Course Project

Course Project should be your main focus for rest of course

  • You have received second round of feedback from me
  • Starting with the tools of today, you can import data into R for EDA
    • Next two weeks will dive deeper into data cleaning and preparation

But you still need to do other mini-projects and pre-assignments(!)

  • Remember one MP dropped

Review Exercise

Flat / Plain Text Files

‘Plain text’ files:

  • Simple human readable and human writeable file formats
  • Not specific to one piece of software
  • Examples: .csv, .txt, .tsv
  • Anti-Examples: .docx, .pdf, .jpg
  • Line isn’t 100% clear

Import into R with read* functions (e.g., readr::read_csv )

SSA Baby Names

From the US Social Security Administration Baby Names records:

  • State, Sex, Name, Number Born

Read into R (readr::read_csv) and make plots to answer various questions:

  1. How popular is the name “Michael” in NY?
  2. Are Marys becoming less common in the past decade?
  3. Did Juan and Jose become popular names at the same time?

See the Review section of this week’s lab for details.

Breakout Rooms

Breakout Rooms
Please join the Breakout Room for your Project Team
Breakout #01: Beat The Odds (T) + King Philip (R) Breakout #02: CitiThatNeverSleeps (T) + NYC Housing Pulse (R)
Breakout #03: Team Rocket (T) + Team Nexus (R) Breakout #04: Beyond the Scale (T) + Emergency Care Bears (R)
Breakout #05: CUNY-naughts (T) + Collide-o-Scope (R) Breakout #06: Table for One (T) + xG Analytics (R)
Breakout #07: EconoMates (T) + Zipliners (R) Breakout #08: Crude Awakening (T) + Unemployment Hub (R)
Breakout #09: Degrees of Freedom (T) + TeamTZ (R) Breakout #10: Breathe Easy (T) + Data Vinci (R)

Go to your team’s breakout room. I will adjust / merge rooms based to ensure every room has enough people.

Files and the File System

Files and the File System

The file system is the way your computer organizes and provides access to files:

  • Tree like-structure:
    • Files in folders in folders in folders …
      • Separated by /
      • e.g. STA9750-2026-FALL/data/mp01/data_file.csv
    • End point (or starting point) is the root:
      • Called / on Mac/Linux
      • Drive name on Windows (C:/)

Home Directory

Typically, all user files are stored in a “home directory”:

  • /Users/YOURNAME on Mac/Linux
  • C:/Users/YOURNAME on Windows
  • Subfolders include Downloads, Desktop, Documents, etc
  • Commonly abbreviated as ~
    • My desktop is ~/Desktop
    • My general course material is in ~/STA9750
    • Semester specific material is in ~/STA9750-2026-FALL

Paths

Two ways to specify a file:

  • Absolute path:
    • Starts from root and gives “full name”
    • /Users/michaelweylandt/STA9750/docs/index.html
    • GPS coordinates
  • Relative path:
    • Starts from working directory (getwd()) and gives directions
    • If I am in STA9750, path is just docs/index.html
    • ./ means “this directory”: could also write ./docs/index.html
    • ../ means “up one level”
      • If I were in STA9750/docs, source at ../index.qmd
    • Driving directions

Using the File System in R

Use the fs package to interact with the file system:

  • dir_ls(), dir_create(), dir_exists(), dir_delete()
  • path(), path_home(), path_abs(), path_rel()
  • file_create(), file_exists(), file_delete(), file_info()

Activity #01

Return to breakout rooms to practice file system usage:

  • Convert relative paths to absolute paths
  • List files in a directory
  • Examine metadata the largest file(s) in your STA9750-2026-FALL directory

Accessing Data from the Web: URLs, HTTP, JSON

URLs

URLs are an extension of file paths for the internet:

  • Protocol / Scheme: How data should be transferred
    • HTTP(S), SMS (Texting), POP3/IMAP (Email)
  • Domain: Name of the other computer
    • In practice, often a ‘placeholder’ for something more complex
  • Path: Files on the other computer
    • In practice, hide functionalty behind file-like paths

URLs

From geeksforgeeks.org

Static Data Transfer

R’s basic download.file can be used for downloading simple files:

args(download.file)
function (url, destfile, method, quiet = FALSE, mode = "w", cacheOK = TRUE, 
    extra = getOption("download.file.extra"), headers = NULL, 
    ...) 
NULL

Basic file download capabilities:

  • url: source
  • destfile: where on your computer to store it

Customizable behavior, but defaults often work well:

  • method: what software to use in the background to download
  • mode: is this a text or binary file
  • cacheOK: are you ok with a cached version of the file
  • headers: do you need to send any additional info in your request

download.file()

download.file("https://raw.githubusercontent.com/michaelweylandt/STA9750/refs/heads/main/births.csv", 
              destfile="births.csv")

Note use of relative path here, so saves in current working directory

Be polite:

  • Try to avoid unnecessarily downloading files
  • Save file and only download if !file_exists(destfile)

Data Transfer - download.file()

if(!file_exists("births.csv")){
    download.file("https://raw.githubusercontent.com/michaelweylandt/STA9750/refs/heads/main/births.csv", 
                  destfile="births.csv")
}

read_csv("births.csv")

JSON

JSON:

  • Short for JavaScript Object Notation
  • Popular plain-text representation for hierarchical data.
  • Closer to Python objects (dicts of dicts of dicts) than R data.frames
  • Widely used for web-based data transfer

JSON

Example:

{
    "data": {
        "id": 27992,
        "title": "A Sunday on La Grande Jatte — 1884",
        "image_id": "1adf2696-8489-499b-cad2-821d7fde4b33"
    },
    "config": {
        "iiif_url": "https://www.artic.edu/iiif/2",
    }
}

JSON

Read JSON in R with jsonlite package (alternatives exist)

library(jsonlite)
# A JSON array of primitives
json <- '["Mario", "Peach", null, "Bowser"]'

# Simplifies into an atomic vector
fromJSON(json)
[1] "Mario"  "Peach"  NA       "Bowser"

JSON

json <-
'[
  {"Name" : "Mario", "Age" : 32, "Occupation" : "Plumber"}, 
  {"Name" : "Peach", "Age" : 21, "Occupation" : "Princess"},
  {},
  {"Name" : "Bowser", "Occupation" : "Koopa"}
]'
mydf <- fromJSON(json)
mydf
    Name Age Occupation
1  Mario  32    Plumber
2  Peach  21   Princess
3   <NA>  NA       <NA>
4 Bowser  NA      Koopa

JSON - A Web Standard

fromJSON("https://official-joke-api.appspot.com/random_joke")
$type
[1] "programming"

$setup
[1] "A DHCP packet walks into a bar and asks for a beer."

$punchline
[1] "Bartender says, \"here, but I’ll need that back in an hour!\""

$id
[1] 368

Compare to browser access

Data Transfer: HTTP

HTTP

  • HyperText Transfer Protocol
  • Most common (but not only) internet protocol
  • Also ftp, smtp, ssh, …

“Low-level” mechanism of internet transfer

  • Many R packages add a friendly UX
  • httr2 for low-level work (today)

HTTP

HTTP has two stages:

  • Request
    • URL (Host + Path)
    • Method (VERB)
    • Headers
    • Content
    • Cookies
  • Response
    • Status Code
    • Headers
    • Content

Modern (easy) APIs put most of the behavior in the URL

HTTP in the Browser

In Firefox: Right-Click + Inspect

In Chrome: Right-Click + Developer Tools

HTTP with httr2

httr2 (pronounced “hitter-2”) is low-level manipulation of HTTP.

library(httr2)
request(example_url())
<httr2_request>
GET http://127.0.0.1:59355/
Body: empty

Pretty simple so far:

  • example_url() starts a tiny local web host
  • 127.0.0.1 is localhost

httr2 Requests

Build a request:

  • request
  • req_method
  • req_body_*
  • req_cookies_set
  • req_auth_basic / req_oauth

httr2 Requests

Behaviors:

  • req_cache
  • req_timeout

Execution:

  • req_perform

httr2 Responses

Request status

  • resp_status / resp_status_desc

Content:

  • resp_header*
  • resp_body_*

Live Demo

Demo: Using httr2 to get a random joke from

https://official-joke-api.appspot.com/

API Usage

Exercise - CRAN Logs API

See Part #03 from Today’s Lab

Wrap-Up

Review

Web Data Access

  • Local Files
  • Static Files & download.file
  • HTTP and JSON
  • API Usage

Orientation

  • Communicating Results (quarto) ✅
  • R Basics ✅
  • Data Manipulation in R ✅
  • Data Visualization in R ✅
  • Getting Data into R ⬅️
    • Files and APIs ✅
    • Web Scraping
    • Cleaning and Processing Text
  • Statistical Modeling in R

Next Time

  • Communicating Results (quarto) ✅
  • R Basics ✅
  • Data Manipulation in R ✅
  • Data Visualization in R ✅
  • Getting Data into R ⬅️
    • Files and APIs ✅
    • Web Scraping ⬅️
    • Cleaning and Processing Text
  • Statistical Modeling in R

Pre-Assignment: Structure of a Web Page

Next Time

Musical Treat