Due in 6 Days Four already submitted - fantastic!
(Hopefully an excess of preparedness, not a last minute panic)
Very impressed by Detailed Analyses, Fancy Tables, Graphics
You don’t need fancy graphics yet, but I love to see above and beyond
Make sure your code is included in your submission
Follow submission instructions
You need to have mp01.qmd and docs/mp01.html in your STA9750-2025-SPRING repository
Helper function can verify submission is properly formatted
Your classmates have asked several excellent questions on Piazza.
Reminders:
To be assigned on GitHub in one week (March 5th):
Use helper functions to find submissions assigned to you. Ask on Piazza if still having trouble.
Submissions may not map perfectly to rubric - use your best judgement
Learn from this! What can you adapt for MP#02?
MP#02 released - Public Transit
Due 2025-03-26 at 11:45pm ET
Pay attention to the rubric
Topics
Brightspace - Wednesdays at 11:45
Next pre-assignment is March 20th - few weeks off
Apologies for misleading question this week. I forgot to change it when giving the extension
Long list of helpful resources added on Piazza, esp. https://learnr.numbat.space/
Ask questions! This course is demanding but rewarding.
Social contract: I push you hard, but I also provide lots of support.
2 teams already formed
All team commitments due via email 2025-03-05 at 11:45pm ET
Multiple Tables:
Will create new (compound) rows - be ware of accidental over-duplication
Keys are unique identifiers for individual records
The history of corporate IT is largely one of (failed) primary keys
Meaningful true keys are vanishingly rare - cherish them when you find them
Often ‘unique enough’ for an analysis
dplyr::count is helpful here
Joins combine tables by identity - not simple ‘stacking’
Specify a join key - ideally this is an actual key, but doesn’t have to be
In dplyr, we use the join_by function:
Here table_1_name and table_2_name are column names from two tables
Join rows where these values are equal (advanced joins possible)
When tables are perfectly matched, not an issue:
library(dplyr)
x <- tribble(~college, ~campus_borough,
"CCNY", "Manhattan",
"Baruch", "Manhattan",
"CSI", "Staten Island",
"York", "Queens")
y <- tribble(~borough_name, ~bus_code,
"Manhattan", "M",
"Staten Island", "S",
"Queens", "Q")
inner_join(x, y, join_by(campus_borough == borough_name))# A tibble: 4 × 3
college campus_borough bus_code
<chr> <chr> <chr>
1 CCNY Manhattan M
2 Baruch Manhattan M
3 CSI Staten Island S
4 York Queens Q Default to inner but irrelevant
Note automatic repetition of "M" row
How to handle ‘unaligned’ values?
x <- tribble(~college, ~campus_borough,
"CCNY", "Manhattan",
"Baruch", "Manhattan",
"CSI", "Staten Island",
"York", "Queens",
"Medgar Evers", "Brooklyn")
inner_join(x, y, join_by(campus_borough == borough_name))# A tibble: 4 × 3
college campus_borough bus_code
<chr> <chr> <chr>
1 CCNY Manhattan M
2 Baruch Manhattan M
3 CSI Staten Island S
4 York Queens Q MEC vanished!
# A tibble: 5 × 3
college campus_borough bus_code
<chr> <chr> <chr>
1 CCNY Manhattan M
2 Baruch Manhattan M
3 CSI Staten Island S
4 York Queens Q
5 Medgar Evers Brooklyn <NA> MEC stays, but no bus code - NA value
inner_join - Keep only matchesleft_join - Keep all rows in left (first) table even w/o matchesright_join - Keep all rows in right (second) table even w/o matchesfull_join - Keep all rows from both tables, even w/o matchesleft_ and right_ are types of ‘outer’ joins
[W]ill we be learning how to perform joins within a subquery?
You don’t need subqueries in R since it’s an imperative language. Just create a new variable to represent the result of the subquery and use that in the next command.
SELECT first_name, last_name
FROM collectors
WHERE id IN (
SELECT collector_id
FROM sales
);collector_ids <- sales |> pull(collector_id)
collectors |> filter(id %in% collector_ids) |> select(first_name, last_name)[H]ow can we ensure that the information [resulting from a join] is accurate and not repeated?
Will joining large data sets […] affect performance?
Somewhat - larger data sets are always slower.
Bigger danger is “bad joins” creating huge data automatically.
Note that R is less “smart” than SQL, so won’t optimize execution order for you automatically.
pivot_wider?Is [
pivot_wider] just for formatting?
library(dplyr); library(tidyr); library(palmerpenguins)
penguins |> drop_na() |>
group_by(sex, species) |>
summarize(weight = mean(body_mass_g)) |>
pivot_wider(id_cols=species,
names_from=sex,
values_from=weight) |>
mutate(gender_diff = male - female)# A tibble: 3 × 4
species female male gender_diff
<fct> <dbl> <dbl> <dbl>
1 Adelie 3369. 4043. 675.
2 Chinstrap 3527. 3939. 412.
3 Gentoo 4680. 5485. 805.dplyr joins vs SQL joinsWhat is the difference between
dplyrandSQLjoins?
Not too much - biggest difference is no INDEX or FOREIGN KEY in R so less guarantees of data integrity.
anti_join?Rare: looking for unmatched rows. - Useful to find data integrity issues or ‘implicit’ missingness. - I use an anti_join to find students who haven’t submitted an assignment.
many-to-many WarningTricky to address, but fortunately pretty rare.
SQL explicitly forbids many-to-manyx and y tablesNo automatic way. Some rules of thumb:
full_outer join is a bit dangerouscross_join is rarely the right answertidyr vs dplyrIs
tidyrmore efficient thandplyr?
Nope - different packages from the same developers.
Designed to work together elegantly.
Please explain what
cross_join, filter joins, andnest_joinare?
cross_join: dangerous.
anti_, semi_):
nest_join: beyond this course.
left_join with extra structure to output.How do I decide which type of join is most approriate for a given analysis?
Topic of today’s work.
dplyr - Joins and PivotsData Set: nycflights13
Exercise: Lab #05
pivot_ changes the shape of a data set
Q: Which penguin species has the largest between-sex mass difference?
library(tidyr)
library(palmerpenguins)
avg_mass_tbl <- penguins |> drop_na() |>
group_by(sex, species) |>
summarize(avg_mass = mean(body_mass_g)) |>
ungroup()
avg_mass_tbl# A tibble: 6 × 3
sex species avg_mass
<fct> <fct> <dbl>
1 female Adelie 3369.
2 female Chinstrap 3527.
3 female Gentoo 4680.
4 male Adelie 4043.
5 male Chinstrap 3939.
6 male Gentoo 5485.We want data that is wider than our current data:
| species | male_avg | female_avg |
|---|---|---|
| Adelie | … | … |
| Chinstrap | … | … |
| Gentoo | … | … |
# A tibble: 3 × 3
species female male
<fct> <dbl> <dbl>
1 Adelie 3369. 4043.
2 Chinstrap 3527. 3939.
3 Gentoo 4680. 5485.pivot_wider Aguments:
id_cols: kept as ‘keys’ for new tablenames_from: existing column ‘spread’ to create new columns namesvalues_from: values in new tablepivot_longer:
Next Wednesday (March 5th) at 11:45pm:
No class on March 6th
Following Wednesday (March 12th) at 11:45pm:
March 13th: Proposal Presentations
West Virginia State Board of Education v. Barnette (March 11, 1943)
Students cannot be compelled to recite the Pleldge of Allegiance, even during a period of war
Iconic First Amendment Victory
Justice Jackson’s Opinion (6-3):
If there is any fixed star in our constitutional constellation, it is that no official, high or petty, can prescribe what shall be orthodox in politics, nationalism, religion, or other matters of opinion or force citizens to confess by word or act their faith therein.
1940 Case Minersville School District v. Gobitis
Justice Frankfurter (8-1 Majority Opinion):
National Unity is the basis of National Security
Students could be forced to pledge
After the decision, waves of violence against JW students and adults accused of “treason” against the war effort
Justice Stone (Dissent):
[T]he guarantees of civil liberty are but guarantees of freedom of the human mind and spirit and of reasonable freedom and opportunity to express them. [… T]he very essence of the liberty which they guarantee is the freedom of the individual from compulsion as to what he shall think and what he shall say.
A few years later, changed court wanted to revisit, leading to Barnette
As governmental pressure toward unity becomes greater, so strife becomes more bitter as to whose unity it shall be.[…] Those who begin coercive elimination of dissent soon find themselves exterminating dissenters. Compulsory unification of opinion achieves only the unanimity of the graveyard.
Authority [in the United States] is to be controlled by public opinion, not public opinion by authority.
[F]reedom to differ is not limited to things that do not matter much. That would be a mere shadow of freedom. The test of its substance is the right to differ as to things that touch the heart of the existing order.
