---
title: "Data Wrangling Basics: Try It Yourself"
output: html_document
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(echo = TRUE, message = FALSE, warning = FALSE)
```

## How to use this file

Follow along with the lecture.
For each function you will see:

1.  **Example** chunks that are already filled in. Run them with the green arrow (or Ctrl/Cmd + Shift + Enter) and look at the output.
2.  **Your turn** chunks with instructions in comments. Write your code under the comments and run it.

Stuck?
Look for hints, and **solutions are at the bottom** of the file. Try first!

> This file uses the pipe `|>`, which reads as "and then".

```{r load_packages}
library(tidyverse)
```

## 0. The data

Twelve students reported their GPA, their average nightly sleep, and how many hours they studied in each of the first three weeks of the term.

```{r raw_data}
raw <- data.frame(
  ID      = 1:12,
  Yr      = c("Freshman", "Freshman", "Freshman", "Sophomore", "Sophomore", "Sophomore", "Junior", "Junior", "Junior", "Senior", "Senior", "Senior"),
  Hsg     = c("on", "off", "on", "on", "off", "on", "off", "on", "off", "on", "off", "on"),
  GPA     = c(3.2, 3.6, 2.9, 3.4, 3.8, 3.0, 3.5, 3.1, 3.9, 3.3, 3.7, 3.0),
  Sleep   = c(7.5, 8.0, 6.5, 7.0, 7.5, 6.0, 7.0, 6.5, 8.0, 7.0, 7.5, 6.5),
  wk1_hrs = c(10, 14, 8, 12, 16, 9, 15, 11, 18, 13, 17, 10),
  wk2_hrs = c(12, 15, 9, 13, 18, 10, 14, 12, 17, 14, 16, 11),
  wk3_hrs = c(11, 16, 10, 14, 17, 11, 16, 12, 19, 15, 18, 12)
)
```

What do the columns mean?

`ID` = Student ID number

`Yr` = Class year

`Hsg` = Housing: `on` or `off` campus

`GPA` = Grade point average

`Sleep` = Average hours of sleep per night

`wk1_hrs`, `wk2_hrs`, `wk3_hrs` = Study hours in weeks 1, 2, and 3

```{r glimpse-raw}
glimpse(raw)
```

## 1. rename(): give columns clear names

`rename(new_name = old_name)`: the new name goes on the **left**.

**Example.** We will use this cleaned-up `students` data frame for the rest of the file, so run this chunk before moving on.

```{r rename-example}
students <- raw |>
  rename(
    id          = ID,
    year        = Yr,
    housing     = Hsg,
    gpa         = GPA,
    sleep_hours = Sleep,
    study_hours_wk1   = wk1_hrs,
    study_hours_wk2   = wk2_hrs,
    study_hours_wk3   = wk3_hrs
  )

students
```

**Your turn 1.1.** Using the data frame `raw`, rename only `Hsg` to `housing_type`.

```{r rename-ex1}

```

**Your turn 1.2.** Starting from `raw`, rename `GPA` to `gpa` **and** `Sleep` to `sleep_hours` in a single `rename()` call.

```{r rename-ex2}

```

## 2. select(): choose columns

`select()` keeps the columns you name and drops the rest.
Rows are never changed.

**Examples.**

```{r select-example}
students |> select(id, year, gpa)                       # by name
students |> select(id, starts_with("study"))      # by pattern
students |> select(-housing)                            # drop a column with a minus sign
students |> select(id, gpa:study_hours_wk1)             # a range of neighboring columns
```

**Your turn 2.1.** Select `id`, `year`, and `housing`.

```{r select-ex1}

```

**Your turn 2.2.** Select every column **except** `sleep_hours`.

```{r select-ex2}

```

**Challenge:** Select every column that contains the word `hours"`. 
Hint: You should use the function `contains`.

```{r select-ex3}

```

## 3. filter(): choose rows

`filter()` keeps the rows where a test is `TRUE`.
A comma means **AND**; `|` means **OR**.

**Examples.**

```{r filter-example}
students |> filter(gpa >= 3.5)                          # one condition
students |> filter(year == "Senior", housing == "off")  # AND
students |> filter(sleep_hours < 7 | gpa > 3.8)         # OR
students |> filter(year %in% c("Junior", "Senior"))     # one of several values
```

Remember: use `==` (two equals signs) to test for equality, and put text in quotes.

**Your turn 3.1.** Keep students who sleep at least 7.5 hours a night. 
Hint: there should be 5 rows!

```{r filter-ex1}

```

**Your turn 3.2.** Keep students who live on campus **and** have a GPA below 3.2. 
Hint: there should be 4 rows!

```{r filter-ex2}

```

**Your turn 3.3.** Keep students whose GPA is between 3.0 and 3.5 (inclusive). Hint: there should be 7 rows. 
Hint: two conditions separated by a comma, 
Challenge: try using the `between()` function

```{r filter-ex3}

```

## 4. slice(): choose rows by position or rank

`slice()` picks rows by *where they are* (position) or by *how they rank*, instead of by a test.

**Examples.**

```{r slice-example}
students |> slice_head(n = 3)       # first 3 rows
students |> slice_tail(n = 2)       # last 2 rows
students |> slice_max(gpa, n = 3)   # the 3 highest GPAs
students |> slice_min(gpa, n = 3)   # the 3 lowest GPAs

students |> slice_sample(n = 3)     # 3 random rows
```

**Your turn 4.1.** Randomly sample 5 `students`. Then run run your code a second time. Did you get the same student IDs?

```{r slice-ex1}

```

**Your turn 4.2.** Find the 2 students who sleep the most.

```{r slice-ex2}

```

**Your turn 4.3.** Find the single student who sleeps the least.

```{r slice-ex3}

```

## 5. mutate(): create or change columns

`mutate(new_column = some calculation)` adds a column and keeps all rows.
You can use a column you just created in the next line.

**Example.** We will keep these new columns, so run this chunk before moving on.

```{r mutate-example}
students <- students |>
  mutate(
    total_study = study_hours_wk1 + study_hours_wk2 + study_hours_wk3,
    avg_study   = total_study / 3
  )

students |> select(id, starts_with("study_hours"), total_study, avg_study)
```

**Your turn 5.1.** Create `sleep_gap`, how many hours short of 8 each student sleeps.
Show it next to `id` and `sleep_hours`.
Hint: this requires also using the select() function

```{r mutate-ex1}

```

**Your turn 5.2.** Create `gpa_pct`, GPA as a percentage of a 4.0 (`gpa / 4 * 100`).

```{r mutate-ex2}

```

## 6. pivot_longer(): wide to long

Right now the three weekly hours columns are really **one variable** (hours studied) measured at three times (week).
`pivot_longer()` stacks them into two columns: one for the week and one for the value.
Long data is what most plotting and summarizing functions expect.

- `cols`: which columns to stack
- `names_to`: name for the new column that holds the old column names
- `values_to`: name for the new column that holds the numbers

**Example.**

```{r pivot-example}
hours_long <- students |>
  select(id, year, starts_with("study_hours_wk")) |>
  pivot_longer(
    cols            = starts_with("study_hours_wk"),
    names_to        = "week",
    names_prefix    = "study_hours_wk", # strips "study_hours_wk", leaving 1, 2, 3
    names_transform = as.integer,       # makes the week a number
    values_to       = "hours"
  )

hours_long
```

12 students times 3 weeks gives 36 rows.

**Your turn 6.1.** Run `pivot_longer()` again in the chunk below, but leave out `names_prefix` and `names_transform`, and call the new column `week_label`.
What do the values in `week_label` look like now?

```{r pivot-ex1}
# Start from: students |> select(id, year, starts_with("hours_wk")) |> pivot_longer(...)

```

**Your turn 6.2.** Starting from `hours_long`, keep only week 3 (use `filter()`).

```{r pivot-ex2}

```

## 7. group_by() and summarize(): one row per group

`group_by()` tells R to treat each group separately.
`summarize()` then collapses each group to **one row** of statistics.
`n()` counts the rows in each group.

**Examples.**

```{r summarize-example}
# one grouping variable
students |>
  group_by(year) |>
  summarize(
    n          = n(),
    mean_total = mean(total_study),
    mean_gpa   = mean(gpa)
  )

# two grouping variables, using the long data
hours_long |>
  group_by(year, week) |>
  summarize(mean_hours = mean(hours), 
            .groups = "drop")
```

`.groups = "drop"` removes the grouping from the result so it does not carry into later steps.

**Your turn 7.1.** Find the mean GPA for on-campus and off-campus students.

```{r summarize-ex1}

```

**Your turn 7.2.** Count how many students in each year and live in each housing type. Hint: you will use two grouping variables and the function `n()`

```{r summarize-ex2}

```

**Your turn 7.3.** Find the highest `sleep_hours` in each class year. 
Hint: you'll need the function `max()`

```{r summarize-ex3}

```

**Your turn 7.4.** Using `hours_long`, find the average number of hours spent studying for each week (across all students). Is studying going up over time?

```{r summarize-ex4}

```

## Challenge: put it all together

Do on-campus or off-campus students study more per week, on average?
Build **one pipeline** that starts from `raw` and uses at least `rename()`, `mutate()`, and `group_by()` with `summarize()`.

```{r challenge}

```

------------------------------------------------------------------------

## Solutions

Try each exercise before looking here!

```{r solutions-rename}
raw |> rename(housing_type = Hsg)                         # 1.1
raw |> rename(gpa = GPA, sleep_hours = Sleep)             # 1.2
```

```{r solutions-select}
students |> select(id, year, housing)                     # 2.1
students |> select(-sleep_hours)                          # 2.2
students |> select(contains("hours"))                     # 2.3
```

```{r solutions-filter}
students |> filter(sleep_hours >= 7.5)                    # 3.1
students |> filter(housing == "on", gpa < 3.2)            # 3.2
students |> filter(gpa >= 3.0, gpa <= 3.5)                # 3.3 (or between(gpa, 3.0, 3.5))
```

```{r solutions-slice}
students |> slice_sample(n = 5)                           # 4.1
students |> slice_max(sleep_hours, n = 2)                 # 4.2
students |> slice_min(sleep_hours, n = 1)                 # 4.3
```

```{r solutions-mutate}
students |> mutate(sleep_gap = 8 - sleep_hours) |> select(id, sleep_hours, sleep_gap)  # 5.1
students |> mutate(gpa_pct = gpa / 4 * 100) |> select(id, gpa, gpa_pct)                # 5.2
```

```{r solutions-pivot}
students |>
  select(id, year, starts_with("study_hours_wk")) |>
  pivot_longer(starts_with("study_hours_wk"), names_to = "week_label", values_to = "hours")   # 6.1

hours_long |> filter(week == 3)                                                               # 6.2
```

```{r solutions-summarize}
students |> group_by(housing) |> summarize(mean_gpa = mean(gpa))                 # 7.1
students |> group_by(year, housing) |> summarize(n = n(), .groups = "drop")      # 7.2
students |> group_by(year) |> summarize(max_sleep = max(sleep_hours))            # 7.3
hours_long |> group_by(week) |> summarize(mean_hours = mean(hours))              # 7.4
```

```{r solutions-challenge}
raw |>
  rename(housing = Hsg,
         study_hours_wk1 = wk1_hrs, study_hours_wk2 = wk2_hrs, study_hours_wk3 = wk3_hrs) |>
  mutate(avg_study_hours = (study_hours_wk1 + study_hours_wk2 + study_hours_wk3) / 3) |>
  group_by(housing) |>
  summarize(n = n(), mean_weekly_hours = mean(avg_study_hours))
```