From Data to Manuscript in R

Author

Natalie Dowling

Published

June 1, 2025

Preface

This is an in-development textbook for the sequence “Data to Manuscript in R” offered at the University of Chicago. What you’re seeing is in very, very early stages, with lots of blank spots, placeholders, unnecessary sections, and random notes and ramblings instead of clear, concise instruction. There’s also a lot of content redundant with content in other, more reliable, course resources, like the course slides and website. If you keep exploring the content even knowing what a mess it is, consider yourself warned!

This book is “published” as a resource for current students in the course as an example of how to use RStudio and Quarto to create and share documents. Students can also contribute to the book for course credit.

Outside of what I assign as readings for D2M-R, this book probably isn’t the best resource for learning R, especially if you aren’t currently a student in the class who will get all my caveats and asterisks.

If you’re looking for stable and complete resources to learn R, I recommend:

Outline

Based on current slides, but definitely not ideal for the book…

One of the main goals for the book is to be a supplement to class slides. The slides are currently way too verbose, so the idea is to migrate the details and nitty-gritty stuff away from the slides and into this book, which can be read at the student’s own pace and be even more verbose when needed. Right now, there’s a lot of overlap and a lot of seemingly random cut-and-paste.

If you’re a student looking to contribute to the book for a core skills project, one way to jump in is to look through the slides and see where things could be migrated to the book to tighten up the slides and expand the book.

Quarter 1

  1. Introduction
    1. PSYC 30550 D2MR - Overview
      1. The class itself - link to course page
      2. Assessment structure
    2. D2MR Workflow
    3. This book
      1. The goals
      2. How to use it
      3. How to contribute
    4. Getting Started/Setting up (in the form of walking through a dummy repo)
      1. Download and install R and RStudio
      2. Get familiar with RStudio
      3. R Notebooks
        1. Preview quarto
      4. Get Git
      5. Connect RStudio to GitHub
    5. (Optional) Set up Copilot in RStudio
  2. Solving Problems
    1. Introduction - why this is a whole section’
    2. Using AI
    3. Best practices - keep your code:
      1. Standardized
      2. Intelligible
      3. Maintainable
      4. Contextualized
    4. Debugging & Troubleshooting
      1. Documentation
        1. Help/lookup functions
        2. Function documentation
        3. Package documentation
        4. Resources
      2. Solving your own problems
        1. Error messages
        2. Garbage in, garbage out (and other tropes)
        3. Line-by-line debugging
        4. Rubber ducking
      3. Ask the internet
        1. Start strong - asking good questions
        2. Crowdsource
        3. Use AI constructively
      4. Ask humans – for D2MR Students
        1. Ask your classmates
        2. Use Slack
        3. Ask your instructor & TAs
        4. The D2MR troubleshooting process
      5. Starting from “Nothing”
  3. RStudio Essentials
    1. RStudio
      1. Overview + what’s an IDE
      2. Layout
        1. Source pane
        2. Console pane
        3. The rest of the panes
          1. Environment
          2. History
          3. Files
          4. Plots
          5. Connections
          6. Packages
          7. Help
          8. Build
          9. VCS
          10. Tutorial
          11. Viewer
          12. Presentations
    2. Keyboard shortcuts
    3. Customizing RStudio
    4. File types - just a preview, we’ll cover these in more detail later!
      1. .Rmd
      2. .qmd
      3. .R
      4. .Rproj
      5. Lots of metadata files you can mostly ignore (including literally in your .gitignore)
        1. .Rhistory
        2. .RData
        3. .Rprofile
        4. .Rproj.user
  4. Git & GitHub Essentials
    1. Introduction – the point of version control
    2. Git
      1. Overview & workflow musts
      2. Git lingo
        1. Repositories
          1. repository / repo
          2. initialize
          3. clone
          4. branch & checkout
        2. Version control
          1. commit
          2. stage
          3. fetch
          4. pull
          5. push
        3. Merging
          1. diff
          2. merge
          3. merge conflict
          4. rebase
          5. fast-forward (ff)
          6. squash
          7. cherry-pick
          8. stash
        4. Remote repositories
          1. remote
          2. origin
          3. upstream
          4. fork
          5. pull request
        5. (Optional) Using Git in the terminal
    3. Repo structure
      1. What does and does not go in a repo?
      2. Top-level essentials
        1. README
        2. .gitignore
      3. The rest of your stuff
      4. Metadata and information
    4. GitHub
      1. What is GitHub? How is it different from Git?
      2. GitHub features
        1. Issues
        2. Pull requests
        3. Pages
        4. Copilot
        5. More
          1. Actions
          2. Projects
          3. Codespaces
      3. Interfacing with GitHub (without RStudio)
        1. GitHub website
        2. GitHub Desktop
        3. Other options
          1. VS Code and other IDEs
          2. GitHub CLI
      4. Interfacing with GitHub in RStudio
        1. Connecting RStudio to GitHub
        2. Using GitHub in RStudio
          1. Cloning a repo
          2. Committing changes
          3. Pushing changes
          4. Pulling changes
          5. Creating pull requests
    5. Git Pains
      1. Common issues
      2. Helpful resources
      3. The nuclear option
  5. R Language Essentials
    1. Introduction
      1. What is R and why use it?
      2. Object-oriented programming
    2. R syntax
      1. Variables
      2. Functions
      3. Data types
      4. Operators
    3. R data structures
      1. Vectors
      2. Lists
      3. Matrices
      4. Data frames
      5. Tibbles
    4. R packages
      1. What are packages?
      2. Installing and loading packages
      3. Same function, different packages, oh no!
        1. Functions with different names in different packages that all do the same thing
        2. Functions with the same name in different packages that do different things
      4. Commonly used packages in D2MR
  6. R Programming
    1. Essential concepts in base R
      1. Object assignment
        1. Creating dummy variables and dataframes
      2. Indexing and subsetting with [] & $
      3. (more base R essentials)
    2. Iteration
      1. Conditional statements
        1. if else
        2. case_when
      2. Loops
        1. for loops
        2. while loops
    3. Functions in R
      1. Writing functions
      2. Function arguments and return values
      3. Scope and environments
    4. Regular expressions
      1. What is regex? What’s the point?
      2. Basic syntax
      3. Common use cases
  7. Welcome to the Tidyverse!
    1. Introduction to the Tidyverse & tidy data
      1. Overview of the tidy ecosystem
      2. Tidy data principles
      3. Core packages in the Tidyverse and general functions
      4. Terminology
    2. Importing and exporting with readr
      1. Overview
        1. Tabular data - what counts?
          1. File types
          2. R objects, including tibbles
        2. Reading, writing, rereading (intermediate datasets)
      2. Reading data with read_* functions
      3. Writing data with write_* functions
      4. Other packages
        1. readxl for Excel files
        2. haven for SPSS, SAS, and Stata files
        3. googlesheets4 for Google Sheets
        4. jsonlite for JSON files
        5. DBI and dbplyr for databases
      5. Handling common import/export issues
    3. Data manipulation with dplyr
      1. Introduction to dplyr
        1. the points of pipelines (highly readable, but verbose)
        2. The pipe operator (%>%) and magrittr
        3. Chaining operations with pipes
      2. Selecting data
        1. select, rename
        2. filter
        3. arrange
      3. Manipulating data
        1. mutate
        2. summarize
        3. group_by
        4. distinct
        5. count
      4. dplyr practice
    4. Data tidying with tidyr
      1. What counts as “tidying” data?
        1. Remember what “tidy” means in the tidyverse
        2. Tidying is reshaping and systematically cleaning data
      2. Reshape data
        1. pivot_longer
        2. pivot_wider
        3. cast, melt, gather, spread, etc.
      3. Combine and split cells
        1. unite
        2. separate functions
      4. Expand tables
        1. expand
        2. complete
      5. Handle missing values
        1. drop_na
        2. fill
        3. replace_na
      6. Advanced: Nested data
        1. What is nested data and why might you use it?
        2. We’re not going to cover this in D2MR, but tidyr has functions for creating, reshaping, and transforming nested data
  8. Working with different data types in the tidyverse
    1. Why specialized packages exist for text, factors, and dates
      1. Base R can do this stuff, but it’s not always great
      2. Tidyverse functions are useful:
        1. Consistent syntax
        2. More intuitive functions
        3. More intuitive pattern matching (stringr)
        4. Seamlessly compatible with dplyr, tidyr, and the rest of the tidyverse
    2. Text data with stringr
      1. Overview
        1. Remember what strings and character vectors are?
        2. Remember what regex is?
        3. Special things about strings (using quotes, escaping characters, etc.)
        4. Base R can do string stuff - Useful base R string functions
          1. paste & paste0
          2. c
          3. toupper, tolower
          4. is.character
          5. toString
          6. Pattern matching with grep family
      2. Match strings
        1. str_detect
        2. str_starts
        3. str_ends
        4. str_count
      3. Subsetting and length
        1. str_sub
        2. str_subset
        3. str_length
        4. str_pad
        5. str_trunc
        6. str_trim
      4. Mutate, join, split
        1. str_sub (again)
        2. str_replace, str_replace_all
        3. str_remove, str_remove_all
        4. str_to_lower, str_to_upper, str_to_title
        5. str_split
        6. str_c, str_glue
        7. str_flatten
    3. Factors with forcats
      1. Overview
        1. What are factors?
        2. They look like strings, how are they different? Why do we have to have a whole separate package for them?
          1. Levels
          2. Order
          3. Closed set
      2. Base R can do factor stuff (and unlike with strings, you’ll use these base R factor functions a lot!)
        1. factor
        2. levels
        3. relevel
        4. other base factor functions are mostly variations on these
      3. Useful forcats functions
        1. fct_relevel
        2. fct_reorder
        3. fct_rev
        4. fct_recode
        5. fct_collapse
        6. fct_other
        7. fct_drop
        8. fct_expand
    4. Dates and times with lubridate
      1. Overview
        1. What are dates and times?
        2. Why do we need a whole package for them?
        3. Base R can do date and time stuff, but it’s not great
          1. as.Date
          2. as.POSIXct
          3. as.POSIXlt
          4. strptime
      2. lubridate functions
        1. Parsing dates and times
          1. ymd, mdy, dmy, ymd_hms, etc.
          2. parse_date_time
        2. Extracting components
          1. year, month, day, hour, minute, second
          2. wday, yday, mday
        3. Manipulating dates and times
          1. %m+% and %m-% for adding and subtracting time
          2. interval and duration
          3. floor_date, ceiling_date, round_date
        4. Formatting dates and times
          1. format
          2. strftime
  9. Basic visualization and summary statistics
    1. This chapter is not the ggplot2 chapter or a deep stats chapter. It’s a connection between the two quarters of the class.
    2. Putting things together
      1. how data manipulation in the tidyverse leads to easier visualization
      2. how visualization leads to better understanding of your data
      3. how visualization and summary statistics fit together
    3. Distributions of continuous variables
      1. Histograms
      2. Box plots
      3. Summary statistics: mean, median, variance, standard deviation
    4. Comparing groups
      1. Bar plots
      2. Summary statistics: grouped means, medians, etc.
      3. t-tests
      4. More advanced for later: ANOVA, chi-squared tests
    5. Relationships between continuous variables
      1. Scatter plots and smoothing lines
      2. Correlation
      3. Basic linear regression
      4. More advanced for later: Multiple regression, logistic regression
  10. Midpoint review

Quarter 2

Appendix

  1. Glossary
  2. References