8 Data Manipulation with dplyr: Streamlined Data Wrangling in R

dplyr revolutionizes data manipulation in R by offering a concise, human-readable grammar of data transformation. Instead of nested function calls, you work with five core verbs—filter, select, mutate, summarize, and join—combined through the pipe operator (%>%) to express complex operations in a linear, intuitive flow. In this post, you’ll learn how to:

• Filter rows with filter()

• Pick columns with select()

• Create or transform variables using mutate()

• Aggregate data with group_by() and summarize()

• Merge tables via left_join() and inner_join()

By mastering these verbs, you’ll wrangle raw data into analysis-ready forms with minimal code and maximal clarity.

1. Getting Started with dplyr

Before diving into examples, install and load the tidyverse ecosystem, which includes dplyr:

r

install.packages("tidyverse")   # installs dplyr, ggplot2, tidyr, etc.
library(dplyr)

dplyr works on data frames and tibbles. Convert base data frames to tibbles for enhanced printing and subsetting behavior:

r

df <- as_tibble(iris)

You’ll use the pipe operator %>% from the magrittr package to chain commands. The basic pattern:

r

df %>%
  verb1(args) %>%
  verb2(args) %>%
  verb3(args)

This reads left to right: take df, apply verb1(), then verb2(), and so on.

2. Filtering Rows with filter()

The filter() function subsets rows based on logical conditions. It supports multiple conditions combined with & (AND), | (OR), and ! (NOT).

2.1 Basic Usage

r

# Keep only species setosa
iris %>%
  filter(Species == "setosa")

2.2 Multiple Conditions

r

# Sepal length > 5 AND petal width < 1.5
iris %>%
  filter(Sepal.Length > 5 & Petal.Width < 1.5)

2.3 Using OR and NOT

r

# Either virginica OR versicolor
iris %>%
  filter(Species == "virginica" | Species == "versicolor")

# Exclude setosa
iris %>%
  filter(!(Species == "setosa"))

2.4 Filtering with %in%

r

# Keep only two species
iris %>%
  filter(Species %in% c("setosa", "versicolor"))

filter() works seamlessly with date and factor variables. Always check str() or use glimpse() to confirm data types.

3. Selecting Columns with select()

While filter() narrows rows, select() trims columns. It supports helper functions like starts_with(), ends_with(), contains(), and matches() for dynamic selection.

3.1 Basic Column Selection

r

iris %>%
  select(Sepal.Length, Sepal.Width, Species)

3.2 Helper Functions

r

# All measurements except species
iris %>%
  select(starts_with("Sepal"))

# Columns containing “Width”
iris %>%
  select(contains("Width"))

# Columns matching regex
iris %>%
  select(matches("Length|Width"))

3.3 Renaming on the Fly

select() can rename columns while selecting:

r

iris %>%
  select(
    sepal_length = Sepal.Length,
    sepal_width  = Sepal.Width,
    species
  )

3.4 Dropping Columns

Precede column names with - to drop them:

r

iris %>%
  select(-Species)

select() preserves column order by default; reorder by re-specifying positions.

4. Creating New Variables with mutate()

mutate() adds new columns or transforms existing ones. Under the hood, all operations are vectorized for speed.

4.1 Adding Simple Variables

r

iris %>%
  mutate(
    Sepal.Ratio = Sepal.Length / Sepal.Width,
    Petal.Area  = Petal.Length * Petal.Width
  )

4.2 Overwriting Existing Columns

r

iris %>%
  mutate(
    Sepal.Length = Sepal.Length * 10   # convert from cm to mm
  )

4.3 Using Conditional Logic

Embed if_else() for vectorized conditionals:

r

iris %>%
  mutate(
    LargeSepal = if_else(Sepal.Length > 5, TRUE, FALSE)
  )

4.4 Chaining Mutations

Complex pipelines can apply multiple mutate() calls:

r

iris %>%
  mutate(Sepal.Ratio = Sepal.Length / Sepal.Width) %>%
  mutate(Petal.Ratio = Petal.Length / Petal.Width)

Or combine them into one call for efficiency.

5. Aggregating with group_by() and summarize()

Exploratory analysis often requires calculating summary statistics by group. group_by() and summarize() work in tandem to achieve this.

5.1 Grouping Data

r

by_species <- iris %>%
  group_by(Species)

5.2 Summarizing

r

by_species %>%
  summarize(
    count       = n(),                     # row count in each group
    mean_sepal  = mean(Sepal.Length, na.rm = TRUE),
    sd_petal    = sd(Petal.Length, na.rm = TRUE)
  )

n() returns group size; n_distinct() counts unique values.

5.3 Multiple Aggregations

You can compute multiple summaries in one call:

r

iris %>%
  group_by(Species) %>%
  summarize(
    count        = n(),
    avg_petal    = mean(Petal.Length),
    max_sepal    = max(Sepal.Width),
    med_sepal    = median(Sepal.Length),
    sepal_range  = max(Sepal.Length) - min(Sepal.Length)
  )

5.4 Ungrouping

After summarization, remove grouping to avoid unintended behavior downstream:

r

iris %>%
  group_by(Species) %>%
  summarize(avg_sep = mean(Sepal.Length)) %>%
  ungroup()

6. Merging Tables with left_join() and inner_join()

Combining datasets is crucial when your data spans multiple tables (e.g., customer info, transactional records). dplyr’s join functions mirror SQL semantics.

6.1 Understanding Join Types

• inner_join(): returns only rows with matching keys in both tables

• left_join(): keeps all rows from the left table, adds matching columns from the right, filling with NA when no match

• right_join(): mirror of left_join()

• full_join(): union of left and right, preserving all rows

6.2 Basic left_join()

Assume two tibbles: sales and products, linked by product_id.

r

sales <- tibble(
  product_id = c(101, 102, 103, 104),
  quantity   = c(2, 5, 3, 4)
)

products <- tibble(
  product_id = c(101, 102, 103),
  price      = c(9.99, 19.99, 14.99)
)

sales %>%
  left_join(products, by = "product_id")

Rows with product_id = 104 will have price = NA.

6.3 inner_join()

r

sales %>%
  inner_join(products, by = "product_id")

Only products 101–103 appear; unmatched rows drop.

6.4 Joining on Multiple Keys

If tables share multiple keys, specify a character vector:

r

orders %>%
  left_join(customers, by = c("cust_id" = "id", "region_id" = "region"))

6.5 Handling Name Conflicts

When both tables have columns with the same name aside from keys, dplyr appends suffixes .x and .y by default. Customize using suffix = c(".sales", ".prod").

r

sales %>%
  left_join(products, by = "product_id", suffix = c(".sales", ".prod"))

7. Putting It All Together: A Practical Example

Imagine you have a raw dataset of online orders and a master product catalog. You need to filter orders in the last quarter, calculate total revenue per order, and summarize revenue by product category.

r

library(dplyr)

orders   <- read_csv("data/orders.csv")        # order_id, product_id, quantity, order_date
products <- read_csv("data/products.csv")      # product_id, category, unit_price

# Pipeline
revenue_by_category <- orders %>%
  filter(order_date >= "2023-10-01" & order_date <= "2023-12-31") %>%
  left_join(products, by = "product_id") %>%
  mutate(revenue = quantity * unit_price) %>%
  group_by(category) %>%
  summarize(
    total_orders  = n(),
    total_revenue = sum(revenue, na.rm = TRUE),
    avg_revenue   = mean(revenue, na.rm = TRUE)
  ) %>%
  arrange(desc(total_revenue)) %>%
  ungroup()

revenue_by_category

This pipeline illustrates how dplyr verbs complement each other:

1. filter() narrows to Q4 orders

2. left_join() brings in pricing and category

3. mutate() computes revenue per order

4. group_by() + summarize() aggregates by category

5. arrange() orders results for reporting

8. Performance Tips and Best Practices

• Work on tibbles, not data frames: tibbles print better and avoid partial matching quirks.

• Minimize copying: chaining on large objects can incur memory overhead; break pipelines into chunks if needed.

• Use database backends: connect a database via tbl() to let dplyr translate operations into SQL, pushing computation to the server.

• Index your keys: when working on database tables, index columns used in joins for faster lookups.

• Profile your code: packages like profvis or bench help identify bottlenecks.

9. Conclusion and Next Steps

You’ve mastered the five core dplyr verbs—filter(), select(), mutate(), group_by() + summarize(), and join()—and seen how they combine into powerful, readable pipelines. With these tools, you can wrangle almost any tabular data into the shape your analysis requires.

In the next post, we’ll explore Data Visualization with ggplot2, where you’ll learn to transform your cleaned and summarized data into compelling charts and dashboards. If you have questions, ideas for further examples, or tips from your own projects, please share them in the comments below. Happy wrangling!