Using Dplyr
dplyr package is about data frames. Each observation is one row and each column is one variable or measure
dplyr Properties
The first argument is a data frame.
The subsequent arguments describe what to do with it, and you can refer to columns in the data frame directly without using the $ operator (just use the names).
The result is a new data frame
Data frames must be properly formatted and annotated for this to all be useful
Using The Functions
First download data from: https://raw.github.com/DataScienceSpecialization/courses/master/03_GettingData/dplyr/chicago.rds
library(dplyr)##
## Attaching package: 'dplyr'
##
## The following object is masked from 'package:stats':
##
## filter
##
## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionoptions(width=105)
chicago <- readRDS("./data/chicago.rds")
str(chicago)## 'data.frame': 6940 obs. of 8 variables:
## $ city : chr "chic" "chic" "chic" "chic" ...
## $ tmpd : num 31.5 33 33 29 32 40 34.5 29 26.5 32.5 ...
## $ dptp : num 31.5 29.9 27.4 28.6 28.9 ...
## $ date : Date, format: "1987-01-01" "1987-01-02" "1987-01-03" ...
## $ pm25tmean2: num NA NA NA NA NA NA NA NA NA NA ...
## $ pm10tmean2: num 34 NA 34.2 47 NA ...
## $ o3tmean2 : num 4.25 3.3 3.33 4.38 4.75 ...
## $ no2tmean2 : num 20 23.2 23.8 30.4 30.3 ...Use the names function to get the names of the columns in the data.frame
names(chicago)## [1] "city" "tmpd" "dptp" "date" "pm25tmean2" "pm10tmean2" "o3tmean2"
## [8] "no2tmean2"The functions are:
select: return a subset of the columns of a data frame
So here I can return the head of the data.frame using select and showing the head of the columns from city to dptp.
head(select(chicago, city:dptp))## city tmpd dptp
## 1 chic 31.5 31.500
## 2 chic 33.0 29.875
## 3 chic 33.0 27.375
## 4 chic 29.0 28.625
## 5 chic 32.0 28.875
## 6 chic 40.0 35.125You can also use the minus sign to show all columns except those in the range specified with select
head(select(chicago,- (city:dptp)))## date pm25tmean2 pm10tmean2 o3tmean2 no2tmean2
## 1 1987-01-01 NA 34.00000 4.250000 19.98810
## 2 1987-01-02 NA NA 3.304348 23.19099
## 3 1987-01-03 NA 34.16667 3.333333 23.81548
## 4 1987-01-04 NA 47.00000 4.375000 30.43452
## 5 1987-01-05 NA NA 4.750000 30.33333
## 6 1987-01-06 NA 48.00000 5.833333 25.77233To do this without dplyr takes a few extra steps:
i <- match("city", names(chicago))
j <- match("dptp", names(chicago))
head(chicago[, -(i:j)])## date pm25tmean2 pm10tmean2 o3tmean2 no2tmean2
## 1 1987-01-01 NA 34.00000 4.250000 19.98810
## 2 1987-01-02 NA NA 3.304348 23.19099
## 3 1987-01-03 NA 34.16667 3.333333 23.81548
## 4 1987-01-04 NA 47.00000 4.375000 30.43452
## 5 1987-01-05 NA NA 4.750000 30.33333
## 6 1987-01-06 NA 48.00000 5.833333 25.77233filter: extract a subset of rows from a data frame based on logical conditions
For example you might want to take all rows in the chicago dataset where PM2.5 is > 30
chic.f <- filter(chicago, pm25tmean2 > 30)
head(chic.f, 10)## city tmpd dptp date pm25tmean2 pm10tmean2 o3tmean2 no2tmean2
## 1 chic 23 21.9 1998-01-17 38.10 32.46154 3.180556 25.30000
## 2 chic 28 25.8 1998-01-23 33.95 38.69231 1.750000 29.37630
## 3 chic 55 51.3 1998-04-30 39.40 34.00000 10.786232 25.31310
## 4 chic 59 53.7 1998-05-01 35.40 28.50000 14.295125 31.42905
## 5 chic 57 52.0 1998-05-02 33.30 35.00000 20.662879 26.79861
## 6 chic 57 56.0 1998-05-07 32.10 34.50000 24.270422 33.99167
## 7 chic 75 65.8 1998-05-15 56.50 91.00000 38.573007 29.03261
## 8 chic 61 59.0 1998-06-09 33.80 26.00000 17.890810 25.49668
## 9 chic 73 60.3 1998-07-13 30.30 64.50000 37.018865 37.93056
## 10 chic 78 67.1 1998-07-14 41.40 75.00000 40.080902 32.59054You can add additional conditions:
chic.f <- filter(chicago, pm25tmean2 > 30 & tmpd > 80)
head(chic.f, 10)## city tmpd dptp date pm25tmean2 pm10tmean2 o3tmean2 no2tmean2
## 1 chic 81 71.2 1998-08-23 39.6000 59.0 45.86364 14.32639
## 2 chic 81 70.4 1998-09-06 31.5000 50.5 50.66250 20.31250
## 3 chic 82 72.2 2001-07-20 32.3000 58.5 33.00380 33.67500
## 4 chic 84 72.9 2001-08-01 43.7000 81.5 45.17736 27.44239
## 5 chic 85 72.6 2001-08-08 38.8375 70.0 37.98047 27.62743
## 6 chic 84 72.6 2001-08-09 38.2000 66.0 36.73245 26.46742
## 7 chic 82 67.4 2002-06-20 33.0000 80.5 47.42673 30.76703
## 8 chic 82 63.5 2002-06-23 42.5000 65.0 54.88043 30.03913
## 9 chic 81 70.4 2002-07-08 33.1000 64.0 45.34969 27.67857
## 10 chic 82 66.2 2002-07-18 38.8500 72.5 44.98045 26.06905arrange: reorder rows of a data frame
This will arrange the variables according to the date variable:
chicago <- arrange(chicago, date)
head(chicago)## city tmpd dptp date pm25tmean2 pm10tmean2 o3tmean2 no2tmean2
## 1 chic 31.5 31.500 1987-01-01 NA 34.00000 4.250000 19.98810
## 2 chic 33.0 29.875 1987-01-02 NA NA 3.304348 23.19099
## 3 chic 33.0 27.375 1987-01-03 NA 34.16667 3.333333 23.81548
## 4 chic 29.0 28.625 1987-01-04 NA 47.00000 4.375000 30.43452
## 5 chic 32.0 28.875 1987-01-05 NA NA 4.750000 30.33333
## 6 chic 40.0 35.125 1987-01-06 NA 48.00000 5.833333 25.77233tail(chicago)## city tmpd dptp date pm25tmean2 pm10tmean2 o3tmean2 no2tmean2
## 6935 chic 35 29.6 2005-12-26 8.40000 8.5 14.041667 16.81944
## 6936 chic 40 33.6 2005-12-27 23.56000 27.0 4.468750 23.50000
## 6937 chic 37 34.5 2005-12-28 17.75000 27.5 3.260417 19.28563
## 6938 chic 35 29.4 2005-12-29 7.45000 23.5 6.794837 19.97222
## 6939 chic 36 31.0 2005-12-30 15.05714 19.2 3.034420 22.80556
## 6940 chic 35 30.1 2005-12-31 15.00000 23.5 2.531250 13.25000This will arrange the variables in descending order:
chicago <- arrange(chicago, desc(date))
head(chicago)## city tmpd dptp date pm25tmean2 pm10tmean2 o3tmean2 no2tmean2
## 1 chic 35 30.1 2005-12-31 15.00000 23.5 2.531250 13.25000
## 2 chic 36 31.0 2005-12-30 15.05714 19.2 3.034420 22.80556
## 3 chic 35 29.4 2005-12-29 7.45000 23.5 6.794837 19.97222
## 4 chic 37 34.5 2005-12-28 17.75000 27.5 3.260417 19.28563
## 5 chic 40 33.6 2005-12-27 23.56000 27.0 4.468750 23.50000
## 6 chic 35 29.6 2005-12-26 8.40000 8.5 14.041667 16.81944rename: rename variables in a data frame
If you want to rename the pm25tmean2 you can rename that variable:
chicago <- rename(chicago, pm25 = pm25tmean2, dewpoint = dptp)
names(chicago)## [1] "city" "tmpd" "dewpoint" "date" "pm25" "pm10tmean2" "o3tmean2"
## [8] "no2tmean2"mutate: add new variables / columns or transform existing variables
create or transform existing variables. So this will create a column showing how much pm25 emissions deviate from the mean.
chicago <- mutate(chicago, pm25detrend = pm25-mean(pm25, na.rm=TRUE))
head(select(chicago, pm25, pm25detrend))## pm25 pm25detrend
## 1 15.00000 -1.230958
## 2 15.05714 -1.173815
## 3 7.45000 -8.780958
## 4 17.75000 1.519042
## 5 23.56000 7.329042
## 6 8.40000 -7.830958This creates a factor variable indicating whether the temperatures are hot or cold.
chicago <- mutate(chicago, tempcat = factor(1 * (tmpd > 80), labels = c("cold", "hot")))
hotcold <- group_by(chicago, tempcat)
hotcold## Source: local data frame [6,940 x 10]
## Groups: tempcat
##
## city tmpd dewpoint date pm25 pm10tmean2 o3tmean2 no2tmean2 pm25detrend tempcat
## 1 chic 35 30.1 2005-12-31 15.00000 23.5 2.531250 13.25000 -1.230958 cold
## 2 chic 36 31.0 2005-12-30 15.05714 19.2 3.034420 22.80556 -1.173815 cold
## 3 chic 35 29.4 2005-12-29 7.45000 23.5 6.794837 19.97222 -8.780958 cold
## 4 chic 37 34.5 2005-12-28 17.75000 27.5 3.260417 19.28563 1.519042 cold
## 5 chic 40 33.6 2005-12-27 23.56000 27.0 4.468750 23.50000 7.329042 cold
## 6 chic 35 29.6 2005-12-26 8.40000 8.5 14.041667 16.81944 -7.830958 cold
## 7 chic 35 32.1 2005-12-25 6.70000 8.0 14.354167 13.79167 -9.530958 cold
## 8 chic 37 35.2 2005-12-24 30.77143 25.2 1.770833 31.98611 14.540471 cold
## 9 chic 41 32.6 2005-12-23 32.90000 34.5 6.906250 29.08333 16.669042 cold
## 10 chic 22 23.3 2005-12-22 36.65000 42.5 5.385417 33.73026 20.419042 cold
## .. ... ... ... ... ... ... ... ... ... ...summarise / summarize: generate summary statistics of different variables in the data frame, possibly within strata
This gives mean pm25, maxo 3tmean2, and median no2tmean2 for both hot and cold days. There are NA values for pm25 so na.rm has to be added to pm25
summarize(hotcold, pm25 = mean(pm25, na.rm=TRUE), o3 = max(o3tmean2), no2 = median(no2tmean2))## Source: local data frame [3 x 4]
##
## tempcat pm25 o3 no2
## 1 cold 15.97807 66.587500 24.54924
## 2 hot 26.48118 62.969656 24.93870
## 3 NA 47.73750 9.416667 37.44444Might want a summary for every year. Using mutate we can add a year variable. This will show us mean pm25, max o3, and median no2 for each year in the dataset.
chicago <- mutate(chicago, year = as.POSIXlt(date)$year + 1900)
years <- group_by(chicago, year)
summarize(years, pm25=mean(pm25, na.rm = TRUE), o3 = max(o3tmean2), no2 = median(no2tmean2))## Source: local data frame [19 x 4]
##
## year pm25 o3 no2
## 1 1987 NaN 62.96966 23.49369
## 2 1988 NaN 61.67708 24.52296
## 3 1989 NaN 59.72727 26.14062
## 4 1990 NaN 52.22917 22.59583
## 5 1991 NaN 63.10417 21.38194
## 6 1992 NaN 50.82870 24.78921
## 7 1993 NaN 44.30093 25.76993
## 8 1994 NaN 52.17844 28.47500
## 9 1995 NaN 66.58750 27.26042
## 10 1996 NaN 58.39583 26.38715
## 11 1997 NaN 56.54167 25.48143
## 12 1998 18.26467 50.66250 24.58649
## 13 1999 18.49646 57.48864 24.66667
## 14 2000 16.93806 55.76103 23.46082
## 15 2001 16.92632 51.81984 25.06522
## 16 2002 15.27335 54.88043 22.73750
## 17 2003 15.23183 56.16608 24.62500
## 18 2004 14.62864 44.48240 23.39130
## 19 2005 16.18556 58.84126 22.62387Pipeline Operator %>%
dplyr has a function that allows you to chain multiple functions together and it allows you see what is happening in a readable way. It uses these symbols: %>%
This will mutate it by month, then take that output and group_by it with the month variable then take the output of that and summarize. This shows a summary of three pollutants grouped by the month of the year.
chicago %>% mutate(month = as.POSIXlt(date)$mon + 1) %>% group_by(month) %>% summarize(pm25 = mean(pm25, na.rm=TRUE), o3 = max(o3tmean2), no2 = median(no2tmean2))## Source: local data frame [12 x 4]
##
## month pm25 o3 no2
## 1 1 17.76996 28.22222 25.35417
## 2 2 20.37513 37.37500 26.78034
## 3 3 17.40818 39.05000 26.76984
## 4 4 13.85879 47.94907 25.03125
## 5 5 14.07420 52.75000 24.22222
## 6 6 15.86461 66.58750 25.01140
## 7 7 16.57087 59.54167 22.38442
## 8 8 16.93380 53.96701 22.98333
## 9 9 15.91279 57.48864 24.47917
## 10 10 14.23557 47.09275 24.15217
## 11 11 15.15794 29.45833 23.56537
## 12 12 17.52221 27.70833 24.45773