# Data import

## Table data

• most of the available data is structured in a table representation
• rows: observations, cases, subjects
• columns: variables, features
• table data can be stored
• in plain text: .txt, .csv, .tsv, …
• in a binary format: Excel, Stata, SPSS, …
• in a relational database: SQL, …

## Text table data

• central function is read.table() to read table data from the hard drive
• questions to answer before importing table data:
• has the data a header line?
• how are values separated (e.g. "," or ";")
• what’s the encoding (e.g. UTF-8 or latin-1)
• use the argument stringsAsFactors = FALSE

## Binary table data

• there exists various packages for importing binary table data
• haven package for Stata, SAS and SPSS files
• readxl package for Excel files
• usually one can always export the data to text format and then import it using read.table

## Other data formats

• Web APIs often represent data in json or xml (text-)format
• packages XML and jsonlite
• R uses the binary .RData or .rds format to save objects persistently
• Spatial data is often stored in a GIS file format (shapefile)
• packages maptools

# Data frames

## Introduction

• when importing table data, the natural R representation is a data.frame
• two-dimensional data structure with rows and columns
• each column is a vector of some data-type (e.g. numeric, logical, character, …)

## Example: creation

myDF <- data.frame(1:10, rep(TRUE, 10), letters[1:10])
myDF
   X1.10 rep.TRUE..10. letters.1.10.
1      1          TRUE             a
2      2          TRUE             b
3      3          TRUE             c
4      4          TRUE             d
5      5          TRUE             e
6      6          TRUE             f
7      7          TRUE             g
8      8          TRUE             h
9      9          TRUE             i
10    10          TRUE             j

## Example: data import

titanic <- read.csv("./www/titanic.csv", stringsAsFactors = FALSE)
sep = ",", stringsAsFactors = FALSE)
head(titanic)
  X.1 X pclass survived                                            name
1   1 1      1        1                   Allen, Miss. Elisabeth Walton
2   2 2      1        1                  Allison, Master. Hudson Trevor
3   3 3      1        0                    Allison, Miss. Helen Loraine
4   4 4      1        0            Allison, Mr. Hudson Joshua Creighton
5   5 5      1        0 Allison, Mrs. Hudson J C (Bessie Waldo Daniels)
6   6 6      1        1                             Anderson, Mr. Harry
sex     age embarked
1 female 29.0000        S
2   male  0.9167        S
3 female  2.0000        S
4   male 30.0000        S
5 female 25.0000        S
6   male 48.0000        S

## Example: summary function

summary(titanic)
      X.1             X            pclass         survived
Min.   :   1   Min.   :   1   Min.   :1.000   Min.   :0.000
1st Qu.: 328   1st Qu.: 328   1st Qu.:2.000   1st Qu.:0.000
Median : 655   Median : 655   Median :3.000   Median :0.000
Mean   : 655   Mean   : 655   Mean   :2.295   Mean   :0.382
3rd Qu.: 982   3rd Qu.: 982   3rd Qu.:3.000   3rd Qu.:1.000
Max.   :1309   Max.   :1309   Max.   :3.000   Max.   :1.000

name               sex                 age
Length:1309        Length:1309        Min.   : 0.1667
Class :character   Class :character   1st Qu.:21.0000
Mode  :character   Mode  :character   Median :28.0000
Mean   :29.8811
3rd Qu.:39.0000
Max.   :80.0000
NA's   :263
embarked
Length:1309
Class :character
Mode  :character



## Length

• use the functions nrows(), ncols()
• length() also gives the number of columns
nrow(titanic)
[1] 1309
ncol(titanic)
[1] 8
length(titanic)
[1] 8

## Names

• function colnames() or names() for the column names
• function rownames() for the row names (used less frequently)
colnames(titanic)
[1] "X.1"      "X"        "pclass"   "survived" "name"     "sex"
[7] "age"      "embarked"

# Extraction and adding of variables

## Column extraction

• each column of a data.frame is a vector of some data type
• columns can be extracted using the $ sign class <- titanic$pclass
class
   [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[35] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[69] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[103] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[137] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[171] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[205] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[239] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[273] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[307] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
[341] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
[375] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
[409] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
[443] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
[477] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
[511] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
[545] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
[579] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3
[613] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[647] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[681] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[715] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[749] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[783] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[817] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[851] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[885] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[919] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[953] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[987] 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[ reached getOption("max.print") -- omitted 309 entries ]

## Column creation

titanic$isOld <- titanic$age > 60
head(titanic$age) [1] 12 12 12 12 12 12 ## Deletion of columns titanic$age <- NULL
head(titanic)
  X.1 X pclass survived                                            name
1   1 1      1        1                   Allen, Miss. Elisabeth Walton
2   2 2      1        1                  Allison, Master. Hudson Trevor
3   3 3      1        0                    Allison, Miss. Helen Loraine
4   4 4      1        0            Allison, Mr. Hudson Joshua Creighton
5   5 5      1        0 Allison, Mrs. Hudson J C (Bessie Waldo Daniels)
6   6 6      1        1                             Anderson, Mr. Harry
sex embarked isOld
1 female        S FALSE
2   male        S FALSE
3 female        S FALSE
4   male        S FALSE
5 female        S FALSE
6   male        S FALSE

# Frequency and contingency tables

## Introduction

• used to represent the distribution of one or two categorical variables
• total frequencies
• relative frequencies (row, column, or absolute percentage)

table(titanic$pclass)  1 2 3 323 277 709  table(titanic$sex, titanic$pclass)  1 2 3 female 144 106 216 male 179 171 493 ## Example relative frequencies, row percentage tab <- table(titanic$sex, titanic$pclass) tab  1 2 3 female 144 106 216 male 179 171 493 prop.table(tab, margin = 1)  1 2 3 female 0.3090129 0.2274678 0.4635193 male 0.2123369 0.2028470 0.5848161 ## Example relative frequencies, column percentage tab <- table(titanic$sex, titanic$pclass) tab  1 2 3 female 144 106 216 male 179 171 493 prop.table(tab, margin = 2)  1 2 3 female 0.4458204 0.3826715 0.3046544 male 0.5541796 0.6173285 0.6953456 ## Example relative frequencies, absolute percentage tab <- table(titanic$sex, titanic\$pclass)
tab

1   2   3
female 144 106 216
male   179 171 493
prop.table(tab, margin = NULL)

1          2          3
female 0.11000764 0.08097785 0.16501146
male   0.13674561 0.13063407 0.37662338