Exploratory Data Analysis: Concepts, Methods, and Practical R Implementation

Introduction

The session began with a brief personal introduction and a reminder that the focus would be on Exploratory Data Analysis (EDA) – the first and essential step before any inferential statistics or modeling.

What is Exploratory Data Analysis?

• Definition: An approach that summarizes the main characteristics of a dataset using numbers and graphs.
• Purpose:
• Build confidence in the data.
• Detect relationships between variables.
• Identify data entry errors, outliers, and violations of statistical assumptions (e.g., normality for ANOVA).
• Guide the choice of analytical tools and hypotheses.

Types of Variables

• Quantitative (numeric) – measurable on a scale (e.g., height, salary, years of education). These can be continuous or discrete.
• Qualitative (categorical) – place observations into groups (e.g., gender, job category, minority status).

Univariate vs. Multivariate EDA

Five‑Number Summary & Box‑Plot

The five‑number summary (minimum, Q1, median, Q3, maximum) provides a quick view of the distribution and is visualized by a box‑plot. Outliers are identified using the rule:

Lower bound = Q1 – 1.5·IQR
Upper bound = Q3 + 1.5·IQR

Values outside these bounds are flagged for further investigation.

Measures of Central Tendency

• Mean – sensitive to extreme values.
• Median – robust; unchanged by outliers.
• Trimmed mean – a compromise that removes a percentage of the most extreme observations.

Measures of Spread

• Range – simple max‑min difference; highly affected by outliers.
• Inter‑quartile range (IQR) – robust, focuses on the middle 50%.
• Variance & Standard Deviation – incorporate all observations; sensitive to extreme values.
• Coefficient of Variation – standard deviation expressed as a percentage of the mean.

Distribution Shape

• Symmetric – left and right tails mirror each other (often normal).
• Skewed right (positive) – long tail on the high‑value side.
• Skewed left (negative) – long tail on the low‑value side. Histograms and box‑plots reveal these patterns and help decide whether transformations or alternative models are needed.

Correlation

• Correlation coefficients range from –1 (perfect negative linear relationship) to +1 (perfect positive linear relationship); 0 indicates no linear relationship.
• Strong correlations (|r| > 0.7) suggest a linear link, but the researcher must still interpret the substantive meaning.
• Correlations should be examined within sub‑groups (e.g., managers only) because patterns can differ dramatically across categories.

Case Study: Employee Dataset

The participants explored a synthetic dataset containing: - Quantitative variables: years of education, current salary, beginning salary, time on the job, previous experience. - Qualitative variables: gender, job category (clerical, managerial, custodial), minority status.

Key analytical steps demonstrated: 1. Identify quantitative variables and compute five‑number summaries for education and salary. 2. Detect outliers (e.g., a negative value for years of education) and discuss possible data‑entry errors. 3. Compare groups using box‑plots and frequency tables to reveal gender imbalances in job categories and salary distributions. 4. Cross‑tabulation to test whether gender is associated with job type (e.g., all custodial positions were male). 5. Scatter plots to explore relationships such as: - Education years vs. current salary (weak/absent trend). - Beginning salary vs. current salary (strong positive correlation, r ≈ 0.88). - Previous experience vs. current salary (negative correlation for some groups, suggesting possible cohort effects). 6. Interpretation – the analysis highlighted potential discrimination, the limited explanatory power of education for certain job categories, and the importance of subgroup analysis.

Practical R Workflow

• Installation: install.packages("car") and other required libraries.
• Setting the Working Directory: Session → Set Working Directory → Choose Directory to point RStudio to the folder containing the CSV files and scripts.
• Loading Data: read.csv("employees.csv").
• Running Scripts: Execute line‑by‑line, checking for errors such as missing packages or incorrect file paths.
• Troubleshooting: Verify that the correct folder (not a hidden sub‑folder) is selected, reinstall missing packages, and consult console messages.

Why Spend Time on EDA?

The presenter emphasized that thorough EDA saves time later: it uncovers data quality issues, informs model selection, and provides a narrative foundation for any statistical report or thesis.

Communication of Results

• Use concise tables, bar charts, or pie charts to convey frequencies.
• Highlight outliers and explain whether they are errors or meaningful observations.
• Tailor the story to the audience—journalists may stress the gender disparity, while a technical report may focus on statistical significance.

Next Steps for Participants

1. Ensure all required R packages are installed.
2. Set the working directory correctly.
3. Run the provided script up to line 19 without errors.
4. Continue exploring the dataset tomorrow, focusing on multivariate visualizations and hypothesis testing.

The session concluded with reminders about the WhatsApp support group, the YouTube channel for additional tutorials, and a light‑hearted farewell.

Effective exploratory data analysis—combining numerical summaries, visualizations, and careful handling of outliers and missing values—lays the groundwork for reliable statistical modeling and clear communication of findings.