Eduzan / Data Science
Statistical Analysis
In data science and statistics, understanding probability, distributions, hypothesis testing, correlation, regression, and statistical significance is essential for making informed decisions and interpreting data correctly.
Probability is the measure of the likelihood that an event will occur. It ranges from 0 (the event will not occur) to 1 (the event will certainly occur).
Types of Probability:
- Classical Probability: Based on equally likely outcomes (e.g., rolling a die).
- Empirical Probability: Based on observed data (e.g., the probability of rain based on historical data).
- Subjective Probability: Based on personal judgment or experience.
Rules of Probability:
- Addition Rule: P(A or B)=P(A)+P(B)−P(A and B)P(A \text{ or } B) = P(A) + P(B) – P(A \text{ and } B)P(A or B)=P(A)+P(B)−P(A and B)
- Multiplication Rule: P(A and B)=P(A)×P(B∣A)P(A \text{ and } B) = P(A) \times P(B|A)P(A and B)=P(A)×P(B∣A) for dependent events, or P(A)×P(B)P(A) \times P(B)P(A)×P(B) for independent events.
- Complementary Rule: P(not A)=1−P(A)P(\text{not } A) = 1 – P(A)P(not A)=1−P(A)
Distributions
A probability distribution describes how the values of a random variable are distributed. It tells us the likelihood of different outcomes.
Types of Distributions:
Discrete Distributions: Concerned with outcomes that are discrete (countable).
- Binomial Distribution: Models the number of successes in a fixed number of independent Bernoulli trials.
- Poisson Distribution: Models the number of events occurring in a fixed interval of time or space.
Continuous Distributions: Concerned with outcomes that are continuous (can take any value within a range).
- Normal Distribution: A symmetric, bell-shaped distribution where most of the data falls around the mean.
- Exponential Distribution: Models the time between events in a Poisson process.
- Uniform Distribution: All outcomes are equally likely within a given range.
Example in Python (Normal Distribution):
import numpy as np
import matplotlib.pyplot as plt
# Generate data from a normal distribution
data = np.random.normal(loc=0, scale=1, size=1000)
# Plotting the distribution
plt.hist(data, bins=30, density=True, alpha=0.6, color='g')
plt.title('Normal Distribution')
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.show()Example in R (Binomial Distribution):
# Generate data from a binomial distribution
data <- rbinom(1000, size=10, prob=0.5)
# Plotting the distribution
hist(data, breaks=10, col="lightblue", main="Binomial Distribution", xlab="Number of Successes")Example in R:
# Example dataset
scores <- c(70, 85, 78, 92, 88, 75, 60, 95, 83, 72)
hours_studied <- c(2, 4, 3, 5, 4.5, 3, 1.5, 6, 4, 2.5)
# Histogram
hist(scores, breaks=5, main="Histogram of Scores", xlab="Scores", col="blue", border="black")
# Box Plot
boxplot(scores, main="Box Plot of Scores", ylab="Scores")
# Scatter Plot
plot(hours_studied, scores, main="Scatter Plot of Hours Studied vs. Scores", xlab="Hours Studied", ylab="Scores", pch=19, col="red")Hypothesis testing is a statistical method used to make decisions about a population parameter based on sample data.
- Null Hypothesis (H₀): A statement that there is no effect or no difference. It is assumed to be true until evidence suggests otherwise.
- Alternative Hypothesis (H₁ or Ha): A statement that there is an effect or a difference.
Steps in Hypothesis Testing:
- State the Hypotheses: Define the null and alternative hypotheses.
- Choose the Significance Level (α): Common choices are 0.05, 0.01, or 0.10.
- Select the Appropriate Test: Based on the data type and distribution (e.g., t-test, chi-square test).
- Calculate the Test Statistic: Use sample data to calculate a value (e.g., t-value, z-value).
- Determine the P-Value: The probability of observing the data if the null hypothesis is true.
- Make a Decision: Compare the p-value to the significance level to accept or reject the null hypothesis.
Common Hypothesis Tests:
- t-test: Compares the means of two groups (independent or paired).
- ANOVA (Analysis of Variance): Compares the means of three or more groups.
- Chi-Square Test: Tests for independence between categorical variables.
- Z-Test: Used when the sample size is large, and the population variance is known.
Example in Python (t-test):
from scipy import stats
# Example data
group1 = [2.3, 1.9, 2.5, 2.1, 2.7]
group2 = [1.7, 1.6, 1.8, 2.0, 1.9]
# Perform t-test
t_statistic, p_value = stats.ttest_ind(group1, group2)
print("t-statistic:", t_statistic)
print("p-value:", p_value)Example in R (Chi-Square Test):
# Example data
observed <- c(20, 30, 50)
expected <- c(25, 25, 50)
# Perform Chi-Square Test
chisq.test(observed, p=expected/sum(expected))Correlation
Correlation measures the strength and direction of the relationship between two variables.
- Pearson Correlation: Measures the linear relationship between two continuous variables.
- Range: -1 to 1, where -1 indicates a perfect negative linear relationship, 0 indicates no linear relationship, and 1 indicates a perfect positive linear relationship.
- Spearman Rank Correlation: Measures the monotonic relationship between two variables, useful for non-linear data.
Example in Python:
from scipy import stats
# Example data
group1 = [2.3, 1.9, 2.5, 2.1, 2.7]
group2 = [1.7, 1.6, 1.8, 2.0, 1.9]
# Perform t-test
t_statistic, p_value = stats.ttest_ind(group1, group2)
print("t-statistic:", t_statistic)
print("p-value:", p_value)# Example data observed <- c(20, 30, 50) expected <- c(25, 25, 50) # Perform Chi-Square Test chisq.test(observed, p=expected/sum(expected))Example in R
import pandas as pd
# Example data
df = pd.DataFrame({
'Variable1': [1, 2, 3, 4, 5],
'Variable2': [2, 4, 5, 4, 5]
})
# Pearson correlation
correlation = df['Variable1'].corr(df['Variable2'])
print("Pearson Correlation:", correlation)Regression analysis estimates the relationship between a dependent variable and one or more independent variables.
- Linear Regression: Models the relationship between two variables by fitting a linear equation.
- Formula: Y=β0+β1X+ϵY = \beta_0 + \beta_1X + \epsilonY=β0+β1X+ϵ, where YYY is the dependent variable, XXX is the independent variable, β0\beta_0β0 is the intercept, β1\beta_1β1 is the slope, and ϵ\epsilonϵ is the error term.
Example in Python (Linear Regression):
import pandas as pd
# Example data
df = pd.DataFrame({
'Variable1': [1, 2, 3, 4, 5],
'Variable2': [2, 4, 5, 4, 5]
})
# Pearson correlation
correlation = df['Variable1'].corr(df['Variable2'])
print("Pearson Correlation:", correlation)import pandas as pd
# Example data
df = pd.DataFrame({
'Variable1': [1, 2, 3, 4, 5],
'Variable2': [2, 4, 5, 4, 5]
})
# Pearson correlation
correlation = df['Variable1'].corr(df['Variable2'])
print("Pearson Correlation:", correlation)Statistical significance indicates whether the observed effect in the data is likely due to chance or if it reflects a true relationship in the population.
- P-Value: The probability of obtaining the observed results, or more extreme, if the null hypothesis is true.
- Interpretation:
- p-value < α: Reject the null hypothesis (statistically significant).
- p-value > α: Fail to reject the null hypothesis (not statistically significant).
- Interpretation:
- Confidence Interval (CI): A range of values within which the true population parameter is expected to fall with a certain level of confidence (e.g., 95%).
- Effect Size: A measure of the strength or magnitude of an observed effect. It’s important to consider both statistical significance and effect size when interpreting results.
Example in Python (Calculating P-Value):
# Using the t-test example from above
if p_value < 0.05:
print("Reject the null hypothesis, statistically significant.")
else:
print("Fail to reject the null hypothesis, not statistically significant.")# Using the Chi-Square test example from above
if(chisq.test(observed, p=expected/sum(expected))$p.value < 0.05) {
print("Reject the null hypothesis, statistically significant.")
} else {
print("Fail to reject the null hypothesis, not statistically significant.")End of lesson.