Pandas

Analyzing and transforming tabular data with pandas.

Quick Start

pip install pandas
python -c "import pandas as pd; print(pd.read_csv('data.csv').head())"

When to Use

• Cleaning and reshaping tabular datasets
• Aggregations, joins, and pivots
• Time series analysis and resampling
• ETL steps before machine learning

Best Practices

Data Loading

• Specify `dtype`, `parse_dates`, and `usecols` to avoid surprises
• Use `read_parquet`/`read_csv` with explicit schema
• Chunk large files with `chunksize` or use Dask/Polars for scale
• Validate shape and nulls right after loading

Cleaning

• Use vectorized ops; avoid row-wise loops
• Handle missing data deliberately (`fillna` vs `dropna`)
• Normalize types: datetime, categorical, numeric
• Keep an audit trail of transformations

Performance

• Prefer vectorized operations over `.apply`/`.iterrows`
• Use `groupby` aggregations instead of loops
• Convert object columns to categorical for speed
• Set indexes for joins; avoid merge on object columns

Dependencies

pip install pandas numpy
# scale: polars, dask, pyarrow

Examples

import pandas as pd

df = pd.read_csv("sales.csv", parse_dates=["date"], dtype={"qty": "int32"})
df.info()
print(df.isna().sum())

# Cleaning
df["price"] = df["price"].str.replace("$", "").astype(float)
df = df.dropna(subset=["order_id"])
df["date"] = pd.to_datetime(df["date"], errors="coerce")
df = df[df["qty"] > 0]

# Aggregation
by_region = (
    df.groupby("region", as_index=False)
      .agg(total=("amount", "sum"), orders=("order_id", "nunique"))
      .sort_values("total", ascending=False)
)
print(by_region.head())

# Time series resampling
daily = df.set_index("date")["amount"].resample("D").sum()
monthly = daily.resample("ME").mean()
print(monthly.head())

Step-by-Step

1. Load data with explicit dtypes and date parsing.

2. Profile: shape, columns, nulls, and unique counts.

3. Clean types and missing values deliberately.

4. Filter and transform with vectorized operations.

5. Aggregate with `groupby`; reshape with pivot/merge.

6. For time series, resample and window correctly.

7. Export with explicit schema (parquet preferred).

8. Verify outputs against the source invariants.

Validation

1. No unexpected nulls or type errors after cleaning

2. Aggregations match manually computed totals

3. Joins don't multiply rows unexpectedly

4. Date ranges and resampling are correct

5. Results reproducible with a fixed seed (if sampling)

Troubleshooting

• MemoryError: use dtypes, chunking, or Polars.
• Wrong joins: check key types and duplicates before merge.
• Slow loops: convert to vectorized or `groupby` operations.