Pandas DataFrames — Fundamentals

TL;DR

A Pandas DataFrame is a 2-D table where each column is a NumPy array (plus a label). Columns can have different dtypes; rows share an index (default: integer RangeIndex). It’s the lingua franca of tabular ML work — feature engineering, dataset audits, label QA, ad-hoc analysis.

The mental model: a DataFrame is a dict of NumPy arrays, with extra machinery for label-based indexing, alignment, and group operations. Most of what makes Pandas fast is “pushing the work down to NumPy.” Most of what makes Pandas slow is operations that fall back to Python (apply with a Python function, iterrows()).

This guide covers the 80% of Pandas you use weekly: building DataFrames, selecting rows and columns, dtypes, missing values, and the unavoidable indexing rules. groupby and merge get their own guides (PY 205, PY 206).

In 2026, also know about Polars — a Rust-based DataFrame library that’s typically 5–30× faster than Pandas for the same operations and has a cleaner API. For new projects with large data, it’s worth considering. For now, Pandas is still what most ML codebases use.

The picture in your head

DataFrame:
                   user_id  age  country  purchased
        index 0:        1   34   "US"     True
        index 1:        2   28   "UK"     False
        index 2:        3   42   "US"     True

Each column is a Series:
    df["age"]  ->  Series([34, 28, 42], dtype=int64, index=[0, 1, 2])

A Series is a NumPy array + index + name.
A DataFrame is a dict of Series sharing an index.

The index is what makes Pandas different from a NumPy 2-D array. It gives every row a label (default 0, 1, 2, …) that you can join on, slice by, and align across DataFrames.

Building DataFrames

The four ways you’ll actually use:

import pandas as pd

# 1) From a dict of column arrays
df = pd.DataFrame({
    "user_id": [1, 2, 3],
    "age":     [34, 28, 42],
    "country": ["US", "UK", "US"],
})

# 2) From a list of dicts (records)
df = pd.DataFrame([
    {"user_id": 1, "age": 34, "country": "US"},
    {"user_id": 2, "age": 28, "country": "UK"},
])

# 3) From a NumPy array + column names
df = pd.DataFrame(np.zeros((3, 4), dtype=np.float32),
                   columns=["a", "b", "c", "d"])

# 4) From a file
df = pd.read_csv("data.csv")
df = pd.read_parquet("data.parquet")          # faster, smaller, typed
df = pd.read_json("data.jsonl", lines=True)

Use parquet, not CSV, for anything bigger than a debug file. Parquet is columnar, typed, compressed, and ~10× faster to load. CSV loses types (everything’s a string), is huge on disk, and parses slowly.

Selecting columns and rows

The four selection idioms, ranked by frequency of use:

# 1) One column -> Series
df["age"]                       # ✓ canonical
df.age                          # works, but breaks if name has a space or dot

# 2) Multiple columns -> DataFrame
df[["age", "country"]]

# 3) Boolean filtering on rows
df[df["age"] > 30]
df.query("age > 30 and country == 'US'")     # often cleaner

# 4) Label-based indexing with .loc
df.loc[5]                       # row with index label 5
df.loc[5, "age"]                # specific cell
df.loc[df["age"] > 30, "age"]   # boolean rows + specific column

# 5) Position-based indexing with .iloc
df.iloc[0]                      # first row, regardless of index label
df.iloc[0:5, 0:2]               # first 5 rows, first 2 cols

# DON'T mix label and position with the bare `[]`. It's ambiguous.

Always use .loc for label, .iloc for position. The bare [] on a DataFrame is column selection (or row mask if it’s boolean); the bare [] on a Series is positional. The asymmetry is a famous source of bugs.

dtypes — the cause of half your bugs

Pandas defaults to NumPy-backed dtypes. From version 2.0, a separate PyArrow backend is also available (often better, especially for strings and missing values).

The most common silent footgun: a column read from CSV that should be int64 ends up as object because of one stray “N/A” string. Cleanup:

df["age"] = pd.to_numeric(df["age"], errors="coerce")   # invalid -> NaN
df["age"] = df["age"].astype("Int64")                    # nullable int

Inspect dtypes early and often:

df.dtypes
df.info(memory_usage="deep")     # also shows true memory cost

Missing values — NaN, None, pd.NA

Numeric columns use float NaN for missing. Object columns use None or NaN. The new nullable dtypes use pd.NA for proper “missing” semantics across types.

df.isna()                        # boolean DataFrame, same shape
df.isna().sum()                  # per-column missing count
df.dropna()                      # rows with any NaN
df.dropna(subset=["age"])        # rows with NaN in 'age' specifically
df.fillna(0)                     # fill all NaN with 0
df.fillna({"age": df["age"].median(), "country": "unknown"})  # per-column

Check missing rates as the first move on any new dataset. A 60%-NaN column is rarely a feature; usually it’s a label leak or a join problem.

Vectorized operations — stay out of apply when you can

Like NumPy, Pandas is fast when you operate on whole columns and slow when you Python-loop. The same rule applies: stay in C.

# Slow — Python function per row
df["price_with_tax"] = df.apply(lambda row: row["price"] * 1.1, axis=1)

# Fast — vectorized arithmetic on the column
df["price_with_tax"] = df["price"] * 1.1

# Slow — Python conditional per element
df["bucket"] = df["age"].apply(lambda a: "adult" if a >= 18 else "minor")

# Fast — np.where on the array
df["bucket"] = np.where(df["age"] >= 18, "adult", "minor")

# Slow — string apply
df["first_word"] = df["text"].apply(lambda s: s.split()[0])

# Fast — string accessor
df["first_word"] = df["text"].str.split().str[0]

The .str. accessor (and .dt. for datetime, .cat. for category) gives vectorized string/datetime/category operations. Reach for them before apply.

Worked example — feature engineering on a sales dataset

import pandas as pd
import numpy as np

df = pd.read_parquet("sales.parquet")

# 1. Cast & clean
df["date"] = pd.to_datetime(df["date"])
df["price"] = df["price"].astype("float32")
df["category"] = df["category"].astype("category")

# 2. Derive new columns (all vectorized)
df["log_price"]    = np.log1p(df["price"])
df["dow"]          = df["date"].dt.dayofweek
df["is_weekend"]   = df["dow"].isin([5, 6])
df["price_bucket"] = pd.cut(df["price"], bins=[0, 10, 50, 200, np.inf],
                              labels=["low", "mid", "high", "premium"])

# 3. Filter
df_clean = df.query("price > 0 and quantity > 0 and not is_refund")

# 4. Sample for a quick training set (deterministic)
sample = df_clean.sample(n=100_000, random_state=42)

# 5. Save efficiently
sample.to_parquet("training.parquet", compression="zstd")

Notice what’s not there: no apply, no iterrows, no Python loops. Every transformation is a vectorized column operation. On a million rows this runs in seconds; written with apply it would take minutes.

The SettingWithCopyWarning — a real warning

sub = df[df["age"] > 30]
sub["adjusted_age"] = sub["age"] - 5     # SettingWithCopyWarning!

Pandas warns because sub might be a view or a copy of df, and the assignment might or might not propagate. Don’t ignore it. Either:

# Be explicit you want an independent copy
sub = df[df["age"] > 30].copy()
sub["adjusted_age"] = sub["age"] - 5

# Or do it directly on the original with .loc
df.loc[df["age"] > 30, "adjusted_age"] = df.loc[df["age"] > 30, "age"] - 5

The warning catches a real bug class. Fix it; don’t suppress it.

Common gotchas

• object dtype. A column of strings as object is much slower and uses much more memory than string or string[pyarrow].
• Mutable index. Setting an index doesn’t copy — the original DataFrame still has it as a column unless you drop it.
• Comparison with NaN. df["x"] == np.nan is always False (because NaN ≠ NaN). Use df["x"].isna().
• Chained assignment. df[df.x > 0]["y"] = 1 modifies a temporary, not df. Use df.loc[df.x > 0, "y"] = 1.
• Memory blow-up on merge with duplicate keys. A many-to-many join can quadratically explode. See PY 206.
• groupby keys with NaN are dropped by default. Pass dropna=False to keep them.
• apply(axis=1) is the slowest thing in Pandas. It’s a Python loop over rows. Vectorize, or use NumPy.

When Pandas is the wrong tool

• Data > RAM. Use Polars (lazy mode), DuckDB, or Dask.
• Need streaming computation. Polars or PyArrow Tables.
• All-numeric, no labels. Just use NumPy or PyTorch.
• Speed-critical inner loop. Polars is typically 5–30× faster for the same operation.

Where Pandas shows up in real ML codebases

• Dataset audits. Loading a parquet, checking dtypes, NaN rates, cardinality, label balance. The 30-minute “is this dataset usable?” check is always Pandas.
• Feature engineering offline. Before everything moves to a feature store, the prototypes are DataFrames.
• Eval result tables. Stack the per-run metric DataFrames, group by experiment config, sort by score.
• Label QA. Filter for high-confidence-but-wrong predictions to spot-check.
• Notebook exploration. Most ML EDA notebooks are 80% Pandas.

The defensive habit: when you start a Pandas script, write df.dtypes and df.isna().sum() as the first two lines after load. Most data-quality bugs jump out immediately.

Resources

• Pandas user guide — pandas.pydata.org — canonical, comprehensive.
• Wes McKinney — Python for Data Analysis (3rd ed.) — wesmckinney.com — by Pandas’s creator. Free online.
• Polars — pola.rs — the modern alternative. Faster, cleaner API.
• DuckDB — duckdb.org — SQL on DataFrames; great for joins on big data.
• Modern Pandas (Tom Augspurger) — tomaugspurger.net — series on idiomatic Pandas.