Selecting and filtering, and the warning that became an error

Three ways to select, and when each is right

import pandas as pd

muac = read_register(RAW / "muac-screening-artibonite-2024.v1.csv")

By label, with loc. Rows by index value, columns by name. This is the one to reach for by default, because it says what it means.

muac.loc[muac["commune"] == "Gonaives", ["child_id", "muac_mm", "outcome"]]

By position, with iloc. Rows and columns by integer position.

muac.iloc[0]           # first row
muac.iloc[:5, :3]      # first five rows, first three columns

iloc is for looking, not for logic. Position depends on the sort order of the file, and a script that says iloc[:, 3] breaks silently the day the export gains a column. Reserve it for inspection and for genuinely positional work.

A single column is a Series; a list of columns is a DataFrame:

muac["muac_mm"]              # Series
muac[["muac_mm"]]            # DataFrame with one column
muac[["commune", "muac_mm"]] # DataFrame with two

The double bracket is not a stylistic choice. Some functions need a DataFrame, and a Series will fail at a distance from where the mistake was made.

Boolean masks

A mask is a Series of True/False the same length as the frame.

severe = muac["muac_mm"] < 115
print(severe.sum())              # how many
print(severe.mean())             # what share
muac.loc[severe]

.sum() and .mean() on a boolean mask are the count and the proportion. That is the fastest way to answer “how many” and “what share” without a groupby.

Combining conditions

# Parentheses are mandatory: & binds tighter than <
under_five = (muac["age_months"] < 60) & (muac["muac_mm"] < 125)

# Use & | ~ — not and / or / not, which cannot operate elementwise
either = (muac["muac_mm"] < 115) | (muac["oedema"] == True)

Writing and here raises ValueError: The truth value of a Series is ambiguous. The message is opaque the first time and always means the same thing: you used a scalar operator on a vector.

Missing values are not False

This is the trap that changes a caseload.

print(len(muac))                                   # 4218
print((muac["muac_mm"] < 125).sum())               # counts only measured children
print(muac["muac_mm"].isna().sum())                # the ones excluded silently

A comparison against NA yields NA, and NA is not selected. So a filter quietly drops every unmeasured child — which is correct if you meant “children measured below 125 mm” and wrong if you meant “children not known to be above 125 mm”.

Be explicit about which you meant:

measured = muac["muac_mm"].notna()
gam = measured & (muac["muac_mm"] < 125)

print(f"assessed: {measured.sum()}, GAM cases: {gam.sum()}")
print(f"GAM rate: {gam.sum() / measured.sum():.3f}")

The denominator is the point. Dividing by len(muac) instead of measured.sum() reports a lower prevalence for a reason that has nothing to do with nutrition.

isin, between, query

north = muac["commune"].isin(["Gonaives", "Terre-Neuve", "Anse-Rouge"])
plausible = muac["muac_mm"].between(80, 220)

between is inclusive on both ends by default, and — the important part — it returns False for NA. So ~muac["muac_mm"].between(80, 220) counts a missing measurement as an impossible one. Those are different failures and only one of them is a data-entry error:

impossible = muac["muac_mm"].notna() & ~muac["muac_mm"].between(80, 220)

query reads well for long conditions and is worth knowing:

muac.query("age_months < 60 and muac_mm < 125")

It is slower and it hides typos in a string, so prefer masks in a script that must fail loudly.

The warning that became an error

Older pandas produced a warning that generations of analysts learned to silence:

SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer, col_indexer] = value instead

It was warning about chained assignment — selecting, then assigning to the selection:

muac[muac["commune"] == "Gonaives"]["muac_mm"] = 0

The first bracket produces a new object. Whether the assignment reached the original frame depended on the memory layout, which meant the same line could work on Monday and not on Tuesday. The warning existed because pandas could not tell you which had happened.

What pandas 3 does instead

Copy-on-Write is always on and cannot be disabled. Two things follow.

The behaviour is now defined. A selection never writes back to its parent — not sometimes, never. SettingWithCopyWarning no longer exists; pd.errors.SettingWithCopyWarning raises AttributeError.

Chained assignment tells you it did nothing.

import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3], "b": [10, 20, 30]})
df[df["a"] > 1]["b"] = 0

ChainedAssignmentError: A value is being set on a copy of a
DataFrame or Series through chained assignment. Such chained
assignment never works ...

ChainedAssignmentError is a Warning subclass, so by default the line prints and continues. In a script whose numbers matter, promote it:

import warnings

warnings.simplefilter("error", pd.errors.ChainedAssignmentError)

Now the script stops instead of reporting a figure computed from an edit that never landed.

The correct form, and what changed for existing code

# One .loc, both axes, one operation.
muac.loc[muac["commune"] == "Gonaives", "muac_mm"] = pd.NA

If you are maintaining code written for pandas 1 or 2, the migration risk runs in one direction only: a chained assignment that used to modify the frame now does not. The number changes and nothing crashes. Grep for ][" and for .copy() calls added to silence the old warning — the copies are now unnecessary, and each one you remove is a real saving on a large frame.

.copy() and when you still want it

Under Copy-on-Write, df[mask] is already independent, so a defensive .copy() buys nothing. It is still worth writing when it documents intent:

# This subset is going to be modified and is not a view of anything.
gonaives = muac.loc[muac["commune"] == "Gonaives"].copy()
gonaives["flag"] = gonaives["muac_mm"] < 115

Adding a column to a selection you did not copy works and is the common case; the .copy() above is a note to the reader, not a requirement.

Adding and changing columns

assign returns a new frame and chains, which keeps a pipeline readable:

summary = (
    muac
    .loc[muac["muac_mm"].notna()]
    .assign(
        gam=lambda d: d["muac_mm"] < 125,
        sam=lambda d: d["muac_mm"] < 115,
    )
)

The lambda d: matters: it refers to the frame at that point in the chain, so sam can be defined against columns assign created a line earlier. Referring to muac directly inside the chain would use the pre-filter frame and misalign.

For conditions with more than two branches, np.select beats nested where:

import numpy as np

muac["band"] = np.select(
    [muac["muac_mm"] < 115, muac["muac_mm"] < 125],
    ["severe", "moderate"],
    default="normal",
)

Conditions are evaluated in order and the first match wins, so < 115 must come first. Note that default also catches missing values, which is almost never what you want — set them back explicitly:

muac.loc[muac["muac_mm"].isna(), "band"] = pd.NA

Sorting, and why it is not a ranking

muac.sort_values(["commune", "screening_date"], ascending=[True, False])

Sorting arranges rows. It says nothing about whether the differences between adjacent rows are real — a point the Nutrition programme dashboard project makes at length, where nine of twelve communes have overlapping confidence intervals and the sort order is not information.

nlargest is the readable form of “top N” and is faster than sorting the whole frame:

muac.groupby("commune")["muac_mm"].mean().nsmallest(5)

What comes next

You can select a subset and change it without wondering whether the change landed. The next lesson aggregates: groupby, named aggregation, and getting a numerator and a denominator out of the same operation so they cannot drift apart.