Back to the lesson·Lesson 5 of 8·Working the table
Selecting and filtering, and the warning that became an error
The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.
What this lesson covers
- Three ways to select, and when each is right
- Boolean masks
- The warning that became an error
- Adding and changing columns
- Sorting, and why it is not a ranking
- What comes next
Speaker notes
loc, iloc and boolean masks used deliberately — plus what SettingWithCopyWarning was warning about, why pandas 3 replaced it with ChainedAssignmentError, and what that changes for code you already have.Three ways to select, and when each is right — In Python
import pandas as pd muac = read_register(RAW / "muac-screening-artibonite-2024.v1.csv")Three ways to select, and when each is right
- By label, with
loc— Rows by index value, columns by name
Speaker notes
By label, withloc. Rows by index value, columns by name. This is the one to reach for by default, because it says what it means.- By label, with
Three ways to select, and when each is right — In Python
muac.loc[muac["commune"] == "Gonaives", ["child_id", "muac_mm", "outcome"]]Three ways to select, and when each is right
- By position, with
iloc— Rows and columns by integer position
Speaker notes
By position, withiloc. Rows and columns by integer position.- By position, with
Three ways to select, and when each is right — In Python
muac.iloc[0] # first row muac.iloc[:5, :3] # first five rows, first three columnsThree ways to select, and when each is right
- A single column — is a Series; a list of columns is a DataFrame:
Speaker notes
ilocis for looking, not for logic. Position depends on the sort order of the file, and a script that saysiloc[:, 3]breaks silently the day the export gains a column. Reserve it for inspection and for genuinely positional work. A single column is a Series; a list of columns is a DataFrame:Three ways to select, and when each is right — In Python
muac["muac_mm"] # Series muac[["muac_mm"]] # DataFrame with one column muac[["commune", "muac_mm"]] # DataFrame with twoSpeaker notes
The double bracket is not a stylistic choice. Some functions need a DataFrame, and a Series will fail at a distance from where the mistake was made.Boolean masks — In Python
severe = muac["muac_mm"] < 115 print(severe.sum()) # how many print(severe.mean()) # what share muac.loc[severe]Speaker notes
A mask is a Series ofTrue/Falsethe same length as the frame.Boolean masks
- Combining conditions
Speaker notes
.sum()and.mean()on a boolean mask are the count and the proportion. That is the fastest way to answer "how many" and "what share" without agroupby.Boolean masks — In Python
# Parentheses are mandatory: & binds tighter than < under_five = (muac["age_months"] < 60) & (muac["muac_mm"] < 125) # Use & | ~ — not and / or / not, which cannot operate elementwise either = (muac["muac_mm"] < 115) | (muac["oedema"] == True)Boolean masks
- Missing values are not False
Speaker notes
Writingandhere raisesValueError: The truth value of a Series is ambiguous. The message is opaque the first time and always means the same thing: you used a scalar operator on a vector. This is the trap that changes a caseload.Boolean masks — In Python
print(len(muac)) # 4218 print((muac["muac_mm"] < 125).sum()) # counts only measured children print(muac["muac_mm"].isna().sum()) # the ones excluded silentlyBoolean masks — In Python
measured = muac["muac_mm"].notna() gam = measured & (muac["muac_mm"] < 125) print(f"assessed: {measured.sum()}, GAM cases: {gam.sum()}") print(f"GAM rate: {gam.sum() / measured.sum():.3f}")Speaker notes
A comparison againstNAyieldsNA, andNAis not selected. So a filter quietly drops every unmeasured child — which is correct if you meant "children measured below 125 mm" and wrong if you meant "children not known to be above 125 mm". Be explicit about which you meant:Boolean masks
- The denominator is the point — Dividing by
len(muac)instead ofmeasured.sum()reports a lower prevalence for a… isin,between,query
Speaker notes
The denominator is the point. Dividing bylen(muac)instead ofmeasured.sum()reports a lower prevalence for a reason that has nothing to do with nutrition.- The denominator is the point — Dividing by
Boolean masks — In Python
north = muac["commune"].isin(["Gonaives", "Terre-Neuve", "Anse-Rouge"]) plausible = muac["muac_mm"].between(80, 220)Boolean masks — In Python
impossible = muac["muac_mm"].notna() & ~muac["muac_mm"].between(80, 220)Speaker notes
betweenis inclusive on both ends by default, and — the important part — it returnsFalseforNA. So~muac["muac_mm"].between(80, 220)counts a missing measurement as an impossible one. Those are different failures and only one of them is a data-entry error:Boolean masks — In Python
muac.query("age_months < 60 and muac_mm < 125")Speaker notes
queryreads well for long conditions and is worth knowing: It is slower and it hides typos in a string, so prefer masks in a script that must fail loudly.The warning that became an error — Example
SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer, col_indexer] = value insteadSpeaker notes
Older pandas produced a warning that generations of analysts learned to silence:The warning that became an error — In Python
muac[muac["commune"] == "Gonaives"]["muac_mm"] = 0Speaker notes
It was warning about chained assignment — selecting, then assigning to the selection:The warning that became an error
- What pandas 3 does instead
- The behaviour is now defined — A selection never writes back to its parent — not sometimes, never
- Chained assignment tells you it did nothing
Speaker notes
The first bracket produces a new object. Whether the assignment reached the original frame depended on the memory layout, which meant the same line could work on Monday and not on Tuesday. The warning existed because pandas could not tell you which had happened. Copy-on-Write is always on and cannot be disabled. Two things follow. The behaviour is now defined. A selection never writes back to its parent — not sometimes, never.SettingWithCopyWarningno longer exists;pd.errors.SettingWithCopyWarningraisesAttributeError. Chained assignment tells you it did nothing.The warning that became an error — In Python
import pandas as pd df = pd.DataFrame({"a": [1, 2, 3], "b": [10, 20, 30]}) df[df["a"] > 1]["b"] = 0The warning that became an error — Example
ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment. Such chained assignment never works ...The warning that became an error — In Python
import warnings warnings.simplefilter("error", pd.errors.ChainedAssignmentError)Speaker notes
ChainedAssignmentErroris aWarningsubclass, so by default the line prints and continues. In a script whose numbers matter, promote it:The warning that became an error
- The correct form, and what changed for existing code
Speaker notes
Now the script stops instead of reporting a figure computed from an edit that never landed.The warning that became an error — In Python
# One .loc, both axes, one operation. muac.loc[muac["commune"] == "Gonaives", "muac_mm"] = pd.NAThe warning that became an error
.copy()and when you still want it
Speaker notes
If you are maintaining code written for pandas 1 or 2, the migration risk runs in one direction only: a chained assignment that used to modify the frame now does not. The number changes and nothing crashes. Grep for]["and for.copy()calls added to silence the old warning — the copies are now unnecessary, and each one you remove is a real saving on a large frame. Under Copy-on-Write,df[mask]is already independent, so a defensive.copy()buys nothing. It is still worth writing when it documents intent:The warning that became an error — In Python
# This subset is going to be modified and is not a view of anything. gonaives = muac.loc[muac["commune"] == "Gonaives"].copy() gonaives["flag"] = gonaives["muac_mm"] < 115Speaker notes
Adding a column to a selection you did not copy works and is the common case; the.copy()above is a note to the reader, not a requirement.Adding and changing columns — In Python
summary = ( muac .loc[muac["muac_mm"].notna()] .assign( gam=lambda d: d["muac_mm"] < 125, sam=lambda d: d["muac_mm"] < 115, ) )Speaker notes
assignreturns a new frame and chains, which keeps a pipeline readable:Adding and changing columns — In Python
import numpy as np muac["band"] = np.select( [muac["muac_mm"] < 115, muac["muac_mm"] < 125], ["severe", "moderate"], default="normal", )Speaker notes
Thelambda d:matters: it refers to the frame at that point in the chain, sosamcan be defined against columnsassigncreated a line earlier. Referring tomuacdirectly inside the chain would use the pre-filter frame and misalign. For conditions with more than two branches,np.selectbeats nestedwhere:Adding and changing columns — In Python
muac.loc[muac["muac_mm"].isna(), "band"] = pd.NASpeaker notes
Conditions are evaluated in order and the first match wins, so< 115must come first. Note thatdefaultalso catches missing values, which is almost never what you want — set them back explicitly:Sorting, and why it is not a ranking — In Python
muac.sort_values(["commune", "screening_date"], ascending=[True, False])Sorting, and why it is not a ranking — In Python
muac.groupby("commune")["muac_mm"].mean().nsmallest(5)Speaker notes
Sorting arranges rows. It says nothing about whether the differences between adjacent rows are real — a point the Nutrition programme dashboard project makes at length, where nine of twelve communes have overlapping confidence intervals and the sort order is not information.nlargestis the readable form of "top N" and is faster than sorting the whole frame:What comes next
- You can select a subset and change it without wondering whether the change landed.
Speaker notes
You can select a subset and change it without wondering whether the change landed. The next lesson aggregates:groupby, named aggregation, and getting a numerator and a denominator out of the same operation so they cannot drift apart.