cassionData Analysis

Back to the lessonLesson 2 of 8The project on disk

Git does not forget, which is the point and the danger

The same deck as the downloads, rendered as a page. Start the slideshow to present it full screen — arrow keys or a click advance one slide, Escape leaves.

Slides · PDFSlides · PowerPoint

  1. Slide 1 / 23

    What this lesson covers

    • The commit that cannot be taken back
    • The .gitignore you write before the first commit
    • What "personal data" means here, precisely
    • The check that runs before the commit does
    • What to do if it has already happened
    • Commits on analysis work
    • What a repository of programme analysis should contain
    • Report it whole
    • What comes next
    Speaker notes
    A repository remembers every version of every file it has ever held, which is exactly what you want for code and exactly what you cannot allow for a beneficiary list. Deleting the file in a later commit does not remove it.

  2. Slide 2 / 23

    The commit that cannot be taken back — Shell

    git add data/raw/beneficiaries.csv
git commit -m "add survey data"
# ... realised three days later ...
git rm data/raw/beneficiaries.csv
git commit -m "remove data"

  3. Slide 3 / 23

    The commit that cannot be taken back

    • The file is still in the repository — It is in the history, it is in every clone, it is on the remote, and it is in the…
    • On a repository holding protection or health data, that is a disclosure — Not a mistake to fix later — a disclosure,…
    • Which is why the rule is preventive rather than corrective — You cannot un-commit personal data; you can only stop it…
    Speaker notes
    The file is still in the repository. It is in the history, it is in every clone, it is on the remote, and it is in the local copy of everyone who pulled. The second commit removed it from the current state and from nothing else. On a repository holding protection or health data, that is a disclosure. Not a mistake to fix later — a disclosure, with the same obligations as any other. Which is why the rule is preventive rather than corrective. You cannot un-commit personal data; you can only stop it entering.

  4. Slide 4 / 23

    The .gitignore you write before the first commit — Example (cont.)

    # Raw data. Nothing under here is ever committed.
data/raw/*
!data/raw/.gitkeep
!data/raw/CHECKSUMS

# Derived data, regenerated by the pipeline
outputs/

# Credentials, in every form they arrive in
.env
*.pem
*_secret*
config.local.*

# Editor and language noise
.Rhistory

  5. Slide 5 / 23

    The .gitignore you write before the first commit — Example (cont.)

    .RData
__pycache__/
*.pyc
.DS_Store
.ipynb_checkpoints/

  6. Slide 6 / 23

    The .gitignore you write before the first commit

    • Write it first, commit it first — A .gitignore added after the first commit is a .gitignore that arrived too late…
    • The negations matter — data/raw/* excludes everything, then !data/raw/CHECKSUMS lets back exactly the file that…
    Speaker notes
    Write it first, commit it first. A .gitignore added after the first commit is a .gitignore that arrived too late for the thing it was meant to catch. The negations matter. data/raw/* excludes everything, then !data/raw/CHECKSUMS lets back exactly the file that proves which export was used — provenance without content.

  7. Slide 7 / 23

    What "personal data" means here, precisely

    Looks anonymousIs not, because
    A case IDIt joins to a case management system that has the name
    GPS coordinates of a householdIt is an address
    Date of birth plus commune plus sexThree fields identify most people in a small commune
    A photograph of a formEvery field on it
    A free-text "notes" columnIt contains whatever the caseworker typed
    Speaker notes
    It is broader than a name column, and this sector's datasets make the point.

  8. Slide 8 / 23

    What "personal data" means here, precisely

    • The protection course established the arithmetic — a four-way disaggregation of an anonymous dataset produced nineteen…
    • So the test is not "does it have a name column" — It is whether any combination of what the file holds could pick one…
    Speaker notes
    The protection course established the arithmetic: a four-way disaggregation of an anonymous dataset produced nineteen cells of one, and a cell of one is an identification. So the test is not "does it have a name column". It is whether any combination of what the file holds could pick one person out — and in a small commune it usually can.

  9. Slide 9 / 23

    The check that runs before the commit does — Shell

    # .git/hooks/pre-commit — or the pre-commit framework, or a CI step
if git diff --cached --name-only | grep -qE '^data/raw/'; then
  echo "Refusing to commit anything under data/raw/" >&2
  exit 1
fi

  10. Slide 10 / 23

    The check that runs before the commit does — In R

    # The same check as an R function, run by whatever CI you have.

  11. Slide 11 / 23

    The check that runs before the commit does

    • A hook that refuses is worth more than a policy that reminds — The failure mode is someone in a hurry at 6pm, and a…
    • Add a scan for the shapes secrets take — because a token pasted into a script is the other common case
    Speaker notes
    A hook that refuses is worth more than a policy that reminds. The failure mode is someone in a hurry at 6pm, and a reminder does not survive that. Add a scan for the shapes secrets take, because a token pasted into a script is the other common case. gitleaks and detect-secrets both do this in a CI step.

  12. Slide 12 / 23

    What to do if it has already happened

    • One: report it — Your organisation has a data-incident procedure
    • Two: rotate anything that was a credential — A committed token is compromised permanently; deleting it changes nothing
    • Three: rewrite the history — with git filter-repo or BFG, and force-push
    • Four: assume it was fetched — On a public repository, assume the file was retrieved and mirrored, and treat the…
    Speaker notes
    The order matters and the first step is not technical. One: report it. Your organisation has a data-incident procedure. This is one, and whether the repository was public decides how urgent rather than whether. Two: rotate anything that was a credential. A committed token is compromised permanently; deleting it changes nothing. Three: rewrite the history, with git filter-repo or BFG, and force-push. Every existing clone must be deleted and re-cloned — a clone nobody re-cloned still has the file. Four: assume it was fetched. On a public repository, assume the file was retrieved and mirrored, and treat the disclosure as complete.

  13. Slide 13 / 23

    What to do if it has already happened — Shell

    git filter-repo --path data/raw/beneficiaries.csv --invert-paths

  14. Slide 14 / 23

    What to do if it has already happened

    • Step three is the only one that looks like a fix and it is the least important
    Speaker notes
    Step three is the only one that looks like a fix and it is the least important.

  15. Slide 15 / 23

    Commits on analysis work

    • Commit the code and the output together when the output is committed — A figure and the script that made it belong in…
    • Write the why in the message — fix commune names is what the diff already shows
    Speaker notes
    The habits differ from software work in two ways worth naming. Commit the code and the output together when the output is committed. A figure and the script that made it belong in one commit, so a reviewer can see whether the chart matches the change. Write the why in the message. fix commune names is what the diff already shows. "Normalise four spellings of Nord-Ouest; ungrouped, the worst district splits into four fragments and none of them looks alarming" is the thing a successor needs and cannot recover.

  16. Slide 16 / 23

    Commits on analysis work — Shell

    git commit -m "Group four spellings of Nord-Ouest before computing coverage

Ungrouped, the worst district splits into fragments of 727, 33, 22 and 20
households and none of them looks alarming. The 20-household fragment shows
0% open defecation, which a district table would report as a solved problem."

  17. Slide 17 / 23

    Commits on analysis work

    • Branch per analysis question, not per day — A branch that answers "does coverage differ by district" can be reviewed…
    Speaker notes
    Branch per analysis question, not per day. A branch that answers "does coverage differ by district" can be reviewed and merged; a branch called work cannot.

  18. Slide 18 / 23

    What a repository of programme analysis should contain — Example

    project/
  .gitignore          committed first
  README.md           what it produces, how to run it, what to know
  uv.lock             the environment, pinned
  src/                every transformation
  tests/              the checks that fail loudly
  data/
    raw/              .gitkeep and CHECKSUMS only
    reference/        committed: thresholds, admin codes, lookup tables
  outputs/            ignored

  19. Slide 19 / 23

    What a repository of programme analysis should contain

    • Reference tables are committed and raw data is not — and the line between them is whether the file describes people
    Speaker notes
    Reference tables are committed and raw data is not, and the line between them is whether the file describes people. WHO growth standards, IPC thresholds and admin boundary codes are reference; a household list is not.

  20. Slide 20 / 23

    Report it whole — Example

    Data handling in this analysis

  No raw data is committed. data/raw/ is excluded by .gitignore, with only a
  CHECKSUMS file tracked so the exports used can be identified.

  A pre-commit hook refuses any staged file under data/raw/ and a CI step
  scans for credential patterns.

  Reference tables (WHO growth standards, IPC thresholds, admin codes) are
  committed; they describe no individual.

  The repository is private and access is limited to the M&E team. Raw
  exports are held in [the organisation's storage], not here.

  21. Slide 21 / 23

    Report it whole

    • The last line is the one that gets forgotten and it is the one an audit asks for — Saying where the data is matters…
    Speaker notes
    The last line is the one that gets forgotten and it is the one an audit asks for. Saying where the data is matters as much as saying where it is not.

  22. Slide 22 / 23

    What comes next

    • Code in version control still does not reproduce a number if the environment differs.
    Speaker notes
    Code in version control still does not reproduce a number if the environment differs. The next lesson pins it, and finds the four things that make a rerun differ when nothing in the repository changed at all.

  23. Slide 23 / 23

    Where this goes next

    Read the full lesson, with runnable code Back to the lesson