Skip to content
TESA Open Knowledge

Dataset engineering: class balance, windows and the train/val/test split

Module 5 — Training and deploying to several targets · Slides: slides.md · Module overview · Course page

Start building our own model from what always comes before training — the dataset. Learn the four-step preparation pipeline (capture, label, window, split), why class balance matters, the stratified train/val/test split, and the iron no-leakage rule, running the whole pipeline on synthetic data first.

Objectives

Section titled “Objectives”

By the end of this lesson, you will:

  1. Order the four-step preparation pipeline, capture → label → window → split, and say which steps run on the board and which run on the PC.
  2. Compute the class share p_c = n_c / N, and use an example to explain why an unbalanced dataset gives a misleading accuracy on paper.
  3. Compute the per-class window counts of a stratified train, val and test split with ⌊0.15·n_c⌋, and state the job of each set.
  4. Identify which kind of normalisation leaks data, and fix it with the rule “fit on train, apply to every set”.

Before you start

Section titled “Before you start”

You’ve been through module 4, understand the sliding window WIN and HOP from lesson 4.5, and have already recorded a CSV with the DAQ logger in lesson 2.2. Have a PC with Python 3 and numpy ready, and download dataset_tools.py.

See it work first

Section titled “See it work first”

Run the whole pipeline before understanding it: python dataset_tools.py --synthesize --out data/gestures.csv, then python dataset_tools.py --out data/gestures.csv. You’ll get samples (3600, 6) windows (143, 50, 6) class counts [48 48 47]. Ask yourself: what do the last three numbers tell you?

Concepts

Section titled “Concepts”

Our dataset is a plain CSV — one line per sample, starting with a label, followed by six values, ax,ay,az,gx,gy,gz, from sensors.bmi270.motion(). The label is the answer a human attached. We use the same three classes as the board’s Motion model (idle, circle, shaking), so we can later compare our own model against the ready-made one. A raw CSV still can’t be trained on — it must pass through four steps: capture and label on the board with MicroPython, then window and split on the PC with dataset_tools.py, which has five functions: load_csv, make_windows, normalize, split and synthesize. Synthetic data lets the pipeline run before we have real data.

Class balance: if 95% of the data is idle, a model that always answers idle is 95% accurate on paper, but useless in practice. We measure it with $p_c = n_c/N$, and it’s balanced when $p_c \approx 1/K$ (about 0.33 for three classes). The most direct fix is to collect it balanced in the first place. Windowing uses WIN = 50 (one second at 50 Hz) and HOP = 25. A window’s label is whichever label dominates inside it, and it must be sliced exactly the way the board feeds the model in deployment.

Train / val / test: train teaches the model, val checks and tunes during training, test is a one-time exam at the end. split() divides it stratified — splitting class by class, shuffling order with rng.permutation, then cutting $n_{c,test} = \lfloor 0.15,n_c \rfloor$, $n_{c,val} = \lfloor 0.15,n_c \rfloor$, with the rest as train. Every set therefore has all classes in the same proportion. The 143-window synthetic data splits into train [34 34 33], val [7 7 7], test [7 7 7]. Last is no leakage: normalize() computes mean/std from train only, then applies it to every set. If computed from all the data combined, test leaks into train, and the accuracy numbers will look too good, with no error to warn you.

Worked example

Section titled “Worked example”

This lesson’s slides also reference files in other lessons and under shared/:

Check your understanding

Section titled “Check your understanding”

The same questions are in quiz.yaml for automated checking.

  1. Put the pipeline for preparing sensor data, from raw to ready-to-train, in order (ordering · objective 1)

    • a) window: slice into 50-sample windows (on the PC)
    • b) capture: read the IMU at 50 Hz (on the board)
    • c) split: divide into train/val/test (on the PC)
    • d) label: attach the motion’s name (on the board)
    Solution

    b → d → a → c — capture and label happen on the board while collecting; window and split happen on the PC with dataset_tools.py, then get passed on to training.

  2. A dataset has idle 950 windows, circle 30, and shaking 20. What accuracy does a model that always answers idle get? (single choice · objective 2)

    • a) 33%
    • b) 50%
    • c) 95%, despite learning nothing at all
    • d) 100%
    Solution

    c — p_idle = 950/1000 = 0.95. The accuracy on paper is high but misleading. Balanced data should give each class about 1/3.

  3. The shaking class has 40 windows, split stratified (val = test = 0.15). How many windows go to train, val and test? (single choice · objective 3)

    • a) 28, 6, 6
    • b) 30, 5, 5
    • c) 26, 7, 7
    • d) 40, 0, 0
    Solution

    a — ⌊0.15 × 40⌋ = 6 go to test and 6 to val; the remaining 40 − 12 = 28 become train. Train always absorbs the remainder.

  4. Why must val be kept separate from test? (single choice · objective 3)

    • a) So train has less data
    • b) We look at val repeatedly while tuning, so val ends up mixed into our decisions. Only test, never touched, tells the real accuracy
    • c) val and test only differ in name
    • d) Because test must be bigger than train
    Solution

    b — val is used during training, so it starts leaning toward our tuning choices. Test is kept for a one-time exam at the end. If you peek at test while tuning, the numbers lie.

  5. Which code causes data leakage? (single choice · objective 4)

    • a) Split first, then normalize(Xtr, Xva, Xte), computing mean/std from Xtr
    • b) Normalize the whole X block with the mean/std of all the data, then split
    • c) Using rng.permutation before slicing each class
    • d) Setting a fixed seed for the split
    Solution

    b — mean/std computed including test lets test’s statistics leak into the data used for training. You must fit on train, then apply to every set.

Lab

Section titled “Lab”
  • Run dataset_tools.py with --synthesize, then reading the file, and note the number of samples, windows and class counts in your learning log.
  • In Python, call split() on the resulting windows, then print np.bincount for all three sets, and check it matches your hand calculation.
  • Write two lines explaining where the leak would occur if normalize() were called before split().

Going further

Section titled “Going further”

In lesson 5.2, we’ll fill in s11_dataset.py to record a real dataset on the board while watching a balance bar, then split it on the PC.

Next lesson: lesson 5.2 — Hands-on: recording a balanced dataset on the board, then splitting it on the PC

Reflect

Section titled “Reflect”
  • If you collected walking gestures from several people, why should train/test be split by “person” instead of by sample?
  • For a class that’s rare in real work, such as the sound of failing machinery, how would you collect it to keep things balanced?

Review questions

Answer on your own first, then open the answer.

  1. Order the preparation pipeline from sensor to ready-to-train. (Objective 1)

    1. window: ตัดเป็นหน้าต่าง 50 sample (บน PC)
    2. capture: อ่าน IMU ที่ 50 Hz (บนบอร์ด)
    3. split: แบ่ง train/val/test (บน PC)
    4. label: ติดป้ายท่าที่ทำ (บนบอร์ด)
    Show answer

    Correct order: B. capture: อ่าน IMU ที่ 50 Hz (บนบอร์ด) → D. label: ติดป้ายท่าที่ทำ (บนบอร์ด) → A. window: ตัดเป็นหน้าต่าง 50 sample (บน PC) → C. split: แบ่ง train/val/test (บน PC)

    capture กับ label เกิดบนบอร์ดขณะเก็บ ส่วน window กับ split ทำบน PC ด้วย dataset_tools.py แล้วจึงส่งต่อให้การฝึก

  2. A dataset has 950 idle, 30 circle and 20 shaking windows. What accuracy does a model that always answers idle get? (Objective 2)

    1. 33%
    2. 50%
    3. 95% ทั้งที่ไม่ได้เรียนอะไรเลย
    4. 100%
    Show answer

    Answer: C. 95% ทั้งที่ไม่ได้เรียนอะไรเลย

    p_idle = 950/1000 = 0.95 ความแม่นบนกระดาษจึงสูงแต่หลอกตา สมดุลควรให้ทุกคลาสได้ราว 1/3

  3. The shaking class has 40 windows. A stratified split (val = test = 0.15) gives how many train, val and test windows? (Objective 3)

    1. 28, 6, 6
    2. 30, 5, 5
    3. 26, 7, 7
    4. 40, 0, 0
    Show answer

    Answer: A. 28, 6, 6

    ⌊0.15 × 40⌋ = 6 ไป test และ 6 ไป val ที่เหลือ 40 − 12 = 28 เป็น train train จึงรับเศษที่เหลือเสมอ

  4. Why keep val separate from test? (Objective 3)

    1. เพื่อให้ train มีข้อมูลน้อยลง
    2. เราดู val ซ้ำ ๆ ตอนจูน val จึงปนการตัดสินใจของเราไปแล้ว test ที่ไม่เคยแตะเท่านั้นที่บอกความแม่นจริง
    3. val กับ test ต่างกันแค่ชื่อ
    4. เพราะ test ต้องใหญ่กว่า train
    Show answer

    Answer: B. เราดู val ซ้ำ ๆ ตอนจูน val จึงปนการตัดสินใจของเราไปแล้ว test ที่ไม่เคยแตะเท่านั้นที่บอกความแม่นจริง

    val ใช้ระหว่างฝึกจึงเริ่มเอนตามการจูน test เก็บไว้สอบครั้งเดียวตอนจบ ถ้าแอบดู test ระหว่างจูน ตัวเลขจะโกหก

  5. Which code leaks data? (Objective 4)

    1. split ก่อน แล้ว normalize(Xtr, Xva, Xte) ที่คิด mean/std จาก Xtr
    2. normalize X ทั้งก้อนด้วย mean/std ของข้อมูลทั้งหมด แล้วค่อย split
    3. ใช้ rng.permutation ก่อนหั่นแต่ละคลาส
    4. ตั้ง seed ของ split ให้คงที่
    Show answer

    Answer: B. normalize X ทั้งก้อนด้วย mean/std ของข้อมูลทั้งหมด แล้วค่อย split

    mean/std ที่คิดรวม test ทำให้สถิติของ test เข้าไปอยู่ในข้อมูลที่ใช้ฝึก ต้อง fit บน train แล้ว apply ทุกกอง

Cite this lesson

If you teach from this lesson or reuse it in slides or documents, credit it with the text below. If you changed it, add (adapted) after the title.

"Dataset engineering: class balance, windows and the train/val/test split" from TESA Open Knowledge by the Thai Embedded Systems Association (TESA), https://github.com/tesaiot/tesa-qualification-program, licensed under CC BY-NC 4.0

Thai attribution: "วิศวกรรมชุดข้อมูล: สมดุลคลาส หน้าต่าง และการแบ่ง train/val/test" จาก TESA Open Knowledge โดยสมาคมสมองกลฝังตัวไทย (Thai Embedded Systems Association: TESA) https://github.com/tesaiot/tesa-qualification-program สัญญาอนุญาต CC BY-NC 4.0

Lesson link: https://tesaiot.github.io/tesa-qualification-program/en/courses/edge-ai-developer/m05-training/l01-dataset-engineering/

Full guide: how to cite TESA

TESA Open Knowledge · © 2026 สมาคมสมองกลฝังตัวไทย (TESA) · CC BY-NC 4.0

Content is licensed CC BY-NC 4.0. Reuse it non-commercially and credit the Thai Embedded Systems Association (TESA) every time. · How to cite TESA