TrueVacancy Academy

Machine Learning & AI · Chapter 4.8 · Pro level

Train/Test Split + Accuracy (Test Your AI Honestly)

Train/Test Split + Accuracy (अपने AI की ईमानदारी से जाँच)

Imagine a teacher who puts the exact same questions in the exam that were on the practice sheet. A student who only memorised the answers scores 100% — but did he really learn? To test an AI honestly, we hide part of the data: the model learns from one part (the training set) and is checked on the other part it has never seen (the test set). scikit-learn does this with train_test_split(X, y, test_size=0.25, random_state=42): 25% of the rows go to the test set, and random_state=42 makes the shuffle the same every run. After fit on the training part and predict on the test part, accuracy_score(y_test, predictions) gives the fraction of correct answers (1.0 = 100%). A model that is great on training data but bad on test data is overfitting — it memorised instead of learning. This chapter is real scikit-learn code (n_neighbors is the real parameter name), so it also runs unchanged in Google Colab.

सोचो, एक टीचर परीक्षा में बिल्कुल वही सवाल दे दे जो practice sheet में थे। जिसने सिर्फ़ जवाब रटे थे, वह भी 100% ले आएगा — पर क्या उसने सच में सीखा? AI को ईमानदारी से जाँचने के लिए हम data का एक हिस्सा छुपा देते हैं: model एक हिस्से से सीखता है (training set) और दूसरे हिस्से पर जाँचा जाता है जिसे उसने कभी नहीं देखा (test set)। scikit-learn यह train_test_split(X, y, test_size=0.25, random_state=42) से करता है: 25% rows test set में जाती हैं, और random_state=42 हर बार एक जैसा shuffle करता है। Training हिस्से पर fit और test हिस्से पर predict के बाद accuracy_score(y_test, predictions) सही जवाबों का हिस्सा बताता है (1.0 = 100%)। जो model training data पर बढ़िया पर test data पर ख़राब हो, वह overfitting कर रहा है — उसने सीखा नहीं, रटा है। यह चैप्टर असली scikit-learn कोड है (n_neighbors असली parameter का नाम है), इसलिए यह Google Colab में भी बिना बदले चलता है।

Open this chapter in the live classroomMachine Learning & AI course outline

In the classroom you can run and edit this code in your browser, get hints and mark the chapter as done. क्लास में यह code ब्राउज़र में चलाकर बदल सकते हो।

Words and symbols in this chapter

from sklearn.model_selection import train_test_split
The tool that splits data into a training part and a hidden test part.
वह टूल जो data को training हिस्से और छुपे हुए test हिस्से में बाँटता है।
test_size=0.25
25% of the rows go to the test set (here 3 of 12).
25% rows test set में जाती हैं (यहाँ 12 में से 3)।
random_state=42
Fixes the shuffle, so the split is the same every run.
Shuffle तय कर देता है, ताकि हर बार वही बँटवारा हो।
KNeighborsClassifier(n_neighbors=3)
KNN that looks at the 3 nearest training rows.
KNN जो सबसे पास की 3 training rows देखता है।
accuracy_score(y_test, predictions)
Fraction of correct answers: 1.0 means 100%.
सही जवाबों का हिस्सा: 1.0 मतलब 100%।
Overfitting
Very good on training data, poor on new data — memorising instead of learning.
Training data पर बहुत अच्छा, नए data पर ख़राब — सीखने की जगह रटना।

Ready example

from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

# [study hours per day, phone hours per day]
X = [[1, 6], [2, 5], [1, 7], [2, 6], [3, 5], [2, 7],
     [6, 1], [7, 2], [5, 1], [6, 2], [8, 1], [7, 1]]
y = ["Fail", "Fail", "Fail", "Fail", "Fail", "Fail",
     "Pass", "Pass", "Pass", "Pass", "Pass", "Pass"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
print("Training rows:", len(X_train), "| Test rows:", len(X_test))

knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train, y_train)
predictions = knn.predict(X_test)

for row, guess, real in zip(X_test, predictions, y_test):
    print(row, "-> AI:", guess, "| real:", real)

print("Accuracy:", accuracy_score(y_test, predictions))

What you will see: The KNN map shows the training points; the terminal lists every test row with the AI answer, the real answer and the accuracy.

KNN मैप पर training points दिखेंगे; terminal में हर test row के साथ AI का जवाब, असली जवाब और accuracy छपेगी।

Line by line

X_train, X_test, y_train, y_test = train_test_split(...)
One line gives 4 lists: training questions/answers and test questions/answers. (Colab may pick different test rows because its shuffle differs — the idea is the same.)
एक लाइन से 4 lists: training के सवाल/जवाब और test के सवाल/जवाब। (Colab में shuffle अलग होने से test rows अलग हो सकती हैं — तरीका वही है।)
knn.fit(X_train, y_train)
The model studies ONLY the training part.
Model सिर्फ़ training हिस्से से पढ़ता है।
predictions = knn.predict(X_test)
It answers questions it has never seen.
वह उन सवालों के जवाब देता है जो उसने कभी नहीं देखे।
accuracy_score(y_test, predictions)
Compares its answers with the real answers.
उसके जवाबों को असली जवाबों से मिलाता है।

Common mistake

✗ Wrong

tree.fit(X, y)
print(accuracy_score(y, tree.predict(X)))

✓ Right

tree.fit(X_train, y_train)
print(accuracy_score(y_test, tree.predict(X_test)))

Measuring accuracy on the same data you trained on is like checking a student with the practice questions — it hides overfitting.

जिस data पर train किया उसी पर accuracy नापना, छात्र को practice वाले सवालों से जाँचने जैसा है — इससे overfitting छुप जाती है।

Your task

Split the cricket data with train_test_split(X, y, test_size=0.25, random_state=42), train a DecisionTreeClassifier on the TRAINING part only, predict X_test and print Accuracy: using accuracy_score.

train_test_split(X, y, test_size=0.25, random_state=42) से cricket data बाँटो, DecisionTreeClassifier को सिर्फ़ TRAINING हिस्से पर train करो, X_test की prediction करो और accuracy_score से Accuracy: print करो।

  1. Split the data: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
    Data बाँटो: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
  2. tree = DecisionTreeClassifier(), then tree.fit(X_train, y_train) — not the full X!
    tree = DecisionTreeClassifier(), फिर tree.fit(X_train, y_train) — पूरा X नहीं!
  3. predictions = tree.predict(X_test) and print("Accuracy:", accuracy_score(y_test, predictions)) — then Run.
    predictions = tree.predict(X_test) और print("Accuracy:", accuracy_score(y_test, predictions)) — फिर Run करो।
Starter code for the task
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

# [runs scored, wickets lost]
X = [[210, 3], [185, 5], [230, 2], [195, 4], [250, 1], [205, 6],
     [120, 9], [140, 8], [150, 10], [130, 7], [160, 9], [110, 10]]
y = ["Win", "Win", "Win", "Win", "Win", "Win",
     "Lose", "Lose", "Lose", "Lose", "Lose", "Lose"]

# TODO 1: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
# TODO 2: tree = DecisionTreeClassifier()  phir  tree.fit(X_train, y_train)   (sirf TRAIN data se!)
# TODO 3: predictions = tree.predict(X_test)
# TODO 4: print("Accuracy:", accuracy_score(y_test, predictions))

Do this task in the classroom: your code is checked automatically and you get hints if something is missing.

Recap

Quick quiz

Why do we keep a separate TEST set that the model never sees during training?

हम एक अलग TEST set क्यों रखते हैं जिसे model training में कभी नहीं देखता?

  1. To make training faster
  2. To check honestly how the model does on new, unseen data
  3. Because sklearn always needs two lists
  4. To make the accuracy number bigger
Show the answer

Answer: B (To check honestly how the model does on new, unseen data)

Only unseen data shows whether the model learnt the pattern or just memorised the examples.

सिर्फ़ अनदेखा data ही बताता है कि model ने पैटर्न सीखा या बस उदाहरण रट लिए।

Practice problems for this chapter

Solve them in the Practice Zone, where each answer is checked automatically and two hints are available.