You have spent fourteen lectures learning the Python language, its standard library, object-oriented design, async programming, testing with pytest and building REST APIs with FastAPI. This final lecture turns to the two things that make Python the most used language in the world outside of web browsers: data analysis and automation.

Every company in India sits on spreadsheets and CSV exports: sales by city, attendance logs, GST invoices, server metrics, exam results. Someone has to clean those files, join them, summarise them and turn them into a chart for a Monday meeting. Doing that by hand in Excel takes hours and is impossible to repeat exactly. Doing it in Python with NumPy and pandas takes a few lines, runs in seconds and can be re-run every day with one command.

Why these two libraries specifically?

• NumPy (Numerical Python) provides the ndarray: a fixed-type, contiguous block of memory on which operations run in compiled C code. A loop that takes 200 milliseconds in pure Python often takes 2 milliseconds in NumPy. Every scientific library in Python (pandas, scikit-learn, PyTorch, OpenCV) is built on NumPy arrays or copies their interface.
• pandas sits on top of NumPy and adds labels: column names, row indexes, dates. It knows how to read CSV, Excel, JSON, SQL and Parquet, how to group, join and pivot, and how to handle missing values honestly. pandas 2.x (2023–2025) and pandas 3.0 (early 2026) are the versions you will meet in production; pandas 3.0 made Copy-on-Write the default and switched text columns to a dedicated string dtype, and the code in this lecture works on both lines.

The second half of the lecture covers automation: renaming hundreds of files with pathlib, calling web APIs with httpx, running a job every morning with a scheduler, and wrapping all of it in a proper command-line tool with argparse so your teammates can run it without opening the code. We finish with a complete sales-report CLI as the hands-on exercise and a course wrap-up that maps out the web, data and AI learning paths on this site.

A note on versions: this lecture targets Python 3.13 and 3.14 (the current stable lines; 3.15 is scheduled for October 2026). Install the libraries with uv add numpy pandas matplotlib httpx openpyxl inside a project created by uv init, or with pip install in an activated venv.

The heart of NumPy is the ndarray (n-dimensional array). Unlike a Python list, which is an array of pointers to separately allocated objects, an ndarray stores all its elements in one contiguous block of memory with a single dtype (data type) such as int64, float64 or bool. That design is what makes NumPy fast: the CPU can stream through the memory without chasing pointers, and the arithmetic runs in compiled code.

Three attributes describe any array:

• shape — a tuple of dimension sizes. A list of 1,000 marks has shape (1000,); a table of 1,000 students with 5 subjects has shape (1000, 5).
• ndim — the number of dimensions (1 for a vector, 2 for a matrix, 3 for an image stack).
• dtype — the element type. Mixing an integer and a float in np.array upcasts everything to float; mixing a string in upcasts everything to a Unicode string, which is almost never what you want.

Ways to create arrays you will use daily:

• np.array([...]) from a Python list (or nested lists for 2-D).
• np.arange(start, stop, step) like range but returns an array and accepts floats.
• np.linspace(start, stop, num) gives num evenly spaced values including both ends — perfect for plotting.
• np.zeros(shape), np.ones(shape), np.full(shape, value) for pre-allocated buffers.
• np.random.default_rng(seed) returns a Generator; call .integers(), .normal(), .choice() on it. (The old np.random.seed / np.random.randint global API still works but the Generator API is the recommended one since NumPy 1.17.)

Indexing and slicing follow Python rules but extend to multiple axes: marks[0, 2] is row 0, column 2; marks[:, 2] is the whole third column; marks[::2] is every second row. Crucially, slices are views, not copies: modifying marks[:, 2] modifies marks. Call .copy() when you need independence. Use reshape to change shape without copying data (the total number of elements must stay the same) and -1 to let NumPy infer one dimension.

Vectorization means expressing a computation as whole-array operations instead of explicit Python loops. Write prices * 1.18 and NumPy applies the GST multiplication to every element inside a single C loop. The code is shorter, reads like the maths, and is typically 50 to 200 times faster than the equivalent for loop because Python's per-iteration overhead (type checks, reference counting, bytecode dispatch) disappears.

Every arithmetic operator, comparison operator and the universal functions (np.sqrt, np.exp, np.log, np.round, np.maximum) are vectorized. Reductions such as sum, mean, std, min, max and argmax take an axis argument: axis=0 collapses rows (giving one value per column), axis=1 collapses columns (one value per row). This trips up every beginner, so remember: the axis you pass is the axis that disappears.

Broadcasting is the rule set that lets NumPy combine arrays of different shapes. Shapes are compared from the right; two dimensions are compatible when they are equal or when one of them is 1. The size-1 dimension is stretched (virtually, with no copy) to match. So a (4, 3) matrix minus a (3,) vector subtracts the vector from every row, and a (4, 3) matrix minus a (4, 1) column subtracts a different value from each row. If the shapes are incompatible, for example (4, 3) with (4,), NumPy raises ValueError: operands could not be broadcast together; the fix is usually vector[:, np.newaxis] or vector.reshape(-1, 1) to add the missing axis.

Boolean masks complete the toolkit. A comparison such as marks >= 40 returns a boolean array of the same shape; use it to filter (marks[marks >= 40]), count ((marks >= 40).sum()), or assign (marks[marks < 0] = 0). np.where(condition, a, b) is the vectorized if-else. When you find yourself writing a Python loop over array elements, stop and ask whether a mask, a ufunc or broadcasting can express it; nearly always one can.

pandas adds two labelled containers on top of NumPy.

A Series is a one-dimensional array with an index: think of it as a single column, or a dictionary whose keys are ordered and whose values share a dtype. Arithmetic between two Series aligns on the index labels, not on position, which is both a superpower and a common source of surprise (misaligned labels produce NaN).

A DataFrame is a two-dimensional table: an ordered dictionary of Series that share the same row index. Each column has its own dtype, so a DataFrame can hold integers, floats, strings and datetimes side by side, unlike a NumPy array. You can build one from a dict of lists, a list of dicts, a NumPy array, or (most often) by reading a file.

First things to do with any new DataFrame:

• df.head(n) / df.tail(n) — peek at rows.
• df.shape — (rows, columns).
• df.info() — column names, non-null counts and dtypes in one screen; this is where you discover that a price column was read as text because of a stray ₹ symbol.
• df.describe() — count, mean, std, min, quartiles and max for numeric columns.
• df.columns, df.dtypes, df.index.

Access a column with df["city"] (returns a Series) and several columns with a list: df[["city", "amount"]] (returns a DataFrame). Create a derived column by assigning to a new name: df["total"] = df["qty"] * df["price"]. Column-wise methods such as .sum(), .mean(), .value_counts(), .unique(), .nunique() and .astype() cover most day-to-day questions.

pandas 3.0 defaults to Copy-on-Write (CoW): every operation that returns a new DataFrame or Series behaves as a copy, and modifying it never affects the original. The practical rule for both 2.x and 3.x is: never use chained assignment like df["qty"][3] = 0; always write df.loc[3, "qty"] = 0. Also prefer returning new objects over inplace=True, which is discouraged and gains nothing under CoW.

Real data starts as a file. pd.read_csv(path) is the function you will call most in your career, and its keyword arguments solve nearly every messy-file problem:

• sep=";" or sep="\t" for non-comma separators (many Indian bank exports use tabs or pipes).
• parse_dates=["date"] converts a column to datetime64 on load so you can use .dt.month, .dt.day_name() and date arithmetic immediately. For unusual formats add date_format="%d-%m-%Y"; without it, pandas infers the format from the first value, and dd-mm-yyyy files can be misread as mm-dd-yyyy.
• dtype={"pincode": "string"} keeps leading zeros that would vanish if the column were parsed as an integer.
• usecols=[...] loads only the columns you need; nrows=1000 is handy for a quick look at a 2 GB file.
• na_values=["-", "NA", "null"] tells pandas which placeholder strings mean missing.
• encoding="utf-8" is the default; older Windows exports may need "cp1252" or "latin-1".
• thousands="," parses "1,25,000"-style numbers (note: Indian lakh grouping works because pandas simply strips the separator).

pd.read_excel(path, sheet_name="Sales") needs the openpyxl package for .xlsx files. Pass sheet_name=None to get a dict of every sheet, header=2 when the real header is on the third row, and skiprows to jump over report titles. Writing is symmetrical: df.to_csv("out.csv", index=False) (almost always pass index=False, otherwise you get an unnamed column of row numbers next time you read it), and df.to_excel("report.xlsx", sheet_name="Summary", index=False). To write several sheets, use pd.ExcelWriter as a context manager.

pandas can also read read_json, read_parquet (fast columnar format used in data engineering; needs pyarrow), read_sql (pass a SQLAlchemy engine or a sqlite3 connection) and even read_html to scrape tables from a web page. The same df.to_* family writes them back.

pandas offers two indexers, and knowing which to use is the difference between clean code and subtle bugs.

• df.loc[rows, cols] selects by label. Rows are chosen by index label (which may be an integer, a string or a date) and columns by name. Slices with loc are inclusive of the end label: df.loc[2:4] returns rows labelled 2, 3 and 4.
• df.iloc[rows, cols] selects by integer position, exactly like NumPy or a Python list. Slices are exclusive of the end: df.iloc[2:4] returns positions 2 and 3.

Both accept a single value, a list, a slice or a boolean array in each dimension. df.loc[df["city"] == "Delhi", ["product", "revenue"]] reads as "rows where city is Delhi, these two columns", and that one pattern covers most selection needs. When the index is the default RangeIndex (0, 1, 2, ...), loc and iloc look identical, which is why people mix them up; the moment you sort_values, drop rows or set_index("order_id"), they diverge.

Filtering is boolean masking, just as in NumPy, with three rules:

1. Combine conditions with & (and), | (or) and ~ (not), never with the Python keywords and/or, which raise ValueError: The truth value of a Series is ambiguous.
2. Wrap every condition in parentheses because & binds tighter than ==: (df["qty"] > 1) & (df["city"] == "Delhi").
3. Use the helper methods for readable conditions: .isin([...]), .between(a, b), .str.contains("Lap"), .str.startswith, .isna(), .dt.year == 2026.

df.query("qty > 1 and city == 'Delhi'") is a string-based alternative that many analysts find more readable, and it accepts local variables with an @ prefix. Finally, df.sort_values(["city", "revenue"], ascending=[True, False]), df.nlargest(3, "revenue") and df.sample(5, random_state=0) are the selection-adjacent helpers you will reach for constantly.

The question every manager asks — "total revenue by city", "average order value by month", "how many orders per product per region" — is answered with groupby. pandas implements the classic split-apply-combine pattern: split the rows into groups by one or more keys, apply an aggregation to each group, combine the results into a new table.

df.groupby("city") by itself does nothing visible; it returns a lazy DataFrameGroupBy object. You then pick columns and an aggregation:

• df.groupby("city")["revenue"].sum() — one Series indexed by city.
• df.groupby(["city", "product"])["revenue"].sum() — a Series with a two-level MultiIndex; call .reset_index() to turn the levels back into columns, or pass as_index=False to groupby.
• .agg(["sum", "mean", "count"]) — several statistics at once, producing one column per function.
• Named aggregation .agg(total=("revenue", "sum"), orders=("order_id", "count"), avg_qty=("qty", "mean")) — the cleanest syntax: each keyword becomes an output column, each tuple is (input column, function). Functions can be strings, NumPy functions or your own lambdas.

Beyond aggregation, groupby supports transform, which returns a result the same length as the input (ideal for "each order's share of its city total": df["revenue"] / df.groupby("city")["revenue"].transform("sum")), and filter, which keeps whole groups meeting a condition (.filter(lambda g: g["revenue"].sum() > 50000)). .size() counts rows per group including missing values; .count() counts non-null values per column.

Two related reshaping tools: pd.pivot_table(df, values="revenue", index="city", columns="product", aggfunc="sum", fill_value=0) builds the Excel-style cross-tab people love, and pd.crosstab(df["city"], df["product"]) counts occurrences. For time-based grouping use df.resample("ME") on a DatetimeIndex, or pd.Grouper(key="date", freq="ME") inside groupby ("ME" is month-end; the older "M" alias is deprecated in pandas 2.2+).

Data rarely lives in one table. Orders reference customers by ID; cities belong to regions; this month's file must be stacked under last month's. pandas gives you two operations for this, and they map directly onto SQL concepts you learned in the FastAPI lecture.

pd.merge(left, right, on="key", how=...) (also available as left.merge(right, ...)) is a SQL join:

• how="inner" (default) keeps only keys present in both tables. Silent data loss happens here: if a city in orders is misspelled, those orders simply vanish.
• how="left" keeps every row of the left table and fills unmatched right columns with NaN. This is the one to use when enriching a fact table (orders) with a lookup table (regions) — you never want to lose orders.
• how="right" and how="outer" mirror and union the above.
• Use left_on / right_on when the key columns have different names, suffixes=("_order", "_cust") to disambiguate same-named non-key columns, and validate="many_to_one" to make pandas raise an error if the lookup table unexpectedly has duplicate keys (a classic cause of row counts doubling after a merge).
• indicator=True adds a _merge column saying both, left_only or right_only — perfect for finding the unmatched rows.

pd.concat([df1, df2]) stacks DataFrames. With the default axis=0 it appends rows (same columns, more rows: month files); with axis=1 it places tables side by side aligned on the index. Pass ignore_index=True to rebuild a clean 0..n index when stacking, otherwise duplicated index labels will haunt later loc calls. A common pattern reads every CSV in a folder: pd.concat([pd.read_csv(p) for p in Path("data").glob("sales_*.csv")], ignore_index=True).

df.join(other) is a thin wrapper around merge that joins on the index; it is convenient after set_index, but merge with explicit on is clearer and is what most teams standardise on. Always check len(result) before and after a merge: an unexpected increase means duplicate keys on the right, an unexpected decrease means you used an inner join where a left join was intended.

Missing values are the normal state of real data: a customer skipped the phone-number field, a sensor went offline for an hour, a join found no match. pandas represents them as NaN (float columns), NaT (datetime columns) and pd.NA (the newer nullable dtypes such as Int64, boolean and string). Most reductions skip missing values by default (mean, sum, max all ignore NaN), which is convenient but means a column average can look fine while half the data is missing.

Step one is always to measure: df.isna().sum() gives the missing count per column, and df.isna().mean().round(3) gives the fraction. Decide per column what the gap means. Then choose a strategy:

• Drop — df.dropna() removes any row with a missing value anywhere, which is usually too aggressive. Prefer df.dropna(subset=["order_id", "date"]) to require only the critical columns, or df.dropna(thresh=5) to keep rows with at least 5 non-null values. df.dropna(axis=1, how="all") drops columns that are entirely empty.
• Fill with a constant — df["qty"].fillna(1) or df.fillna({"qty": 1, "discount": 0, "city": "Unknown"}) with a dict for per-column values.
• Fill statistically — df["price"].fillna(df["price"].median()) (median resists outliers better than mean), or group-aware: df["price"].fillna(df.groupby("product")["price"].transform("median")).
• Fill from neighbours — .ffill() (carry the last valid value forward, right for daily balances or sensor readings) and .bfill(). The old fillna(method="ffill") form is deprecated.
• Interpolate — df["temperature"].interpolate() draws a straight line between known points; use method="time" on a DatetimeIndex for irregular spacing.

A column with one NaN becomes float64 even if every other value is an integer, which is why you see 2.0 instead of 2. Convert to the nullable Int64 dtype (capital I) with .astype("Int64") to keep integers alongside pd.NA. Never compare with == np.nan (it is always False); use .isna() and .notna(). Document every fill decision in code comments; a silent fillna(0) on a revenue column is the kind of thing that produces a wrong board presentation.

A number nobody reads is wasted; a chart gets looked at. matplotlib is the foundational plotting library in Python; pandas' df.plot() and the statistical library seaborn are both built on it. Learn the object-oriented API (fig, ax = plt.subplots()) rather than the plt.plot() shortcuts: it scales to multiple panels and makes it explicit which axes you are drawing on.

The vocabulary: a Figure is the whole canvas (window or image file); an Axes is one plot with its own x and y axis, title and legend; a figure can hold several Axes in a grid via plt.subplots(nrows, ncols). Draw with ax.bar, ax.plot (line), ax.scatter, ax.hist, ax.pie; label with ax.set_title, ax.set_xlabel, ax.set_ylabel; call ax.legend() when you have passed label= to the drawing calls; and finish with fig.tight_layout() so labels do not overlap. fig.savefig("chart.png", dpi=150) writes an image (also .svg and .pdf); plt.show() opens an interactive window. In a script that runs on a server or in a scheduled job, set matplotlib.use("Agg") before importing pyplot so it never tries to open a window.

pandas shortcuts: df.plot(x="month", y="revenue", kind="line"), series.plot.bar(), df.plot.hist(bins=20). They accept an ax= argument, so you can mix pandas plotting with matplotlib customisation. Number formatting matters for Indian audiences: divide by 1e5 and label the axis "₹ lakh", or use a FuncFormatter to format ticks as 12.5L.

Chart selection rule of thumb: bar for comparing categories (revenue by city), line for trends over time (monthly revenue), histogram for distributions (order values), scatter for relationships (price vs quantity). Keep one message per chart, start bar axes at zero, and prefer a sorted bar chart to a pie chart whenever there are more than three slices. The hands-on exercise at the end saves a bar chart automatically every time the report runs.

Automation is where Python pays for itself on day one. Three building blocks cover most office and DevOps tasks.

File operations with pathlib. Path.iterdir() lists a folder, Path.glob("*.jpg") filters by pattern and Path.rglob searches subfolders. Each Path exposes .stem (name without extension), .suffix, .parent, .stat().st_mtime (modification time) and .with_name() / .with_suffix() for computing the new name. Rename with path.rename(new_path) or, to move across drives, shutil.move. Two safety rules from painful experience: always support a --dry-run flag that prints what would happen without touching anything, and never overwrite silently: check new_path.exists() first. Python 3.14 added Path.copy() and Path.move() methods directly on pathlib; on 3.13 use shutil.

HTTP with httpx. httpx is the modern requests-compatible client with sync and async APIs, HTTP/2 and strict timeouts. Use a with httpx.Client(base_url=..., timeout=10.0) as client: block to reuse connections across many calls; call response.raise_for_status() so a 404 or 500 becomes an exception instead of silently returning an error page; read JSON with response.json(); send JSON with client.post(url, json={...}). Pass API keys through headers={"Authorization": f"Bearer {token}"} with the token loaded from an environment variable, never hard-coded. Wrap calls in try/except httpx.HTTPError and add a simple retry loop with exponential backoff for flaky networks. For hundreds of URLs use httpx.AsyncClient with asyncio.gather as you learned in the async lecture.

Scheduling. For a script that must run at 9:00 every morning you have three realistic options: the operating system's scheduler (cron on Linux/macOS: 0 9 * * * /path/to/.venv/bin/python /path/to/job.py; Task Scheduler on Windows), which is the most robust because it survives reboots and needs no running Python process; the tiny schedule package (schedule.every().day.at("09:00").do(job) inside a while True: schedule.run_pending(); time.sleep(30) loop) for a long-running process; or APScheduler when you need cron-style expressions, persistence and multiple jobs inside a FastAPI app. In the cloud, GitHub Actions on: schedule with a cron expression runs a script for free on a timetable. Whichever you choose, log every run with the logging module to a file, because a scheduled job that fails silently at 3 a.m. is worse than no job at all.

A script that needs its code edited to change the input file is not a tool; a script that accepts python report.py sales.csv --top 5 --chart is. The standard library's argparse turns a function into a professional command-line interface with automatic --help, type conversion, validation and error messages.

The workflow is always the same:

1. Create parser = argparse.ArgumentParser(prog="report", description="..."). The description appears in --help.
2. Declare positional arguments (parser.add_argument("csv", type=Path)) for required inputs and optional flags ("--top") for settings.
3. Give each argument a type (int, float, Path, or any one-argument function), a default, a help string, and where useful choices=["csv", "xlsx"] for enumerations. action="store_true" creates a boolean switch; nargs="+" collects one or more values into a list; required=True makes an option mandatory.
4. Call args = parser.parse_args() and read args.csv, args.top. Passing a list to parse_args instead of relying on sys.argv makes the parser testable with pytest.

For tools with several verbs (generate, report, clean), use subparsers: sub = parser.add_subparsers(dest="command", required=True) then sub.add_parser("report", help=...) with its own arguments. A match args.command: statement (Python 3.10+) dispatches cleanly. Exit with raise SystemExit(code) or parser.error("message"), which prints usage and exits with status 2, so shell scripts and CI pipelines can detect failure. Python 3.14 made argparse help output coloured by default in a terminal and added suggest_on_error=True so a typo like --tpo suggests --top.

When should you reach for a third-party library instead? Typer builds the CLI from type-hinted function signatures and Click uses decorators; both are excellent for large tools with many commands. But argparse ships with Python, needs no dependency, and is what you will find in most existing codebases and interview questions. Add a [project.scripts] entry in pyproject.toml (report = "sales_report:main") and uv sync installs your tool as a real command.

Here is how the techniques in this lecture appear in actual Indian workplaces, so you can picture where your new skills fit.

• Finance and accounting reconciliation — a fintech in Mumbai receives a daily settlement CSV from a payment gateway and a transactions export from its own database. A pandas script reads both (read_csv with dtype={"txn_id": "string"} to protect long IDs), merges with how="outer", indicator=True, and every left_only or right_only row becomes a ticket. What took an analyst four hours now runs in 20 seconds at 6 a.m. via cron.
• E-commerce analytics — a Bengaluru marketplace computes weekly category revenue, repeat-purchase rate and average order value per city with groupby and pivot_table, writes a multi-sheet Excel with ExcelWriter, renders charts with matplotlib and emails the file to category managers. The whole pipeline is an argparse CLI with --week and --dry-run flags, scheduled by GitHub Actions.
• Data engineering pipelines — pandas is the glue in ETL: read Parquet from S3, clean with fillna/astype/drop_duplicates, validate row counts, write to a warehouse with to_sql. Teams at larger scale move to Polars, Spark or DuckDB, but the DataFrame vocabulary (select, filter, groupby, join) transfers directly.
• Machine learning preprocessing — every scikit-learn or PyTorch project begins with a DataFrame: filtering outliers with boolean masks, encoding categories, filling missing values with group medians, then .to_numpy() to hand a NumPy array to the model. The AI track on this site builds on exactly these operations.
• IoT and sensor monitoring — a factory in Pune logs temperature every 10 seconds. NumPy computes rolling means and standard deviations on millions of readings via vectorized operations; pandas resample("5min") downsamples for dashboards; interpolate(method="time") patches short outages.
• IT and DevOps automation — renaming and archiving thousands of log files with pathlib, polling a health-check API with httpx and posting to Slack when it fails, rotating backups on a schedule, and small internal CLIs (deploytool status --env prod) that wrap cloud SDKs with argparse.
• Education and HR — colleges merge attendance and marks sheets from different Excel formats, compute per-student percentages with broadcasting, and generate per-department histograms. HR teams deduplicate applicant spreadsheets with drop_duplicates(subset=["email"]) and str.lower().str.strip() normalisation.

The common thread: read, clean, join, summarise, visualise, schedule. Master that loop and you are productive in any of these domains within a week.

• Looping over rows with iterrows() — it is 100 to 1,000 times slower than vectorized code and returns each row as a Series with a single upcast dtype. Fix: express the logic with column arithmetic, np.where, .map, or groupby().transform. Reach for .apply(axis=1) only as a last resort, and itertuples() if you truly must iterate.
• Chained assignment such as df[df["qty"] > 1]["discount"] = 0.1 — in pandas 2.x this triggers SettingWithCopyWarning and silently does nothing; under pandas 3.0 Copy-on-Write it does nothing without a warning. Fix: a single df.loc[mask, "discount"] = 0.1.
• Using and / or between masks — raises "The truth value of a Series is ambiguous". Fix: &, |, ~ with parentheses around each comparison.
• Confusing loc and iloc after sorting or dropping rows — df.loc[0] is the row labelled 0, which may now be anywhere, while df.iloc[0] is the first row. Fix: call reset_index(drop=True) after reordering, or be explicit about which indexer you want.
• Wrong axis — df.drop("city") fails because the default axis is rows; marks.mean(axis=1) when you wanted per-column. Fix: remember "the axis you pass is the one that collapses", and use columns=["city"] in drop instead of axis=1.
• Row count silently changing after merge — duplicate keys on the right side multiply rows; an inner join drops unmatched rows. Fix: validate="many_to_one", how="left" for enrichment, and assert len(before) == len(after).
• Forgetting index=False in to_csv — the next read produces an Unnamed: 0 column. Fix: always pass index=False unless the index is meaningful.
• Comparing with == np.nan — always False. Fix: .isna() / .notna(); for nullable dtypes use pd.isna(value).
• Misparsed dates — "03-04-2026" read as 4 March instead of 3 April. Fix: pass date_format="%d-%m-%Y" (or dayfirst=True) and spot-check df["date"].min() and .max().
• Lost leading zeros and precision — pincodes, phone numbers and Aadhaar-like IDs parsed as integers. Fix: dtype={"pincode": "string"} at read time.
• Modifying a NumPy view by accident — a slice shares memory with the original. Fix: .copy() when you intend to mutate independently.
• Broadcasting shape errors — (n, 3) minus (n,). Fix: vec[:, np.newaxis] so the trailing dimensions line up.
• Automation without a dry run or logging — a rename script that overwrote 400 photos. Fix: --dry-run flag, exists() checks, and logging to a file.
• Hard-coded API keys and no timeouts — a leaked key on GitHub, or a script that hangs forever on a dead endpoint. Fix: os.environ, timeout=10.0, raise_for_status(), bounded retries with backoff.
• Using plt.plot globals in a multi-chart script — charts bleed into each other. Fix: the object-oriented fig, ax = plt.subplots() API and matplotlib.use("Agg") for scripts.

What is the difference between a NumPy array and a Python list?

A NumPy array stores elements of one dtype in a contiguous memory block and performs arithmetic in compiled C code, so it is far faster and more memory-efficient for numeric work and supports vectorized operations, broadcasting and multi-dimensional indexing. A Python list stores pointers to arbitrary objects, can mix types and grow dynamically, but every operation runs element by element in the interpreter.

What is the difference between loc and iloc in pandas?

loc selects by index and column labels and its slices include the end label; iloc selects by integer position like a Python list and its slices exclude the end. They look the same on a fresh DataFrame with a 0..n index but diverge after sorting, filtering or set_index.

How do I handle missing values in a pandas DataFrame?

First measure them with df.isna().sum(), then choose per column: dropna(subset=[...]) for rows that are unusable, fillna with a constant, a median or a group median, ffill for time series, or interpolate for numeric gaps. Use the nullable Int64 and string dtypes to keep integers and text alongside pd.NA.

What is broadcasting in NumPy?

Broadcasting is the set of rules that lets NumPy combine arrays of different shapes without copying data. Shapes are compared from the right; dimensions must be equal or one of them must be 1, and the size-1 dimension is stretched to match. That is how a (1000, 3) matrix can subtract a (3,) row vector of column means in one expression.

How do I merge two DataFrames in pandas?

Use pd.merge(left, right, on="key", how="left") or left.merge(right, ...). how chooses inner, left, right or outer join semantics exactly like SQL; left_on/right_on handle differently named keys, suffixes disambiguate overlapping columns, validate guards against duplicate keys and indicator=True reveals unmatched rows.

Is pandas 3.0 different from pandas 2.x? Will my code break?

pandas 3.0 (early 2026) enables Copy-on-Write by default, so chained assignment never modifies the original, and text columns use a dedicated string dtype instead of object. Code that already uses df.loc[mask, col] = value, avoids inplace=True and does not rely on object dtype for strings runs unchanged; the pandas team recommends upgrading to 2.3 first and fixing its deprecation warnings.

Should I use argparse, Click or Typer for a Python CLI?

argparse is in the standard library, needs no installation and is enough for most scripts, including ones with subcommands. Typer (built on Click) generates the CLI from type hints with less boilerplate and better help output, and is a good choice for larger tools you distribute to others. Learn argparse first because it is everywhere.

How do I schedule a Python script to run every day?

On Linux and macOS use cron (0 9 * * * /path/.venv/bin/python /path/job.py), on Windows use Task Scheduler, and in the cloud use GitHub Actions on: schedule or a managed scheduler. Use the schedule or APScheduler packages only when the job must live inside an already long-running Python process, and always log each run to a file.

Q1. Why is NumPy faster than pure Python for numeric work?

Because an ndarray stores homogeneous data contiguously and its operations are implemented in C, a single call like a * b runs one tight compiled loop with no per-element type checking, reference counting or bytecode dispatch. The interpreter is entered once instead of a million times, and contiguous memory lets the CPU use its cache and SIMD instructions efficiently.

Q2. Explain the axis parameter in NumPy and pandas.

axis=0 runs down the rows (operating on each column), axis=1 runs across the columns (operating on each row). The axis you pass is the one that is collapsed by a reduction: df.sum(axis=0) gives one value per column. In drop and concat the axis names what you are removing or stacking: rows (0) or columns (1).

Q3. What is a view versus a copy in NumPy, and why does it matter?

Basic slicing (a[2:5], a[:, 0]) returns a view that shares memory with the original, so writing to the slice changes the original; fancy indexing with lists or boolean masks returns a copy. It matters because accidental mutation through a view is a silent bug, and unnecessary copies of large arrays waste memory. Use .copy() to be explicit, and np.shares_memory(a, b) to check.

Q4. What does groupby do internally, and what is split-apply-combine?

groupby computes the group each row belongs to (split), applies an aggregation, transformation or filter to each group independently (apply), and assembles the results into a new Series or DataFrame indexed by group key (combine). Aggregation returns one row per group, transform returns the original shape, and filter returns a subset of the original rows.

Q5. What is the difference between merge, join and concat?

merge performs SQL-style joins on one or more key columns with inner/left/right/outer semantics. join is a convenience wrapper that joins on the index. concat does not match keys at all; it stacks DataFrames vertically (new rows) or horizontally (new columns, aligned on index).

Q6. How does pandas represent missing data, and what is the difference between NaN and pd.NA?

NaN is a float value, so an integer column with a missing value becomes float64, and NaT is its datetime equivalent. pd.NA is a dtype-agnostic missing marker used by the nullable dtypes (Int64, boolean, string), which keep their type and propagate NA through comparisons instead of returning False. Always test with isna() rather than equality.

Q7. What is the SettingWithCopyWarning and how does Copy-on-Write change things?

In pandas 2.x, df[mask]["col"] = value first creates a possibly-temporary copy and then assigns into it, so the warning tells you the original may not have changed. pandas 3.0's Copy-on-Write makes the rule simple: any derived object is a logical copy and mutating it never affects the parent, so chained assignment is simply ineffective. The correct form in both versions is df.loc[mask, "col"] = value.

Q8. How would you read a 10 GB CSV that does not fit in memory with pandas?

Use pd.read_csv(path, chunksize=500_000) to iterate over chunks and aggregate incrementally, load only needed columns with usecols, downcast dtypes (category for repeated strings, float32), or convert the file once to Parquet and query it with pyarrow, DuckDB or Polars. Reading the whole file into a DataFrame is not the only option.

Q9. How do you make an argparse CLI testable?

Separate parser construction (build_parser()) from execution (main(argv=None)) and have main call parser.parse_args(argv). Tests then call main(["report", "file.csv", "--top", "3"]) directly and inspect the return code and captured output with pytest's capsys, without touching sys.argv or spawning a subprocess.

Q10. How do you make an automation script that calls an external API robust?

Set an explicit timeout, call raise_for_status(), catch httpx.HTTPError, retry a bounded number of times with exponential backoff, read credentials from environment variables, make the script idempotent so a rerun does no harm, log every run to a file, and provide a --dry-run mode for any destructive step.

This exercise combines everything in the lecture into one tool a small business could actually use. sales_report.py has two subcommands:

• generate creates a realistic sales.csv with 500 orders across eight Indian cities, deliberately including missing quantities and duplicate rows, so you practise cleaning.
• report loads the CSV, removes duplicates, fills missing quantities, computes revenue, merges in a city-to-region lookup, prints revenue by city and region, month-on-month growth, the top products, and optionally saves a bar chart.

Set up and run it:

1. uv init sales-tool && cd sales-tool && uv add numpy pandas matplotlib
2. Save the code as sales_report.py.
3. uv run python sales_report.py generate
4. uv run python sales_report.py report sales.csv --top 3 --chart
5. uv run python sales_report.py report sales.csv --month 2026-03

Read the code top to bottom and notice the patterns: a fixed random seed for reproducibility, drop_duplicates(subset="order_id"), fillna with a documented reason, named aggregation, a left merge with validate, pct_change for growth, the object-oriented matplotlib API with a headless backend, match on the subcommand, and raise SystemExit(main()) so the exit code reaches the shell.

Extension ideas once it works: add a --to xlsx option that writes every summary table to a multi-sheet workbook; add a --region filter; write pytest tests for load_and_clean using tmp_path; schedule the report command with cron and httpx.post the chart to a Slack webhook.

• NumPy gives Python a fast, fixed-dtype ndarray; think in shape, dtype and axis, remember that slices are views, and use default_rng for reproducible randomness.
• Vectorization replaces loops with whole-array expressions; broadcasting aligns shapes from the right (equal or 1); boolean masks and np.where express conditional logic.
• pandas adds labels: a Series is a labelled column, a DataFrame a table of them. Start every analysis with head, info and describe.
• read_csv / read_excel with parse_dates, dtype, na_values and thousands turn messy files into typed tables; write back with index=False.
• Select by label with loc, by position with iloc; filter with &, |, ~, isin, between, str and dt accessors, or query.
• groupby is split-apply-combine; prefer named aggregation, use transform for same-shape results and pivot_table for cross-tabs.
• merge is a SQL join (use how="left", validate, indicator); concat stacks tables; always check row counts.
• Missing data: measure with isna().sum(), then dropna(subset), fillna (constant, median, group median), ffill or interpolate; use nullable Int64 / string dtypes.
• matplotlib: fig, ax = plt.subplots(), label everything, savefig with the Agg backend in scripts.
• Automation: pathlib for files with --dry-run and existence checks; httpx with timeouts, raise_for_status and retries; cron / Task Scheduler / GitHub Actions for scheduling, with logging.
• argparse turns scripts into real tools: positional and optional arguments, types, choices, subparsers, parse_args(argv) for testability, match for dispatch.

Course wrap-up. You have now completed the Complete Python Course: from variables and control flow, through functions, data structures, OOP, error handling, files, modules and packaging, iterators and generators, decorators and context managers, async programming, testing with pytest and REST APIs with FastAPI, to data analysis and automation today. That is the full toolkit of a professional Python developer in 2026. The single most important next step is to build something you will actually use: a tool that reconciles your expenses, a scraper for job listings in your city, a FastAPI service with a pandas report endpoint, or a scheduled script that saves you one boring task every morning. Put it on GitHub with a README, tests and a pyproject.toml; that repository will do more for your interviews than any certificate.

Where to go next on this site:

• Web track — if you enjoyed FastAPI, continue with the Complete Next.js Course and the React notes to build full-stack products: a Next.js frontend talking to your FastAPI backend, with authentication, caching and deployment.
• Data track — go deeper into pandas with time series, window functions and performance (Polars, DuckDB, Parquet), then SQL for analysts and dashboarding; the groupby/merge/missing-data skills from this lecture are the foundation of every data analyst and data engineer role.
• AI track — the NumPy array operations you practised today are exactly what scikit-learn, PyTorch and the LLM tooling expect. Follow the AI notes on this site to go from DataFrames to machine learning models, embeddings and building agents with Python.

Thank you for completing the course. Keep writing Python every day, read the official documentation when something surprises you, and come back to these notes whenever you need a reference. Happy coding!

Next lecture: This is the final lecture of the Complete Python Course. Continue with the Next.js, React, Java or AI tracks on this site.