Almost every real program reads or writes data that lives outside its own memory: a configuration file, a log, a CSV export from Tally or an ERP, a JSON response saved from an API, images uploaded by users, or a 5 GB server log that must be scanned for errors. File handling is the bridge between your Python code and that outside world, and it is one of the first things a working developer is asked to do on day one of a job.

Python makes this unusually pleasant. The built-in open() function, the with statement, the pathlib module for paths, and the json and csv modules in the standard library mean you can parse a 50,000-row spreadsheet export or persist application settings without installing anything. Yet the same simplicity hides traps: forgetting to close a file, reading a whole 8 GB file into memory, writing a Hindi or Tamil string with the wrong encoding and getting garbage, or building paths with string concatenation that break on Windows.

In this lecture you will learn, in order:

• How to open files safely with context managers and understand text versus binary modes.
• Why encoding="utf-8" should be in nearly every open() call you write.
• Reading and writing line by line, so memory stays flat no matter how large the file is.
• The object-oriented pathlib.Path API, globbing for files, and the older os and shutil helpers you still need.
• Serialising Python data to JSON with json.dump and reading it back with json.load.
• Processing CSV files correctly with csv.DictReader and csv.DictWriter, including the famous newline="" rule.
• Keeping secrets such as API keys out of code with environment variables and .env files.
• Techniques for large files: streaming, chunked reads, generators and atomic writes.

Lecture 8 taught you exceptions and logging; here you will see FileNotFoundError, PermissionError, UnicodeDecodeError and json.JSONDecodeError in their natural habitat. Lecture 10 will go deeper into iterators, generators and writing your own context managers, which is exactly the machinery that makes with open(...) work.

This lecture targets Python 3.13/3.14 (Python 3.14 is the current stable series as of writing, with 3.15 scheduled for release in October 2026). Features are labelled with the version that introduced them where it matters.

The built-in open(file, mode="r", encoding=None, newline=None, ...) returns a file object. Reading and writing go through that object, and when you are done the file must be closed so the operating system can release the file handle and flush any buffered data to disk. Every process has a limit on open handles (often 1,024 on Linux by default), and buffered writes that are never flushed simply disappear if the program crashes.

The old way was f = open(...), work, then f.close(). The problem is exceptions: if an error is raised between open and close, the close() line never runs. You could wrap everything in try/finally, but Python gives you a cleaner tool: the with statement, also called a context manager. with open(...) as f: guarantees that f.close() is called when the block ends, whether it ends normally, via return, or because of an exception. This is the single most important habit in Python file handling: always open files with with.

A file object supports several reading methods:

• f.read() — the entire remaining content as one string (or bytes). Fine for small files, dangerous for large ones.
• f.read(n) — at most n characters (text mode) or bytes (binary mode).
• f.readline() — one line including its trailing newline, or an empty string at end of file.
• f.readlines() — a list of all lines; same memory concern as read().
• Iterating with for line in f: — one line at a time, lazily. This is the preferred way.

For writing you have f.write(string), which returns the number of characters written and does not add a newline, and f.writelines(iterable), which also adds nothing between items. You can open several files in one with statement by separating them with commas, and since Python 3.10 you can wrap that list in parentheses across multiple lines.

A file object also remembers its position. f.tell() returns the current offset and f.seek(0) jumps back to the start, which is how you re-read a file without reopening it.

The mode string tells Python two things: what you intend to do and whether the data is text or bytes.

The intent characters:

• "r" — read (default). The file must exist or you get FileNotFoundError.
• "w" — write. Creates the file if missing and truncates it to zero length if it exists. This silently destroys existing content, which is the most common cause of "my data vanished" bugs.
• "a" — append. Creates if missing, otherwise writes go to the end. Ideal for logs.
• "x" — exclusive create. Fails with FileExistsError if the file already exists, which is safer than "w" when you must never overwrite.
• Adding "+" enables both reading and writing: "r+" (read and write, no truncation), "w+" (truncate then read/write), "a+".

The data-type character:

• Text mode (default, or explicit "t"): Python decodes bytes into str using an encoding, and translates platform newlines (\r\n on Windows) into \n when reading. You get strings in, strings out.
• Binary mode ("b", as in "rb", "wb", "ab"): no decoding and no newline translation. You get bytes objects. Use it for images, PDFs, ZIP files, audio, pickles, and anything you download from the internet whose format you are not parsing as text.

Why does this distinction matter? If you open a JPEG in text mode, the decoder will hit byte sequences that are not valid UTF-8 and raise UnicodeDecodeError, or worse, on Windows the newline translation will corrupt the data. Conversely, if you open a text file in binary mode, you must call .decode("utf-8") yourself and remember to handle line endings.

A practical rule: "what would I see if I opened this in Notepad?" If it is readable text, use text mode with an explicit encoding. If it is gibberish, use binary mode.

A file on disk is just bytes. An encoding is the rule that maps characters to bytes and back. ASCII covers only 128 English characters; Latin-1 and Windows-1252 cover Western European languages; UTF-8 covers every character in Unicode, including Devanagari, Tamil, Bengali, the rupee sign ₹ (U+20B9, stored as the three bytes E2 82 B9) and emoji. UTF-8 is backwards compatible with ASCII, is the standard on the web and in Linux, and is what JSON, Git, GitHub and practically every modern tool expect.

The catch is that open() in text mode, when you do not pass encoding, uses the locale's preferred encoding. On macOS and most Linux systems that is UTF-8. On many Windows machines in India it is still cp1252. So code that works on your Ubuntu laptop can crash on a colleague's Windows desktop with UnicodeEncodeError: 'charmap' codec can't encode character '\u20b9' when it tries to write a rupee sign, or produce "mojibake" like नमसà¥à¤¤à¥‡ instead of नमस्ते when reading.

The fix is a habit: always pass encoding="utf-8" to open(), Path.read_text() and Path.write_text(). Note the version story:

• Python 3.7 added UTF-8 Mode: run with python -X utf8 or set PYTHONUTF8=1 to make UTF-8 the default everywhere.
• Python 3.10 added EncodingWarning, emitted when you run python -X warn_default_encoding and call open() without an encoding, so you can find the missing arguments in a large codebase.
• Python 3.11 added locale.getencoding().
• PEP 686 schedules UTF-8 as the default file encoding in Python 3.15, finally ending the problem. Until every machine you deploy to runs 3.15, keep passing it explicitly.

When you must read a file of unknown or dirty encoding, open() accepts an errors argument: errors="replace" substitutes undecodable bytes with the U+FFFD replacement character, errors="ignore" drops them, and errors="surrogateescape" lets you round-trip bad bytes unchanged. Use these deliberately for salvage jobs, never as a way to hide a bug. If a file comes from Excel on Windows it is often cp1252 or utf-8-sig (UTF-8 with a byte-order mark); opening with encoding="utf-8-sig" strips the invisible BOM that would otherwise appear glued to your first column name.

A text file object is an iterator over lines. Writing for line in f: asks the operating system for a buffered chunk (8 KB by default), hands you one line at a time, and only fetches more when needed. Memory usage stays roughly constant whether the file is 2 KB or 20 GB. Compare that with f.read() or f.readlines(), which must hold the entire file as one Python object: a 3 GB log needs more than 3 GB of RAM (Python strings have overhead), and your process gets killed.

Each line you receive keeps its trailing newline ("\n"). Almost every line-processing loop therefore starts with line.rstrip("\n") or line.strip(). Use rstrip("\n") when leading spaces matter (indented data, fixed-width records) and strip() when you want everything trimmed. Also be aware that the last line of a file may or may not end with a newline, so never assume it does.

Writing line by line is symmetrical. f.write() does not add a newline, so you add it yourself, usually with an f-string: f.write(f"{name},{amount}\n"). Another idiom is print(name, amount, sep=",", file=f), which adds the newline for you and is handy for quick reports. For large outputs, do not accumulate a giant string and write it once; write as you go and let Python's buffer batch the disk writes efficiently.

Common line-oriented tasks you will meet in practice:

• Counting lines (sum(1 for _ in f)) without loading the file.
• Filtering: copy only lines containing "ERROR" to another file.
• Transforming: convert every line to a record and yield it from a generator (next lecture).
• Reading the first N lines with itertools.islice(f, N).
• Reading the last few lines of a huge file efficiently by seeking from the end in binary mode, as a tail command does.

Because the file object is an iterator, you can also pass it straight to functions that accept iterables: csv.reader(f), max(f, key=len), enumerate(f, start=1) for line numbers, and so on.

Before Python 3.4, paths were plain strings glued together with os.path.join(), and every operation lived in a different module. pathlib replaced that with one object, Path, that understands what a path is. A Path knows its parts, can be joined with the / operator, and can create, read, rename, inspect and delete the thing it points to. In modern Python, pathlib is the default choice; you reach for os and shutil only for the few things Path does not do.

Building paths:

• Path("data") / "raw" / "sales.csv" — join with /; the right separator is used on every OS.
• Path.cwd() — current working directory; Path.home() — the user's home folder.
• Path(__file__).parent — the folder containing the current script. This is the correct base for files that ship with your project, because the current working directory depends on where the user ran python from.

Reading parts of a path (pure operations, no disk access): .name ("sales.csv"), .stem ("sales"), .suffix (".csv"), .suffixes, .parent, .parents[1], .parts, .anchor, .with_suffix(".json"), .with_name("other.csv"), .with_stem("sales_2026") (3.9+), .is_absolute(), .relative_to(base).

Touching the disk: .exists(), .is_file(), .is_dir(), .stat().st_size, .stat().st_mtime, .resolve() (absolute path with symlinks resolved), .mkdir(parents=True, exist_ok=True), .touch(), .rename(target), .replace(target) (overwrites), .unlink(missing_ok=True) (delete a file, 3.8+ for the flag), .rmdir() (empty directory only), .iterdir().

Convenience I/O: .read_text(encoding="utf-8"), .write_text(text, encoding="utf-8"), .read_bytes(), .write_bytes(), and .open(mode, encoding=...) which behaves exactly like the built-in open(). The convenience methods are perfect for small files; for large files use .open() and iterate.

Recent additions worth labelling: Path.walk() (3.12) mirrors os.walk; Path.from_uri() and Path.full_match() (3.13); and Python 3.14 added Path.copy(), Path.copy_into(), Path.move() and Path.move_into() so copying and moving no longer require shutil. If your code must also run on 3.13, keep using shutil for those two operations.

Finally, Path objects are accepted anywhere a path string is accepted in the standard library (open, os, shutil, json, subprocess) thanks to the os.PathLike protocol. Some third-party libraries still want a string; call str(path) for them.

Real projects rarely deal with one file. You get a folder of daily exports (sales_2026-10-01.csv, sales_2026-10-02.csv...) or a photo dump of mixed .jpg and .png files, and you need to find the right ones. Globbing is pattern matching for file names, using shell-style wildcards rather than regular expressions:

• * — any sequence of characters within one path segment (does not cross /).
• ? — exactly one character.
• [abc] and [0-9] — one character from the set.
• ** — any number of nested directories (recursive).

With pathlib, Path("data").glob("*.csv") yields matching paths in the data folder only, and Path("data").rglob("*.csv") (equivalent to glob("**/*.csv")) searches every subfolder. Both return generators, so results arrive lazily and in no guaranteed order; wrap them in sorted() when order matters, which it usually does for reproducible scripts. Python 3.12 added a case_sensitive keyword to glob/rglob, and 3.13 added recurse_symlinks.

Globbing matches names, not content, so combine it with path properties for finer filtering: p.suffix.lower() in {".jpg", ".png"} handles both upper- and lower-case extensions; p.stat().st_size > 10_000_000 finds files above 10 MB; p.stat().st_mtime finds files modified in the last 24 hours. For truly complex name rules, fall back to re on p.name.

Two related tools: Path.iterdir() lists direct children (files and folders) without any pattern, and Path.match("*.csv") tests a single path against a pattern. The older glob module (glob.glob("data/**/*.csv", recursive=True)) returns a list of strings and is still common in tutorials; prefer the pathlib version in new code.

Be careful with ** on huge trees such as node_modules or a home directory: the walk itself takes time, and a pattern typo can silently return nothing. Print the count before processing.

pathlib covers most path work, but two older modules remain essential. os exposes operating-system services: the current directory, environment variables, process IDs and low-level file operations. shutil ("shell utilities") provides the high-level copy, move and delete operations you would do with cp, mv and rm -r in a terminal.

From os you will regularly use:

• os.getcwd() and os.chdir(path) — read or change the working directory (avoid chdir in libraries; it is global state).
• os.environ — a mapping of environment variables (Section 11).
• os.makedirs(path, exist_ok=True) — create nested folders; same as Path.mkdir(parents=True, exist_ok=True).
• os.listdir(path) — names only, as strings; os.scandir(path) — faster, yields entries with cached is_file()/stat() information.
• os.remove(), os.rename(), os.replace() — delete and rename; os.replace is atomic on the same filesystem and overwrites on Windows too.
• os.walk(top) — yields (dirpath, dirnames, filenames) for every folder; os.path.getsize(), os.path.exists(), os.path.join() for code that still uses strings.

From shutil:

• shutil.copy(src, dst) copies content and permission bits; shutil.copy2() also preserves timestamps; shutil.copyfile() copies content only. If dst is a directory, the file is placed inside it.
• shutil.copytree(src, dst, dirs_exist_ok=True) — copy a whole folder (the flag, added in 3.8, allows merging into an existing target).
• shutil.move(src, dst) — move or rename; works across drives, where a plain rename would fail.
• shutil.rmtree(path) — delete a folder and everything inside it. There is no undo. Guard it with a check that the path is where you expect, and never build it from unvalidated user input.
• shutil.disk_usage(path) — total, used and free bytes; shutil.make_archive("backup", "zip", folder) — create a ZIP; shutil.which("ffmpeg") — locate an executable on PATH.

Python 3.14's Path.copy(), Path.copy_into(), Path.move() and Path.move_into() wrap the common shutil cases on a Path object; on 3.13 and earlier, shutil is still the only way.

JSON (JavaScript Object Notation) is the universal format for APIs, configuration and data exchange: a text format with objects ({}), arrays ([]), strings, numbers, true/false and null. Python's json module converts between JSON text and Python objects with four functions. Remember the naming: the ones ending in s work with strings, the others with files.

• json.dumps(obj) — Python object to JSON string.
• json.loads(text) — JSON string to Python object.
• json.dump(obj, file) — write JSON to an open file object.
• json.load(file) — read JSON from an open file object.

The type mapping is: dict ↔ object, list and tuple → array (arrays always come back as list), str ↔ string, int/float ↔ number, True/False ↔ true/false, None ↔ null. Anything else, such as datetime, Decimal, set, Path or your own classes, raises TypeError: Object of type datetime is not JSON serializable. The simplest fix is the default= argument: a function that converts unknown objects, commonly default=str. For dataclasses, dataclasses.asdict() turns them into dicts first. Dictionary keys are always converted to strings, so {1: "a"} becomes {"1": "a"} and the integer key is lost on the way back.

Useful dump/dumps options: indent=2 for human-readable files, sort_keys=True for stable diffs in Git, ensure_ascii=False so that Hindi text and ₹ are stored as real UTF-8 characters instead of \u20b9 escapes (pair it with encoding="utf-8" on the file), and separators=(",", ":") for the most compact output when sending over a network.

Reading: invalid JSON raises json.JSONDecodeError, a subclass of ValueError, with .lineno, .colno and .msg attributes that tell you exactly where the file is broken. Catch it specifically and log those details. Note that JSON does not allow trailing commas, single quotes or comments; if a hand-edited config file "looks fine" but fails to parse, those are the usual suspects.

When saving application state, write the JSON to a temporary file and then os.replace() it over the real one. If the program crashes halfway through json.dump, the old file stays intact instead of being left as a truncated, unparseable fragment.

CSV (comma-separated values) is the lingua franca of spreadsheets, banks, government portals and every "Export" button. It looks trivial, one record per line with commas between fields, but the details bite: fields that contain commas must be quoted ("Sharma, Priya"), quotes inside quoted fields are doubled, some exports use semicolons or tabs, and Excel on Windows writes \r\n line endings. Never parse CSV with line.split(","); use the csv module, which handles all of these rules.

The module offers two reader styles. csv.reader(f) yields each row as a list of strings, and you index by position (row[2]), which is fragile when a column is added. csv.DictReader(f) reads the first row as the header and yields each subsequent row as a dict keyed by column name, so your code says row["amount"] and survives column reordering. Likewise csv.writer takes lists, while csv.DictWriter(f, fieldnames=[...]) takes dicts, writes a header with .writeheader(), and refuses unexpected keys unless you pass extrasaction="ignore".

Three rules you must remember:

1. Open the file with newline="" for both reading and writing. The csv module does its own newline handling; without this, on Windows you get a blank line after every record when writing, and embedded newlines inside quoted fields are corrupted when reading. This is documented, not a bug, and it is the number one CSV question on Stack Overflow.
2. Everything is a string. row["amount"] is "1250.50", not a float; row["qty"] is "3". Convert explicitly with float(), int(), Decimal() or datetime.strptime(), and expect empty strings for missing values.
3. Pass encoding="utf-8" (or "utf-8-sig" for Excel exports with a BOM; see Section 4).

Dialects and delimiters: pass delimiter=";" or delimiter="\t" for semicolon- or tab-separated files, quoting=csv.QUOTE_MINIMAL (default) or csv.QUOTE_ALL, and csv.Sniffer().sniff(sample) to guess the format of an unknown file. For large CSVs, DictReader is itself an iterator, so a for row in reader: loop streams the file without loading it.

When should you use pandas instead? When you need joins, group-bys, pivot tables or statistics over the data: pd.read_csv() parses types automatically and handles millions of rows. For a simple ETL step, validation or a script with no other dependencies, the standard csv module is lighter and faster to start.

A database password, a Razorpay API key or an AWS secret must never be typed into a .py file. The moment that file is committed to Git, the secret is in the history forever, and public GitHub repositories are scanned by bots within minutes. The standard solution is environment variables: named values the operating system passes to a process when it starts. Your code reads them at runtime; the actual values live in the deployment environment (Docker, systemd, GitHub Actions secrets, Render, Railway, AWS), not in the repository. This is item III of the "Twelve-Factor App" methodology and every hosting platform supports it.

In Python, os.environ is a dictionary-like view of these variables. os.environ["DATABASE_URL"] raises KeyError if the variable is missing, which is often what you want for a required secret (fail fast at startup rather than deep inside a request). os.environ.get("LOG_LEVEL", "INFO") and os.getenv("LOG_LEVEL", "INFO") return a default instead. Every value is a string: convert "8000" with int() and treat "true"/"1" carefully for booleans, since bool("false") is True.

During local development, exporting ten variables by hand before every run is tedious, so the community convention is a .env file in the project root with one KEY=value per line. The third-party package python-dotenv (install with uv add python-dotenv or pip install python-dotenv) provides load_dotenv(), which reads that file and inserts its entries into os.environ without overriding variables that are already set, so production values always win. Then:

• Add .env to .gitignore immediately. Commit a .env.example with placeholder values so teammates know which variables exist.
• Keep .env for development only. In production, set real environment variables through the platform.
• Call load_dotenv() once, at the top of your entry point, before importing modules that read settings.

For larger applications, pydantic-settings (part of the Pydantic v2 family) defines a BaseSettings class with typed fields; it reads from the environment and .env, converts types, validates, and gives you a single settings object with autocomplete. FastAPI projects almost universally use this pattern.

Finally, environment variables are inherited by child processes and can appear in crash dumps and logs. Never log os.environ wholesale, and mask secrets when printing configuration.

A "large" file is one that does not comfortably fit in RAM: a 4 GB Apache log, a 20 GB CSV dump from a data warehouse, or a video upload. The techniques below keep memory usage flat and are all standard library.

1. Stream lines, never read(). for line in f: (Section 5) reads one buffered line at a time. The same applies to csv.DictReader and to json only if the file is JSON Lines (one JSON object per line, .jsonl), which is why logging systems and ML datasets prefer JSONL over one giant array. A single huge JSON array cannot be streamed by the standard json module; use the third-party ijson package for that or split the file.

2. Read binary data in chunks. For hashing, uploading or copying, loop with while chunk := f.read(1024 * 1024): (the walrus operator, Python 3.8+) and process each 1 MB block. hashlib.file_digest(f, "sha256") (3.11+) does exactly this internally. shutil.copyfileobj(src, dst, length=1024*1024) copies with a chunk size you choose.

3. Wrap the stream in generators. A function that yields parsed records from a file lets you compose filters and transformations as a pipeline, each stage holding one record. Lecture 10 covers generators in depth; you will see a preview in the snippet.

4. Avoid repeated stat and open calls in loops. Opening a file is expensive (a system call plus buffer allocation). Open once, process everything, close once. Appending to a log inside a loop of 100,000 iterations by reopening each time can be 50x slower than keeping the file open.

5. Use mmap for random access. mmap.mmap maps the file into virtual memory so you can slice it like a bytes object; the OS pages in only the parts you touch. Great for searching a large binary file or jumping to offsets.

6. Tune the buffer when it matters. open(..., buffering=1024*1024) increases the buffer for sequential throughput on fast disks; buffering=1 gives line buffering for logs you tail in real time.

7. Write atomically and in one pass. Write to a temporary file and os.replace() it into place (Section 9). For a multi-GB output, flush progress periodically and log it so operators can see it is alive.

8. Measure before optimising. Use tracemalloc or resource to check peak memory, and time.perf_counter() for timing. On a laptop SSD, streaming a 2 GB text file line by line takes about 10-20 seconds in pure Python and uses under 50 MB of RAM; readlines() on the same file needs more than 2 GB and may be killed.

For truly heavy analytics (aggregations over hundreds of millions of rows), move to pandas.read_csv(chunksize=...), Polars, DuckDB or a database, all of which stream or use multiple cores.

The techniques in this lecture are not academic; they are the daily bread of backend, data and DevOps work. Some concrete patterns you will meet in Indian product companies and services firms alike:

• ETL and reconciliation scripts. A payments team receives a daily settlement CSV from a bank or from Razorpay, reads it with csv.DictReader, matches each row against orders in the database, and writes a mismatch report as another CSV plus a JSON summary posted to Slack. newline="", utf-8-sig for the bank's Excel export and Decimal for amounts are all non-negotiable here.
• Configuration management. Services load settings from environment variables and .env (development) with defaults in code, while feature flags and static lookup tables (GST slabs, pin-code to city maps) ship as JSON files read once at startup with json.load and cached in a module-level variable.
• Log processing and incident analysis. When an outage happens, an engineer streams a multi-GB Nginx or application log line by line, filters by request ID or status code, and aggregates counts per minute. The generator pipeline from Section 12 is exactly this job.
• Data engineering pipelines. Airflow or cron jobs glob rglob("*.parquet") or rglob("*.csv.gz") in a landing folder, process new files, move them to processed/ with Path.replace, and write a manifest JSON. Idempotency is achieved by checking the manifest before reprocessing.
• Machine learning. Training scripts read JSONL datasets line by line, write checkpoints and metrics as JSON, and use Path(__file__).parent so they work from any working directory on the GPU server.
• Web backends. A FastAPI endpoint accepts an uploaded file (UploadFile), streams it to disk in chunks under a safe generated name (never the user-supplied filename), validates the magic bytes, and stores metadata as JSON in the database. Downloads use FileResponse which streams from a Path.
• CLI tools and automation. Internal tools rename thousands of files by pattern, generate reports to ~/reports/2026-10/, back up folders with shutil.make_archive, and clean temporary directories older than seven days using stat().st_mtime.
• Testing. pytest's tmp_path fixture gives every test a fresh Path in a temporary directory, so file-handling code is tested against real files without touching the project folder. The standard library's tempfile.TemporaryDirectory() does the same outside pytest.

In every one of these, the same hygiene applies: explicit encodings, context managers, pathlib, streaming for anything large, and secrets in the environment.

1. Not using with. f = open(...) followed by work and no guaranteed close() leaks file handles and loses buffered writes on a crash. Fix: always with open(...) as f:. If you need the file open across several functions, pass the file object in, or use contextlib.ExitStack.

2. Omitting encoding="utf-8". Works on your Mac, breaks on a Windows server with UnicodeEncodeError or produces mojibake. Fix: pass the encoding to every open(), read_text() and write_text() until Python 3.15 makes it the default.

3. Using "w" when you meant "a". The log file has only the last line, or the data file is empty after a second run. Fix: "a" for appending; "x" when you must never overwrite.

4. Reading a huge file with read() or readlines(). MemoryError or the OOM killer. Fix: iterate line by line or in chunks (Section 12).

5. Forgetting newline="" with the csv module. Blank lines between rows on Windows, broken multi-line fields. Fix: open(path, newline="", encoding="utf-8") for every CSV read and write.

6. Treating CSV values as numbers. row["amount"] > 1000 raises TypeError because row["amount"] is a string. Fix: convert explicitly, and handle empty strings.

7. Relative paths that depend on the working directory. The script works from the project folder but fails from cron or from VS Code's debugger with FileNotFoundError. Fix: anchor on Path(__file__).resolve().parent for bundled files, or accept the path as a command-line argument.

8. Building paths with string concatenation. folder + "/" + name or hard-coded backslashes break across operating systems and with trailing slashes. Fix: Path(folder) / name.

9. Checking exists() before open() (race condition). The file can be deleted between the check and the open; the check also doubles the system calls. Fix: just open it and catch FileNotFoundError (EAFP, "easier to ask forgiveness than permission"), as Lecture 8 taught.

10. json.dump of datetime, Decimal, set or dataclass instances. TypeError: not JSON serializable. Fix: default=str, asdict(), or convert to ISO strings before dumping; use parse_float=Decimal in json.loads for money.

11. Naming your own file csv.py, json.py or pathlib.py. Your script shadows the standard module and you get AttributeError: module 'csv' has no attribute 'DictReader'. Fix: rename the file and delete the stale __pycache__.

12. Writing while iterating the same file, or opening the same path for reading and writing simultaneously, produces corrupted or empty output. Fix: write to a temporary file and os.replace() it over the original.

13. Hard-coding secrets or committing .env. Fix: .gitignore the .env, commit .env.example, rotate any key that was ever pushed.

14. shutil.rmtree on a path built from user input. One stray .. or an empty string and you delete the wrong directory. Fix: resolve the path and assert it is inside the folder you own before deleting.

What is the difference between json.load and json.loads in Python?

json.load(f) reads JSON from an open file object; json.loads(s) parses JSON from a string that is already in memory (the trailing "s" stands for string). The same relationship holds for json.dump(obj, f) and json.dumps(obj). Use loads/dumps for API responses and messages, and load/dump for files.

Why do I get a blank line after every row when writing CSV in Python on Windows?

Because the file was opened without newline="". The csv module writes \r\n as the row terminator, and text mode on Windows translates that \n into another \r\n, producing \r\r\n. Opening with newline="" disables the translation and the module handles line endings itself, as the documentation requires.

How do I read a file line by line in Python without loading it into memory?

Iterate over the file object: with open(path, encoding="utf-8") as f: for line in f: .... Python reads buffered chunks and yields one line at a time, so memory stays constant for files of any size. Avoid f.read() and f.readlines() for large files.

Should I use pathlib or os.path in Python?

Use pathlib.Path for new code: it is object-oriented, joins paths with /, and includes reading, writing, globbing and metadata in one API. os.path is not deprecated and is still fine in older codebases, but mixing both in a new project adds confusion. Keep os for environment variables and shutil for copy, move and recursive delete on Python 3.13 and earlier.

How do I write Hindi or the rupee symbol to a file in Python without UnicodeEncodeError?

Open the file with encoding="utf-8". The error happens when Python falls back to the system encoding (often cp1252 on Windows), which cannot represent those characters. For JSON, also pass ensure_ascii=False to json.dump so the characters are stored directly rather than as \uXXXX escapes.

How do I store API keys securely in a Python project?

Put them in environment variables and read them with os.environ. For local development, keep them in a .env file loaded with python-dotenv and add .env to .gitignore. In production, set the variables through your hosting platform or a secrets manager. Never commit real keys, and rotate any key that was ever pushed to a repository.

What is the difference between text mode and binary mode in Python open()?

Text mode (the default) decodes bytes into str using an encoding and normalises newlines; you work with strings. Binary mode ("rb", "wb") gives raw bytes with no decoding or newline translation; use it for images, PDFs, archives and any non-text data. Opening binary data in text mode raises UnicodeDecodeError or silently corrupts it.

How can I process a CSV file that is bigger than my RAM in Python?

Stream it: csv.DictReader yields one row at a time, so a for row in reader loop handles a 20 GB file with a few MB of memory, as long as you aggregate as you go rather than appending rows to a list. For heavy analytics, pandas.read_csv(path, chunksize=100_000) returns DataFrame chunks, and Polars or DuckDB can query the file directly.

Q1. Why should you use the with statement when opening files?

with invokes the file object's context manager protocol: __enter__ returns the file, and __exit__ closes it when the block ends, even if an exception is raised or the function returns early. This guarantees the file handle is released and buffered data is flushed, which a bare open()/close() pair cannot guarantee without try/finally.

Q2. What does open() use as the encoding when you do not specify one, and why is that a problem?

It uses locale.getencoding(), the platform's preferred encoding: typically UTF-8 on Linux and macOS but often cp1252 on Windows. The same code therefore behaves differently across machines and can raise UnicodeEncodeError or decode text incorrectly. Python 3.7 added UTF-8 mode (PYTHONUTF8=1), 3.10 added EncodingWarning to detect the omission, and PEP 686 makes UTF-8 the default in Python 3.15.

Q3. Explain the file modes r, w, a, x, r+ and the b flag.

r reads an existing file; w creates or truncates for writing; a appends, creating if needed; x creates exclusively and raises FileExistsError if the file exists; r+ opens for reading and writing without truncating. Adding b switches to binary mode, which returns bytes and disables encoding and newline translation.

Q4. What is the difference between csv.reader and csv.DictReader?

csv.reader yields each row as a list of strings indexed by position. csv.DictReader uses the first row (or a supplied fieldnames list) as keys and yields each row as a dictionary, so code refers to columns by name and keeps working when columns are reordered. DictWriter is the writing counterpart and requires fieldnames plus a writeheader() call.

Q5. How does Python's json module handle a datetime or a Decimal, and how do you fix it?

Neither type has a JSON equivalent, so json.dumps raises TypeError: Object of type datetime is not JSON serializable. Supply default= with a function that converts unknown objects, for example default=str or one that calls .isoformat(), or convert values before dumping. For reading money precisely, pass parse_float=Decimal to json.loads.

Q6. How would you read the last 10 lines of a 10 GB log file efficiently?

Do not read from the start. Open the file in binary mode, seek() to a position near the end (f.seek(0, 2) then f.seek(size - block)), read that block, decode it, and split into lines; if fewer than 10 lines were found, step back another block and repeat. This touches only a few kilobytes regardless of file size, which is how the Unix tail command works.

Q7. What does Path(__file__).resolve().parent give you and when do you need it?

__file__ is the path of the current module; resolve() makes it absolute and resolves symlinks; .parent is the directory containing it. Use it to locate files that ship with your code (templates, data, .env) so the script works regardless of the current working directory, which differs when launched from cron, Docker, a test runner or an IDE.

Q8. What is an atomic file write and why does it matter?

Writing directly to the target file leaves it truncated or half-written if the process crashes or the disk fills midway. An atomic write writes the complete content to a temporary file in the same directory and then calls os.replace(tmp, target), which the OS performs as a single rename operation. Readers see either the complete old file or the complete new file, never a partial one. This is standard practice for configuration, state and checkpoint files.

Q9. What is the difference between shutil.copy, shutil.copy2 and shutil.copyfile?

copyfile copies only the content and requires a file path as destination. copy copies content and permission bits and accepts a directory as destination. copy2 is copy plus metadata such as modification time. In Python 3.14, Path.copy() and Path.copy_into() provide the same capability on Path objects.

Q10. How do os.environ.get, os.environ[] and load_dotenv differ?

os.environ["KEY"] raises KeyError when the variable is absent, which is appropriate for required secrets. os.environ.get("KEY", default) returns a fallback for optional settings. load_dotenv() from python-dotenv reads a .env file and populates os.environ for development, without overriding variables already set, so production environment values take precedence.

Put everything together in one small tool that a store chain might actually run. The program:

1. Reads settings from environment variables (SALES_DATA_DIR, REPORT_DIR, MIN_ORDER_INR) with sensible defaults, using a tiny built-in .env loader so it needs no third-party packages.
2. Creates realistic sample CSV files for three cities the first time it runs, including a BOM, a quoted name with a comma, an empty amount and a bad number, so the parsing code is exercised properly.
3. Globs all sales_*.csv files under the data folder with rglob, streams each with csv.DictReader using newline="" and utf-8-sig, validates rows and collects bad ones into a separate CSV instead of crashing.
4. Aggregates revenue, order count and top product per city with plain dictionaries and Decimal.
5. Writes summary.json atomically (temporary file plus os.replace), writes city_summary.csv with DictWriter, and moves processed inputs into a processed/ folder.
6. Logs progress with the logging module from Lecture 8.

Run it with python sales_report.py, then open the reports folder. Then try these extensions on your own:

• Add a --since 2026-10-01 command-line argument using argparse and filter rows by date.
• Write the per-city totals as JSON Lines (one city per line) and read them back with a generator.
• Replace the hand-written .env loader with python-dotenv and the settings dictionary with a dataclass.
• Write a pytest test that uses tmp_path to create a two-row CSV and asserts the summary numbers.

Expected console output ends with INFO processed 3 files, 12 valid rows, 2 rejected and a printed table of city totals.

• Always open files with with open(path, mode, encoding="utf-8") as f:; the context manager closes the file even when exceptions occur.
• Modes: r read, w truncate and write, a append, x create exclusively, + for read and write; add b for binary data such as images and archives.
• The default encoding is locale-dependent until Python 3.15 (PEP 686). Pass encoding="utf-8" explicitly; use utf-8-sig for Excel exports with a BOM and errors="replace" only for salvage jobs.
• Iterate files line by line (for line in f) or in chunks (f.read(1024*1024)) so memory stays flat; read() and readlines() are for small files only.
• pathlib.Path is the modern path API: join with /, inspect with .name, .stem, .suffix, .parent, create with mkdir(parents=True, exist_ok=True), read and write with read_text/write_text, find files with glob and rglob, anchor project files on Path(__file__).resolve().parent.
• Use os for the environment and low-level operations and shutil for copy2, copytree, move, rmtree and make_archive; Python 3.14 adds Path.copy and Path.move.
• JSON: json.dump/json.load for files, json.dumps/json.loads for strings; use indent, ensure_ascii=False, default=str for unsupported types, catch json.JSONDecodeError, and write atomically with a temp file plus os.replace.
• CSV: csv.DictReader and csv.DictWriter with newline=""; every value is a string, so convert with int, float or Decimal; pass delimiter for TSV and semicolon files.
• Keep secrets in environment variables; load a git-ignored .env with python-dotenv in development and use pydantic-settings for typed configuration in larger apps.
• For large files, stream with generators, hash and copy in chunks, seek from the end for tail-style reads, and measure memory with tracemalloc.

Next lecture: Iterators, Generators, Decorators & Context Managers