Text Files - Basic Operations (in Python)
- Opening a text file
- Challenges opening files
- Reading the contents of a text file
- Character encoding issues
- Strings in Python
- Lists in Python
- Example programs
- More examples
- Real-world examples
Opening a text file
Text files can be opened in one of three distinct modes:
Read mode
Only allows reading from the file.
#e.g.
f = open("data/data.csv", "r") #open in read mode
Write mode
Creates the file if it doesn’t yet exist, and allows overwriting of the file.
#e.g.
f = open("data/data.csv", "w") #open in write mode (erases any existing contents!)
Append mode
Creates the file if it doesn’t yet exist, and allows appending new content to the end of the file.
#e.g.
f = open("data/data.csv", "a") #open in append mode (adds onto the end of the file)
Challenges opening files
There are “good”, “better”, and “best” ways to open a text file.
The risks that can be mitigated by proper technique are:
- ensuring code readability.
- problems due to character encoding.
- code portability problems because file paths for Windows computers are written differently from those for UNIX/Linux/Mac computers.
- errors due to trying to open a file that does not exist.
- forgetting to close the file after opening it.
Better readability
A large problem maintaining and working on code is our inability to understand what the code is doing. Often, when we overly rely on “magic” in our code, we can’t understand what it is doing, which hinders our ability to work on it.
At the most basic, the open() function defaults to opening a file in read mode with whatever character encoding is available on the system. This is a form of magic:
f = open("data/data.csv") # defaults to read mode with whatever default character encoding
We can improve code readability by explicitly mentioning the mode:
f = open("data/data.csv", mode="r") # default character encoding... whatever that is
Avoiding character encoding issues
Computers store all data as numbers. Text files, which look like characters and symbols when viewed in a text editor, actually contain numeric data. The letters and symbols we humans see are encoded behind the scenes by the computer according to one of several standard encoding systems. Dealing with the various character encoding systems can be complicated.
At the most basic, text files have their data encoded into numbers according to either ASCII or Unicode standard encoding systems. If we read or write to a file encoded with one system using a different system, we will create all sorts of headaches and problems.
Character encoding issues can be mostly avoiding by also specifying the desired encoding system, today usually (but not always) utf-8, a specific implementation of the Unicode standard:
f = open("data/data.csv", mode="r", encoding="utf-8") # ok, but still has problems
This last example is highly readable, avoids most character encoding issues, but does not help us make the code portable among different operating systems, doesn’t deal with failure due to trying to open a file that does not exist, and does not help us avoid forgetting to close the file.
Better portability
Windows file paths use \ as the directory separator, while UNIX/Linux/Mac operating systems use /. So, for example:
-
C:\Users\foo\Documents\programming\this_example.pyis a standard Windows-compatible file path. -
/Users/foo/Documents/programming/this_example.pyis the UNIX/Linux/Mac equivalent
In general, it is best to avoid absolute paths and use relative paths to files you wish to open in code. This allows the programs to run correctly on different computers that most likely will not have the same full set of directories as the computer you’re using.
So, for example, if we assume our current working directory is the programming directory in the example above, a relative file path to a text file in a data would look something like:
-
data\data.csv- a Windows-compatible file path. -
data/data.csv- a UNIX/Linux/Mac-compatible file path.
Windows users are especially unlucky, because the backslash character \ is interpreted in most high-level programming languages (including Python) as an escape character, giving it special (unwanted) meaning in file paths.
- So, on Windows, a path like
data\data.csvwould have to have its backslash escaped to avoid Python interpreting\dindata\data.csvas an escape character, e.g.data\\data.csv. - Python’s best solution to avoid this Windows shame is to offer raw strings,
r'data\data.csvto instruct the Python interpreter to avoid interpreting the backslash as an escape character.
But portability requires the file path to work no matter what kind of operating system is being used when running the program.
Python’s standard pathlib module has useful tools for making file paths work correctly across all operating systems.
from pathlib import Path
#make an operating-system-agnostic version of the file path
filepath = Path("data/data.csv").resolve()
# open the file
f = open(filepath, mode="r", encoding="utf-8")
This solves the problem of operating system-specific file paths. Allows even Windows users to write file paths in UNIX/Linux/Mac style to avoid the Windows backslash embarassment.
But we still have to worry about errors that may arise when trying to open a file that does not actually exist, and the high chance that we will forget to close a file after opening it.
Closing files automatically
When reading from a file in Python, it frees up system resources to close the file when you’re done with it.
When writing or appending to a file in Python, it is critical that you close the file when you’re done with it, otherwise your changes will be lost.
Forgetting to close the file is a common mistake. Fortunately, Python offers a solution to automatically close the file: the with keyword.
from pathlib import Path
filepath = Path("data/data.csv").resolve()
#the 'with' statement closes the file for us automatically
with open(filepath, mode="r", encoding="utf-8") as f:
# do something with the data in the file
Using the with keyword, Python close the file for you as soon as the indented block finishes.
Avoiding errors due to files that don’t exist
There are two or three basic strategies to avoid errors due to trying to open files that do not actually exist.
import os
from pathlib import Path
# create an operating system-agnostic file path
filepath = Path("data/data.csv").resolve()
# only try to open the file if it actually exists
if os.path.isfile(filepath):
with open(filepath, mode="r", encoding="utf-8") as f:
# do something with the data in the file
else:
print("Sorry, we could not find that file.")
This version is highly readable, uses operating-system agnostic file paths for greater code portability, automatically closes the file after we’re done with it, and avoids errors if we try to open a non-existent file.
Another technique instead of if/else logic is to use the try/except error handlers in Python, e.g.
from pathlib import Path
# create an operating system-agnostic file path
filepath = Path("data/data.csv").resolve()
#try to open the file, and handle the error if it isn't there
try:
# if an error is encountered with a "try" block, the interpreter will
# immediately jump into the "except" block
with open(filepath, mode="r", encoding="utf-8") as f:
# do something with the data in the file
except FileNotFoundError:
#this block runs only if the file could not be found
print("Sorry, we could not find that file.")
Reading the contents of a text file
Read the entire file in one fell swoop:
the_full_monty = f.read() #read entire file
Read just a single line at a time:
line1 = f.readline() #get the first line
line2 = f.readline() #get the second line
#etc...
Loop through all lines:
for line in f:
#do something
Removing line break characters
Reading lines from the text file will always include the line break character(s) (\n, \r, or \r\n) at the end of each line string.
Usually you want to remove line breaks using the strip() string function, e.g.
line = line.strip() # remove all leading or trailing whitespace
or
line = line.rstrip() # remove only trailing whitespace
Character encoding issues
The problem
Python's open() function defaults to using whatever the default encoding scheme is on the computer you're using. Often that's not good enough. If you don't specify the correct encoding scheme, your program may crash if it encounters a character that is not found in the encoding scheme it is using.
One solution
The easiest solution is to tell Python exactly which encoding scheme to use when you open a file. The built-in open() function accepts an encoding argument for exactly this purpose. utf-8 is a good default choice for most modern text files.
#the following example opens the file with utf-8 encoding
f = open("data.csv", mode="r", encoding="utf-8")
You’ll also see this work inside a with statement:
with open("data.csv", mode="r", encoding="utf-8") as f:
contents = f.read()
Strings in Python
Python has lots of useful String-related functions.
String-related functions that return a String
-
.upper()- returns an uppercase version of the string -
.lower()- returns a lowercase version of the string -
.title()- returns a version of the string with the first letter of every word capitalized. -
.capitalize()- returns a version of the string with the first letter capitalized -
.strip()- returns a version of the string with any leading whitespace removed -
.rstrip()- returns a version of the string with any trailing whitespace removed -
.join(_some_list_)- returns a string that has all items from the list argument separated using the string as separator
String-related functions that return an Integer
-
.find(x)- returns the index position in the String at which x is found
String-related functions that return a Boolean
-
.isupper()- returns boolean True if the string is uppercase, False otherwise -
.islower()- returns boolean True if the string is lowercase, False otherwise -
.isnumeric()- returns a boolean True if the string represents a number, False otherwise
String-related functions that return a List
-
.split(_some_delimiter_)- returns a list based on the contents of the string, by splitting the string everywhere it finds the delimiter specified as the argument
Lists in Python
Useful functions
List functions that modify an existing List:
-
.append(some_value)- adds the value as a new element at the end of the list
Example programs
The following examples do not use our best version of opening text files in order to reduce the amount of code you have to read to understand the focus of each example. In a real program, we would open files the way we recommend above.
Data files
The following programs assume you have two text files in the a subdirectory named data:
mydata.txt:
The first line in the text file
The second line in the text file
sloppy_data.csv:
1,Peace Food,Manhattan,New York
2,Bareburger,manhattan,new York
3,Why not,manhattan, New york
4,five guys, Manhattan, New York
5,katz DELI,manhattan,new york
Grab entire contents of a text file
Read the entire contents of the file using .read(), and print them out.
#open a text file in read mode
f = open("data/mydata.txt", "r")
#pull all the data out of the file into a variable
the_text_in_the_file = f.read()
#print out the contents of the file
print(the_text_in_the_file)
Loop through each line of a text file and chop off line breaks
Remove the line break from the end of each file, and then print it out.
#open a text file in read mode
f = open("data/mydata.txt", "r")
#loop through each line in the file, one by one
for line in f:
#remove the line break from the end of the string
line = line.rstrip()
#print out the line
print(line)
Loop through all words in a text file
Loop through each word in the file (assuming words are separated by spaces), and print it out.
#open a text file in read mode
f = open("data/mydata.txt", "r")
#get the full text from the file
the_full_text = f.read()
#split the text into a List of words by space delimiters
words = the_full_text.split()
#loop through each word and analyze it
for word in words:
#print for debugging
print(word)
Count the occurrences of a given word in a text file
Loop through each word in the file (assuming words are separated by spaces), and print out how many times a search term is found. For simplicity, this program does not account for punctuation, which would cause problems in this code. (See the Word frequency counter example below for one way to strip punctuation.)
#open a text file in read mode
f = open("data/mydata.txt", "r")
#get the full text from the file
the_full_text = f.read()
#split the text into a List of words by space delimiters
words = the_full_text.split()
#what word are we looking for?
search_term = "Ronkonkoma"
#keep a counter of how many times we found the word that we're looking for
counter = 0
#loop through each word and analyze it
for word in words:
#check whether the word matches our search_term (case-insensitive)
if word.lower() == search_term.lower():
#if so, increment the counter
counter = counter + 1
#print out the result
print("We found the word", search_term, counter, "times.")
Fix sloppy capitalization in a CSV data file
Loop through each line in a CSV text file, loop through each value in the line and modify it in some way (in this example, we simply capitalize a word we are searching for). Store the modified values in a two-dimensional list. Loop through this two-dimensional list and output the modified values to a file.
#####################################################
#PART 1 - SCRAPE THE DATA FROM A CSV FILE
#####################################################
#open a file in read mode
f = open("sloppy_data.csv", "r")
clean_data = [] #this will store the cleaned up data from all lines
#loop through each line in the file
for line in f:
new_line = [] #this will store the cleaned up list of values in this one line
line = line.strip() #get rid of line break on every line
data = line.split(',') #split by commas to get a list of values
#loop through every value in this list
for thing in data:
#if the value is 'ronkonkoma', convert it to uppercase 'RONKONKOMA'
if thing == "ronkonkoma":
thing = thing.upper()
#append this value to the "clean" list of values in this line
new_line.append(thing)
#append this cleaned up data to the list
clean_data.append(new_line)
#close the file
f.close()
#####################################################
#PART 2 - WRITE THE CLEANED UP DATA TO A NEW CSV FILE
#####################################################
#open a file in write mode
f = open("sloppy_data_fixed.csv", "w")
#loop through each list of cleaned up data... each list represents one line
for line_as_list in clean_data:
#convert this line (currently stored a list) to a string with comma-separated values
line_as_string = ",".join(line_as_list)
#write this line's data to a file, along with a line break
f.write(line_as_string + "\n")
#close the file
f.close()
Parsing CSV files properly - the csv module
The example above splits each line on commas by hand. That works for simple data, but it breaks as soon as a value itself contains a comma - for example a restaurant named "Joe\'s, Inc." or an address like "123 Main St, Apt 4". Real CSV files surround such values in quotes, and splitting on every comma would chop them in the wrong place.
Python’s built-in csv module handles all of these tricky cases for you. Instead of .split(","), it hands you each row already broken into a clean list of values.
import csv
#open the file and let csv break each line into a list of values for us
#(the newline="" argument is recommended by Python's docs when using the csv module)
with open("sloppy_data.csv", "r", newline="") as f:
#create a reader object that knows how to parse CSV
reader = csv.reader(f)
#loop through each row - each row is already a list, no .split() needed
for row in reader:
print(row) #e.g. ['1', 'Peace Food', 'Manhattan', 'New York']
Writing CSV data is just as easy with a writer object, which correctly adds quotes around any value that needs them:
import csv
#some data to save, where each inner list is one row
data = [
["1", "Peace Food", "Manhattan", "New York"],
["2", "Joe\'s, Inc.", "Manhattan", "New York"] #note the comma inside the name!
]
#open a file for writing
with open("clean_data.csv", "w", newline="") as f:
#create a writer object
writer = csv.writer(f)
#write all the rows at once (or use writer.writerow(row) one at a time)
writer.writerows(data)
More examples
Data file
These programs assume you have a text file named “data.csv” in a data subdirectory from where your Python program exists. The data.csv file stores student grades as comma separated values (CSV format). This type of format is commonly used by spreadsheet programs like Microsoft Excel.
Example data file
This is our example "data.csv" file in CSV format.
Adam,85
Mark,22
Erica,100
Kaitlin,98
Spencer,69
John,88
Wilson,95
Spencer,49
Faith,89
Andrew,90
Celia,90
Mike,90
Writing to a file
This program allows users to append new grades to the data.txt file.
Note that this program exhibits a bug if the user enters "exit" in response to the first question.
#this program allows us to append student grades to an existing data.txt file
#(for a cleaner approach that closes the file automatically, see the "with" statement above)
#flag to indicate whether we opened the file or not
is_file_open = False
try:
my_file = open("data/data.csv", mode="a")
is_file_open = True
except FileNotFoundError:
print("oops, sorry, didn't find the file... my bad")
except IOError:
print("There was an IO error")
except:
print("Sorry, I don't know what went wrong")
#if the file is open, start writing/appending to it
if is_file_open:
student_name = input("Please enter a student name:")
student_grade = input("Please enter this student's grade:")
while student_name != "exit" and student_grade != "exit":
if student_name == "exit":
break
my_file.write(student_name + "," + student_grade + "\n")
student_name = input("Please enter a student name:")
if student_name == "exit":
break
student_grade = input("Please enter this student's grade:")
if student_grade == "exit":
break
my_file.close()
Reading from a file
This program reads the grades from the data.txt file and outputs the average grade for all students found in the file.
#this program opens a file named "data.txt" that holds a student grade in each line in the format <name>,<grade>
#the program outputs the average grade of all students
#flag to keep track of whether we opened the file or not
is_file_open = False
#open up a text file in read-only mode
try:
my_file = open("data/data.csv", "r")
is_file_open = True #set flag to true now that we've opened the file
except FileNotFoundError:
print("oops, sorry, didn't find the file... my bad")
except IOError:
print("There was an IO error")
except:
print("Sorry, I don't know what went wrong")
#if we've opened the file successfully, read from it
if is_file_open:
#read file all at once
#data_from_file = my_file.read() #read entire file and store in variable
#print(data_from_file) #print out entire file
#keep track of the sum of all grades so we can get the average later
running_total = 0
#keep track of how many students we find in the text file
num_students = 0
#read one line from a file
line = my_file.readline()
#make sure line has something in it
while line != "":
#increment the student counter
num_students = num_students + 1
#print it out
line = line.rstrip("\n") #strip off trailing line break
#print(line)
#break up the string along the commas
data = line.split(",")
#print(data)
#add the current student's grade to the running total
running_total = running_total + int(data[1])
#read another line
line = my_file.readline()
#calculate the average grade
average = running_total/num_students
#format it to look nice as a string
nice_looking_average = format(average, ".2f")
#print out the average grade
print(nice_looking_average)
Real-world examples
Reading and writing text files is one of the most common things real programs do - logs, configuration files, exported spreadsheets, and saved data are all just text files. The following examples show some everyday tasks. They use the with statement so the files close automatically, and they tie together strings, lists, and dictionaries.
Searching a real-world data file
Governments and organizations publish huge amounts of data as downloadable CSV files. This program reads the NYC Open Data Wi-Fi hotspot dataset and lets the user search for free public Wi-Fi networks in a particular borough.
It pulls together the robust file-opening pattern from above: an os-agnostic pathlib path, utf-8 encoding, the with statement, and a try/except guard against a missing file.
Example data file
wifi.csv (the real file has thousands of rows; here are the header row and one data row):
"OID","Hotspot Dataset Object ID","Public Space (Open Space) Name","Public Space (Open Space) Proximity","Borough Name", ... ,"Provider","SSID","Latitude","Longitude","Published Date"
"1","1400","Abe Stark Skating Rink","Inside","Brooklyn", ... ,"ALTICEUSA","GuestWiFi","40.57278","-73.98563","05/30/2019"
from pathlib import Path
#build an os-agnostic path to the data file
filepath = Path("data/wifi.csv").resolve()
#protect against the file not being found
try:
#open the file in a way that closes it automatically when we're done
with open(filepath, mode="r", encoding="utf-8") as f:
#ask the user which borough they want to search
user_borough = input("Which borough do you want to search? ")
user_borough = user_borough.title() #match the capitalization used in the data
#look at each line - each line is one Wi-Fi hotspot
for line in f:
line = line.strip() #remove the line break at the end
values = line.split(",") #split the line into a list of values
#the borough is the 5th value (index 4); the data wraps values in quotes
line_borough = values[4].replace('"', "")
#if this hotspot is in the borough the user asked about, print it out
if line_borough == user_borough:
space = values[2].replace('"', "") #name of the park/space
network_name = values[9].replace('"', "") #the Wi-Fi network name (SSID)
print(f"{network_name} in {space}, {line_borough}")
except FileNotFoundError:
#this block runs only if the file could not be found
print("Sorry, we could not find that file.")
Note: this data wraps every value in quotes, so we strip them off by hand with
.replace('"', ""). Thecsvmodule shown earlier handles quoting for you automatically and is the better tool for real CSV files.
Organizing a file-reading program into functions
As a program grows, cramming everything into one long block of code becomes hard to read and hard to debug. A common next step is to break the work into small functions that each do one job, and then a main() function that ties them together. Here is the exact same Wi-Fi search as above, reorganized this way.
from pathlib import Path
def get_file_path(filename):
"""Return an os-agnostic version of a file path."""
return Path(filename)
def ask_for_borough():
"""Ask the user which borough they want to search and return it."""
borough = input("Which borough do you want to search? ")
return borough.title() #match the capitalization used in the data
def find_matches(filepath, user_borough):
"""Open the data file and print every Wi-Fi hotspot in the chosen borough."""
#open the file safely, closing it automatically when done
with open(filepath, mode="r", encoding="utf-8") as f:
for line in f:
line = line.strip()
values = line.split(",")
line_borough = values[4].replace('"', "")
if line_borough == user_borough:
space = values[2].replace('"', "")
network_name = values[9].replace('"', "")
print(f"{network_name} in {space}, {line_borough}")
def main():
"""The main logic of the program, tying the helper functions together."""
filepath = get_file_path("data/wifi.csv")
user_borough = ask_for_borough()
#guard against the file not being found
try:
find_matches(filepath, user_borough)
except FileNotFoundError:
print("Sorry, we could not find that file.")
# this means "only run main() if this file is run directly,
#not when it is imported into another program"
if __name__ == "__main__":
main()
Both versions do exactly the same thing - but the second one is easier to read, because each function has a clear name and a single responsibility. The if __name__ == "__main__": line at the bottom is a common Python convention that lets your file be run as a program and be imported by other programs without automatically running main().
Word frequency counter
This program reads a text file and counts how many times each word appears, using a dictionary where each key is a word and each value is its count. It strips punctuation and ignores case so that “The”, “the”, and “the.” all count as the same word.
Example data file
poem.txt:
The owl and the pussycat went to sea
In a beautiful pea green boat
They took some honey and plenty of money
Wrapped up in a five pound note
import string #the string module gives us a handy list of punctuation characters
#a dictionary to hold each word and how many times we've seen it
word_counts = {}
#open the file and read it line by line
with open("poem.txt", "r") as f:
for line in f:
#split the line into words by whitespace
words = line.split()
#look at each word one at a time
for word in words:
#lowercase the word and strip punctuation from its edges
#e.g. "Sea." becomes "sea"
word = word.lower().strip(string.punctuation)
#skip empty strings (e.g. if a "word" was just punctuation)
if word == "":
continue
#if we've seen this word before, add 1 to its count
#otherwise, start its count at 1
if word in word_counts:
word_counts[word] = word_counts[word] + 1
else:
word_counts[word] = 1
#sort the words from most frequent to least frequent
#(sorted() with a key tells Python to sort by the count, reverse=True means highest first)
sorted_words = sorted(word_counts.items(), key=lambda pair: pair[1], reverse=True)
#print out the top 5 most common words
print("The 5 most common words are:")
for word, count in sorted_words[:5]:
print(word, "appears", count, "times")
Log file analysis
Programs and servers often record what they’re doing in a log file, where each line is one event. A very common task is to scan a log for problems. This program counts how many lines are errors and prints each one.
Example data file
app.log:
2026-06-11 09:58:01 INFO server started
2026-06-11 10:02:13 ERROR could not connect to database
2026-06-11 10:02:14 INFO retrying connection
2026-06-11 10:02:20 ERROR could not connect to database
2026-06-11 10:05:00 WARNING disk space is low
2026-06-11 10:07:42 INFO request handled successfully
#keep track of how many error lines we find
error_count = 0
#open the log file and look at each line
with open("app.log", "r") as f:
for line in f:
#remove the trailing line break
line = line.rstrip()
#a line is an error if the word "ERROR" appears in it
if "ERROR" in line:
error_count = error_count + 1
print("found an error:", line)
#print a summary
print("-----")
print("Total errors found:", error_count)
Simple to-do list app
This program keeps a to-do list in a text file. It shows a menu that lets the user view the current list (reading the file) or add a new task (appending to the file). It exercises read mode and append mode in one relatable program. The list is saved between runs because it lives in a file.
#the name of the file where we store the to-do items
FILENAME = "todo.txt"
#keep showing the menu until the user chooses to quit
while True:
print()
print("What would you like to do?")
print(" 1 - view my to-do list")
print(" 2 - add a new task")
print(" 3 - quit")
choice = input("Enter 1, 2, or 3: ")
if choice == "1":
#VIEW: read the file and print each task
#use try/except in case the file doesn't exist yet
try:
with open(FILENAME, "r") as f:
print("Your to-do list:")
for line in f:
print(" -", line.rstrip())
except FileNotFoundError:
print("Your to-do list is empty.")
elif choice == "2":
#ADD: append a new task to the end of the file
task = input("Enter the new task: ")
with open(FILENAME, "a") as f:
f.write(task + "\n")
print("Added!")
elif choice == "3":
#QUIT: break out of the while loop
print("Goodbye!")
break
else:
print("Sorry, I didn't understand that choice.")
Reading a configuration file
Many programs store their settings in a simple key=value text file. This program reads such a file into a dictionary, skipping blank lines and comment lines that start with #. This is a good example of defensive parsing - quietly ignoring lines that don’t fit the expected format instead of crashing.
Example data file
settings.ini:
# this is the configuration for my program
username=alice
theme=dark
# the line above is intentionally blank
font_size=14
#a dictionary to hold all of our settings
settings = {}
#open the config file and read it line by line
with open("settings.ini", "r") as f:
for line in f:
#remove leading/trailing whitespace
line = line.strip()
#skip blank lines and comment lines (those starting with #)
if line == "" or line.startswith("#"):
continue
#split into a key and a value at the FIRST "=" only
#(the 1 means "split at most once", so a value containing "=" stays intact)
key, value = line.split("=", 1)
#store the cleaned-up key and value in our dictionary
settings[key.strip()] = value.strip()
#now we can use the settings like any other dictionary
print("All settings:", settings)
print("The username is:", settings["username"])
print("The theme is:", settings["theme"])