Server-side coding in Python

From Knowledge Kitchen
Jump to navigation Jump to search


Python can be used as a server-side programming language. However it is not explicitly designed for this purpose. As a result, the amount of code necessary to allow Python to function well as a modern server-side language can be a burden.

Use of specialized libraries or frameworks, such as Django or Flask, designed to assist coding server-side tasks in Python can simplify the code significantly. However, the examples in this document do not depend on any such frameworks, and use only modules / libraries included in the standard Python distribution.

Debugging

Debugging programs running on the server in a client-server environment like the Web can be challenging, since there are many variables involved.

Barebones example

The following shows how to use Python to output a simple HTML document.

index.py

 1 #!/usr/bin/env python3
 2 
 3 #enable debugging... any errors will be output as HTML so they show up clearly in the web browser
 4 import cgi, cgitb
 5 cgitb.enable()
 6 
 7 #make sure all output from the print statement in Python is utf-8 encoded
 8 import sys, codecs
 9 sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
10 
11 #determine the current version of python on this server using the sys library
12 python_version = str(sys.version[0]) + "." + str(sys.version[2])
13 
14 #send raw HTTP headers to browser
15 #NOTE: this must be the first thing sent to the web browser
16 print("Content-type: text/html;charset=utf-8")
17 print("\n\n")  #two line breaks indicates the end of the headers
18 
19 #send body content to browser
20 #NOTE: this must follow two line breaks separating it from the headers
21 print('''
22 <!DOCTYPE html>
23 <html>
24     <head>
25         <meta charset="utf-8" />
26         <title>It's snowing</title>
27         <link rel="stylesheet" type="text/css" href="css/main.css" />
28     </head>
29     <body>
30         <div class="container">
31             <h1>Python works</h1>
32             <p>This script was executed by the Python {version} interpreter on the web server.  The output was sent to the web browser.</p>
33             <p>
34                 Running Python from the web is what's called a CGI script.  See <a href="https://docs.python.org/3.5/library/cgi.html">for info on Python's support for CGI</a>.
35             </p>
36             <p>
37                 A popular framework for simplifying server-side Python for web applications is <a href="http://flask.pocoo.org/">Flask</a>.  We are not using that here.
38             </p>
39         </div>
40     </body>
41 </html>
42 '''.format(version=python_version))

Barebones example using an HTML template

Use of a templating module can help separate the HTML from the Python code to make writing and maintaining each sort of code easier. Templating engines also allow a single page template to be reused for many actual pages, without rewriting the client-side code (HTML, CSS, Javascript) for each page that uses the template.

Note that the string templating functionality included in the Python standard distribution used here is quite limited. More sophisticated templating libraries exist as add-ons for Python, but have been avoided in this document.

index.py

 1 #!/usr/bin/env python3
 2 
 3 #enable debugging... any errors will be output as HTML so they show up clearly in the web browser
 4 import cgi, cgitb
 5 cgitb.enable()
 6 
 7 #make sure all output from the print statement in Python is utf-8 encoded
 8 import sys, codecs
 9 sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
10 
11 #import the string templating library to do allow us to separate the HTML code from the Python code
12 from string import Template
13 
14 #determine the current version of python on this server
15 python_version = str(sys.version[0]) + "." + str(sys.version[2])
16 
17 #open the template file and extract the contents
18 f = open('page_template.txt')
19 template = Template(f.read())
20 
21 #set up the variables to plug into the template
22 data = {
23  'title': "It's snowing",
24  'heading': "Python works!",
25  'python_version': python_version,
26  'css_url': 'css/main.css',
27  'link_url': "https://wiki.python.org/moin/Templating"
28 }
29 
30 #plug the variables into the template to generate the complete HTML
31 html = template.substitute(data)
32 
33 #output filled-in template
34 print(html)

page_template.txt

 1 Content-type: text/html;charset=utf-8
 2 
 3 <!doctype html>
 4 <html lang="en">
 5     <head>
 6         <title>$title</title>
 7         <meta charset="utf-8">
 8         <link rel="stylesheet" type="text/css" href="$css_url" />
 9     </head>
10     <body>
11         <div class="container">
12 
13             <h1>$heading</h1>
14             <p>
15                 This script was executed by the Python $python_version  interpreter on the web server.  The output was sent to the web  browser.
16             </p>
17             <p>
18                 Using Templates is a common way to keep Python code separate from  HTML code.  There are many sophisticated "templating engines"  that can be added to python, although this script uses a very  basic one incldued in the Python standard library.  See <a  href="$link_url">for more info on Templating in Python</a>.
19             </p>
20 
21         </div><!-- //.container -->
22     </body>
23 </html>

Connecting to a MySQL database

The pymysql library can be used to connect to a MySQL database. The following two examples show how to perform a Read on a MySQL table and output a well-formatted HTML document with the results. The only difference between the two examples are in the usage of templates - the first example does not use an HTML template, while the second example does.

Reading from a MySQL table

Example of connecting to a MySQL database from Python, performing a Read operation, and outputting a well-formatted HTML document as a result.

index.py

 1 #!/usr/bin/env python3
 2 
 3 #enable debugging... any errors will be output as HTML so they show up clearly in the web browser
 4 import cgi, cgitb
 5 cgitb.enable()
 6 
 7 #make sure all output from the print statement in Python is utf-8 encoded
 8 import sys, codecs
 9 sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
10 
11 #connect to database
12 import pymysql
13 cxn = pymysql.connect(
14     host="your_host_name", 
15     user="your_username", 
16     passwd="your_password", 
17     db="your_database_name", 
18     charset='utf8mb4',
19     cursorclass=pymysql.cursors.DictCursor) 
20 
21 #get a 'cursor'... a pointer to the database
22 cur = cxn.cursor()
23 
24 #execute a query
25 cur.execute("SELECT * FROM favorite_viking_metal_bands ORDER BY origin ASC")
26 
27 #store the results
28 rows = cur.fetchall()
29 
30 #send raw HTTP headers to browser
31 print("Content-type: text/html;charset=utf-8")
32 print("\n\n")
33 
34 #print the top of the document
35 print('''
36 <!DOCTYPE html>
37 <html>
38     <head>
39         <meta charset="utf-8" />
40         <title>MySQL Example Query from Python</title>
41         <link rel="stylesheet" type="text/css" href="css/main.css" />
42     </head>
43     <body>
44         <div class="container">
45             <h1>Viking Metal Bands</h1>
46 ''')
47 
48 #loop through the query results and print them out as well-formmated HTML
49 for row in rows:
50     print('''
51              <article>
52                 <h1>{name}</h1>
53                 <p>
54                     {country}
55                     <br />
56                     ({year})
57                 </p>
58              </article>
59             '''.format(name=row['band'], country=row['origin'], year=row['formed']))
60 
61 #print out the bottom of the document
62 print('''
63             <!-- clearfix -->
64             <div class="clear"></div>
65         </div>
66     </body>
67 </html>
68 ''')

Reading from a MySQL table using HTML templates

The same content as the example above, but with the HTML code separated from the Python code into three distinct template files - one for the Python code, another for the HTML template for a single article, and another for the HTML template for an entire page.

index.py:

 1 #!/usr/bin/env python3
 2 
 3 #enable debugging... any errors will be output as HTML so they show up clearly in the web browser
 4 import cgi, cgitb
 5 cgitb.enable()
 6 
 7 #make sure all output from the print statement in Python is utf-8 encoded
 8 import sys, codecs
 9 sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
10 
11 #import the string templating library to do allow us to separate the HTML code from the Python code
12 from string import Template
13 
14 #connect to database
15 import pymysql
16 cxn = pymysql.connect(
17     host="your_host_name", 
18     user="your_username", 
19     passwd="your_password", 
20     db="your_database_name", 
21     charset='utf8mb4',
22     cursorclass=pymysql.cursors.DictCursor) 
23 
24 #get a 'cursor'... a pointer to the database
25 cur = cxn.cursor()
26 
27 #execute a query
28 cur.execute("SELECT * FROM favorite_viking_metal_bands ORDER BY origin ASC")
29 
30 #store the results
31 rows = cur.fetchall()
32 
33 #open the article template
34 f = open('article_template.txt')
35 article_template = Template(f.read())
36 f.close()
37 
38 #open the page template
39 f = open('page_template.txt')
40 page_template = Template(f.read())
41 f.close()
42 
43 #generate the html for all articles
44 
45 #a blank list that will be filled in with the html for all articles
46 articles_html = []
47 
48 for row in rows:
49     #put together data foe this article
50     article_data = {
51         "name": row['band'],
52         "country": row['origin'],
53         "year": row['formed']
54     }
55     article_html = article_template.substitute(article_data)
56     articles_html.append(article_html)
57 
58 
59 #set up the variables to plug into the template
60 page_data = {
61     'title': "Viking Meral Bands",
62     'heading': "List of Viking Metal Bands",
63     'css_url': 'css/main.css',
64     'articles': "\n".join(articles_html)
65 }
66 
67 #plug the variables into the template to generate the complete HTML
68 page_html = page_template.substitute(page_data)
69 
70 #output filled-in template
71 print(page_html)

article_template.txt

1              <article>
2                 <h1>$name</h1>
3                 <p>$country</p>
4                 <p>$year</p>
5              </article>

page_template.txt

 1 Content-type: text/html;charset=utf-8
 2 
 3 
 4 <!DOCTYPE html>
 5 <html>
 6     <head>
 7         <meta charset="utf-8" />
 8         <title>$title</title>
 9         <link rel="stylesheet" type="text/css" href="$css_url" />
10     </head>
11     <body>
12         <div class="container">
13             <h1>$heading</h1>
14 
15 
16             $articles
17 
18 
19             <div class="clear"></div>
20         </div>
21     </body>
22 </html>
23 ''')

Form example

An example of how a simple HTML form might submit data to the server, where Python code captures that data and processes it.

index.html

 1 <!DOCTYPE html>
 2 <html lang="en">
 3     <head>
 4         <meta charset="utf-8" />
 5         <title>Form Example</title>
 6         <link rel="stylesheet" type="text/css" href="css/main.css" />
 7     </head>
 8     <body>
 9         <div class="container">
10 
11             <h1>Rate This Example!</h1>
12 
13             <p>How would you rate this example?</p>
14 
15             <form action="processor.py" method="GET">
16                 <div class="row">
17                     <label for="your_name">Your name</label>
18                     <input type="text" id="your_name" name="your_name" />
19                     <div class="clearfix"></div>
20                 </div>
21 
22                 <div class="row">
23                     <label for="your_rating">Your rating of this example</label>
24                     <select id="your_rating" name="your_rating">
25                         <option value="">- select rating -</option>
26                         <option value="excellent">Excellent!</option>
27                         <option value="good">Good enough</option>
28                         <option value="okay">Okay</option>
29                         <option value="poor">Poor quality</option>
30                         <option value="awful">Awful!</option>
31                     </select>
32                     <div class="clearfix"></div>
33                 </div>
34 
35                 <div class="row">
36                     <label for='submit'>Ready?</label>
37                     <input type="submit" id='submit' value="Rate it!" />
38                     <div class="clearfix"></div>
39                 </div>
40 
41             </form>
42 
43         </div><!-- //.container -->
44     </body>
45 </html>

processor.py

 1 #!/usr/bin/env python3
 2 
 3 #enable debugging... any errors will be output as HTML so they show up clearly in the web browser
 4 import cgi, cgitb
 5 cgitb.enable()
 6 
 7 #make sure all output from the print statement in Python is utf-8 encoded
 8 import sys, codecs
 9 sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
10 
11 #import the string templating library to do allow us to separate the HTML code from the Python code
12 from string import Template
13 
14 # use the CGI library to get any form data sent to the server from the client
15 form = cgi.FieldStorage() 
16 
17 # validate that data has been submitted via the form...
18 if "your_name" not in form or "your_rating" not in form:
19     # if the user did not enter form data, send the browser back to the form page
20     # we do this by sending a special HTTP location header to the browser
21     print("Location: ./index.html\n\n")
22     #exit now!
23     sys.exit()
24 
25 #assuming the user entered form data, get the form data into simple variables
26 your_name = form.getfirst('your_name')
27 your_rating = form.getfirst('your_rating')
28 
29 #load up the template
30 #open the template file and extract the contents
31 f = open('page_template.txt')
32 template = Template(f.read())
33 
34 #set up the variables to plug into the template
35 data = {
36     'name': your_name,
37     'rating': your_rating
38 }
39 
40 #plug the variables into the template to generate the complete HTML
41 html = template.substitute(data)
42 
43 #output filled-in template
44 print(html)

page_template.txt

 1 Content-type: text/html;charset=utf-8
 2 
 3 <!doctype html>
 4 <html lang="en">
 5     <head>
 6         <title>Thank You!</title>
 7         <meta charset="utf-8">
 8         <link rel="stylesheet" type="text/css" href="css/main.css" />
 9     </head>
10     <body>
11         <div class="container">
12 
13             <h1>Thanks $name for your feedback!</h1>
14             <p>
15                 Hey, <strong>$name</strong>!  
16                 Thanks for the <strong>$rating</strong> rating!
17             </p>
18             
19         </div><!-- //.container -->
20     </body>
21 </html>

Scraping data from web pages

This example shows how to use the add-on Beautiful Soup module to scrape web pages. In this case, the code scrapes the names of Rolling Stone's top 100 songs from this web site.

index.py

 1 #!/usr/bin/env python3
 2 
 3 #enable debugging... any errors will be output as HTML so they show up clearly in the web browser
 4 import cgi, cgitb
 5 cgitb.enable()
 6 
 7 #make sure all output from the print statement in Python is utf-8 encoded
 8 import sys, codecs
 9 sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
10 
11 #import urllib.request and beautiful soup to make web requests and parse them
12 import urllib.request
13 from bs4 import BeautifulSoup
14 
15 #get the HTML code of the web page
16 url = "http://lyrics.wikia.com/LyricWiki:Lists/Rolling_Stone:_The_500_Greatest_Songs_of_All_Time"
17 html = urllib.request.urlopen(url)
18 
19 #parse this page
20 soup = BeautifulSoup(html)
21 
22 #now find and parse just the div.WikiaArticle element
23 div = soup.find_all('div', {"class": "WikiaArticle"})
24 soup = BeautifulSoup(str(div))
25 
26 #now find and parse just the ol tag element in here
27 ol = soup.find_all('ol')
28 soup = BeautifulSoup(str(ol))
29 
30 #now find just the a elements in here
31 links = soup.find_all("b")
32 
33 # output the top of the document
34 print('''Content-type: text/html
35 
36 <!doctype html>
37 <html lang='en'>
38     <head>
39         <title>The 500 Greatest Songs of All Time, According to Rolling Stone</title>
40         <meta charset="utf-8" />
41         <link rel="stylesheet" type="text/css" href="css/main.css" />
42     </head>
43     <body>
44         <div class="container">
45 
46             <h1>The 500 Greatest Songs of All Time</h1>
47             <h2>
48                 According to 
49                 <a href='{url}'>Rolling Stone</a>
50             </h2>
51 
52 '''.format(url=url))
53 
54 # output each of the songs
55 for link in links:
56     #get the text inside each link in the original html
57     print('''
58 
59             <article>
60                 <h2>{song_title}</h2>
61             </article>
62 
63         '''.format(song_title=link.string))
64 
65 # output the bottom of the document
66 print('''
67 
68             <!-- clearfix -->
69             <div class="clear"></div>
70 
71         </div><!-- //.container -->
72     </body>
73 </html>
74 
75 ''')

Connecting to a MongoDB database

The following examples use the pymongo library for Python to connect to a MongoDB database and perform operations on a collection.

Reading from a MongoDB collection

index.py

 1 #!/usr/bin/env python3
 2 
 3 #enable debugging... any errors will be output as HTML so they show up clearly in the web browser
 4 import cgi, cgitb
 5 cgitb.enable()
 6 
 7 #make sure all output from the print statement in Python is utf-8 encoded
 8 import sys, codecs
 9 sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
10 
11 #import mongodb 
12 import pymongo
13 
14 #connect to db server
15 client = pymongo.MongoClient('mongodb://your_username:your_password@your_host_name/your_database_name')
16 
17 #send raw HTTP response headers to web browser
18 #NOTE: this must be the first thing sent to the web browser
19 print("Content-type: text/html;charset=utf-8")
20 print("\n\n")  #two line breaks indicates the end of the headers
21 
22 #output the top of the HTML document
23 print('''
24 <!DOCTYPE html>
25 <html>
26     <head>
27         <meta charset="utf-8" />
28         <title>MongoDB Read From Python</title>
29         <link rel="stylesheet" type="text/css" href="css/main.css" />
30     </head>
31     <body>
32         <div class="container">
33         	<h1>View the books in the library</h1>
34             <p>
35                 <a href="create.html">Or add a new book to the collection</a>
36             </p>
37 ''')
38 
39 #loop through all the books in the collection in the database
40 for book in client.your_database_name.books.find():
41 
42     #get the data about this book from the dictionary... MongoDB is schemaless, so replace any missing fields are replaced by default values
43     title = book.get('title', 'Untitled')
44     author = book.get('author', 'Unknown Author')
45     edition = book.get('edition', 'Unknown Edition')
46     price = book.get('price', 'Unknown price')
47 
48     print('''
49 
50             <article>
51                 <h2>{title}</h2>
52                 <p>
53                     by {author} 
54                     <br />
55                     ({edition})
56                 </p>
57                 <div class='price'>
58                     ${price}
59                 </div>
60             </article>      
61 
62     '''.format(title=title, author=author, edition=edition, price=price))
63 
64 
65 #output the bottom of the HTML document
66 print('''
67     
68             <!-- clearfix -->
69             <div class="clear"></div>
70         </div>
71     </body>
72 </html>
73 ''')

Writing to MongoDB collection

This example allows the user to fill in a form to add a new document to a MongoDB collection.

create.html

 1 <!DOCTYPE html>
 2 <html lang="en">
 3     <head>
 4         <meta charset="utf-8" />
 5         <title>MongoDB Write Example</title>
 6         <link rel="stylesheet" type="text/css" href="css/main.css" />
 7     </head>
 8     <body>
 9         <div class="container">
10 
11             <h1>Add a new book to the library</h1>
12             <p>
13                 <a href="index.py">
14                     Or view the books in the collection
15                 </a>
16             </p>
17 
18             <p>Enter the details below?</p>
19 
20             <form action="processor.py" method="POST">
21                 <div class="row">
22                     <label for="title">Book title</label>
23                     <input type="text" id="title" name="title" />
24                     <div class="clearfix"></div>
25                 </div>
26 
27                 <div class="row">
28                     <label for="author">Author (last name, first name)</label>
29                     <input type="text" id="author" name="author" />
30                     <div class="clearfix"></div>
31                 </div>
32 
33                 <div class="row">
34                     <label for="edition">Edition</label>
35                     <input type="text" id="edition" name="edition" />
36                     <div class="clearfix"></div>
37                 </div>
38 
39                 <div class="row">
40                     <label for="price">Price</label>
41                     <input type="text" id="price" name="price" />
42                     <div class="clearfix"></div>
43                 </div>
44 
45                 <div class="row">
46                     <label for='submit'>Ready?</label>
47                     <input id='submit' type="submit" value="Add it!" />
48                     <div class="clearfix"></div>
49                 </div>
50             </form>
51 
52         </div><!-- //.container -->
53     </body>
54 </html>

processor.py

 1 #!/usr/bin/env python3
 2 
 3 #enable debugging... any errors will be output as HTML so they show up clearly in the web browser
 4 import cgi, cgitb
 5 cgitb.enable()
 6 
 7 #make sure all output from the print statement in Python is utf-8 encoded
 8 import sys, codecs
 9 sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
10 
11 #import mongodb 
12 import pymongo
13 
14 # use the CGI library to get any form data sent to the server from the client
15 form = cgi.FieldStorage() 
16 
17 # validate that data has been submitted via the form...
18 if "title" not in form or "author" not in form or "edition" not in form or "price" not in form:
19     # if the user did not enter form data, send the browser back to the form page
20     # we do this by sending a special HTTP location header to the browser
21     print("Location: ./create.html\n\n")
22     #exit now!
23     sys.exit()
24 
25 #assuming the user entered form data, get the form data into simple variables
26 title = form.getfirst('title')
27 author = form.getfirst('author')
28 edition = form.getfirst('edition')
29 price = form.getfirst('price')
30 
31 #connect to db server
32 client = pymongo.MongoClient('mongodb://your_username:your_password@your_host_name/your_database_name')
33 
34 #make a new document in Python's JSON closest equivalent... a dictionary
35 document = {
36     'title': title,
37     'author': author,
38     'edition': edition,
39     'price': price
40 }
41 
42 #insert a new document into the MongoDB collection
43 client.your_database_name.books.insert(document)
44 
45 #send the browser to view the previous example page so the user can see the new set of books
46 # we do this by sending a special HTTP location header to the browser
47 print("Location: ./index.py\n\n")


What links here