Part 7 - Our own WSGI server in python
(The changes introduced in this post start here.)
As discussed in the post about the WSGI protocol, a WSGI server can be written in several different ways:
- as an extension to a python-based webserver
- as a python program that communicates with a webserver based on a gateway protocol (e.g. CGI, FastCGI)
- as a C program that extends a C-based webserver and interacts with the WSGI application through the python C-API
We’re going to use the first option, since we’re focused on python here and it’s the natural next step for our existing python HTTP server.
A quick refactor of the HTTP parser protocol
Up until now, we implemented the HTTP parser protocol (containing the callback functions to call on certain parse events, e.g. on_url, on_header, on_body, on_message_complete) in a separate class HttpRequestParserProtocol in http_request.py. That was alright for our dummy HTTP server that didn’t actually do anything with the request and just sent a blanket response regardless. But now we want to be much more reactive and the data in the request needs to be gathered and then submitted to the WSGI application to create an appropriate response.
Because of that, I now want the Session object (which runs in a separate thread for each client we’re communicating with) to be the central point where the request data is gathered, the application triggered and the response sent back to the client. In other words, the Session object will now become our HTTP parser protocol (i.e. it will implement the necessary callbacks for the parser to call).
We can delete http_request.py and instead alter server.py in the following way:
#./server.py
from typing import Tuple
import socket
import threading
from http_parse import HttpRequestParser
from http_response import make_response
class Session:
def __init__(self, client_socket, address):
self.client_socket = client_socket
self.address = address
self.response_sent = False
# self is now the protocol for the parser to interact with
self.parser = HttpRequestParser(self)
def run(self):
while True:
if self.response_sent:
break
data = self.client_socket.recv(1024)
print(f"Received {data}")
self.parser.feed_data(data)
self.client_socket.close()
print(f"Socket with {self.address} closed.")
def send_response(self):
body = b"<html><body>Hello World</body></html>"
response = make_response(status_code=200, headers=[], body=body)
self.client_socket.send(response)
print("Response sent.")
self.response_sent = True
# parser callbacks
def on_url(self, url: bytes):
print(f"Received url: {url}")
self.http_method = self.parser.http_method.decode("utf-8")
self.url = url.decode("utf-8")
self.headers = []
def on_header(self, name: bytes, value: bytes):
print(f"Received header: ({name}, {value})")
self.headers.append((name, value))
def on_body(self, body: bytes):
print(f"Received body: {body}")
def on_message_complete(self):
print("Received request completely.")
self.send_response()
def serve_forever(host: str, port: int):
server_socket = socket.socket()
server_socket.bind((host, port))
server_socket.listen(1)
while True:
client_socket, address = server_socket.accept()
print(f"Socket established with {address}.")
session = Session(client_socket, address)
t = threading.Thread(target=session.run)
t.start()
if __name__ == "__main__":
serve_forever("127.0.0.1", 5000)Integrating WSGI into the HTTP server
For our basic WSGI server implementation, we’ll want to wait until the parser has successfully parsed the entire HTTP request and given us the status line items, headers and (optional) body. Once that is done, we build the correct environ dict and call the application. The application should give us the response status and headers and then return with the (optional) response body. All of that we can then use to create a well-formed HTTP response and send it to the client.
The sequence should look roughly like the following:
You can see 2 new objects there req (Request) and resp (Response). We’ll implement those to hold the accumulated state of the HTTP request and response, respectively. They’ll also contain some convenience methods to translate between WSGI interactions and HTTP.
So let’s start with those in a new file wsgi.py.
#./wsgi.py
from dataclasses import dataclass, field
from typing import List, Tuple
from io import BytesIO
from http_response import make_response
@dataclass
class WSGIRequest:
http_method: str = ""
path: str = ""
headers: List[Tuple[str, str]] = field(default_factory=lambda: [])
body: BytesIO = BytesIO()
# build the environ dict from the accumulated request state
# (i.e. translate HTTP to WSGI)
def to_environ(self):
path_parts = self.path.split("?")
headers_dict = {k: v for k, v in self.headers}
environ = {
"REQUEST_METHOD": self.http_method,
"PATH_INFO": path_parts[0],
"QUERY_STRING": path_parts[1] if len(path_parts) > 1 else "",
"SERVER_NAME": "127.0.0.1",
"SERVER_PORT": "5000",
"SERVER_PROTOCOL": "HTTP/1.1",
"CONTENT_TYPE": headers_dict.get("Content-Type", ""),
"CONTENT_LENGTH": headers_dict.get("Content-Length", ""),
"wsgi.version": (1, 0),
"wsgi.url_scheme": "http",
"wsgi.input": self.body,
"wsgi.errors": BytesIO(),
"wsgi.multithread": True,
"wsgi.multiprocess": False,
"wsgi.run_once": False,
**{f"HTTP_{name}": value for name, value in self.headers},
}
return environ
@dataclass
class WSGIResponse:
status: str = ""
headers: List[Tuple[str, str]] = field(default_factory=lambda: [])
body: BytesIO = BytesIO()
is_sent: bool = False
# we will register this function as the application start_response calback
def start_response(
self, status: str, headers: List[Tuple[str, str]], exc_info=None
):
print("Start response with", status, headers)
self.status = status
self.headers = headers
# create a valid HTTP response from the accumulated response state
# (i.e. translate WSGI to HTTP)
def to_http(self):
self.is_sent = True
return make_response(self.status, self.headers, self.body)As you can see, both are straightforward data containers. The details of the entries that need to appear in the environ dict can be seen here. Most of them are quite self-explanatory. Of special interest is the wsgi.input, which must be a file-like object (i.e. supporting operations such as .write() and .read()). It is from this object that the application will read the request body (if there is any). We’ll instantiate and use both objects in the Session, fill them up with data and use their convenience methods to relay the communication with the WSGI application.
Simplifying our HTTP response creator
Previously our http_response.py generated a valid response based on a status code, headers and body. However, the WSGI specification says that applications need to not only return the code (e.g. 200) but also the status phrase (e.g. 200 OK). Another change is that we communicate with the WSGI application mainly in real unicode strings, the only thing that is still a bytes is the request/response body. So, e.g. headers are now List[Tuple[str, str]] internally, and are only converted to bytes right before sending the response out over TCP.
Overall, the changes to http_response.py are more of a cosmetic nature:
#./http_response.py
from typing import List, Tuple
def create_status_line(status: str = "200 OK") -> str:
return f"HTTP/1.1 {status}\r\n"
def format_headers(headers: List[Tuple[str, str]]) -> str:
return "".join([f"{key}: {value}\r\n" for key, value in headers])
def make_response(
status: str = "200 OK",
headers: List[Tuple[str, str]] = None,
body: bytes = b"",
):
if headers is None:
headers = []
content = [
create_status_line(status).encode("utf-8"),
format_headers(headers).encode("utf-8"),
b"\r\n" if body else b"",
body,
]
return b"".join(content)Also there is a mini-change to our HTTP parser in http_parse.py where now on line 32 we actually save the parsed HTTP method, such that we can later retrieve it.
@@ -30,6 +30,7 @@ def parse_startline(self):
line = self.buffer.pop(separator=b"\r\n")
if line is not None:
http_method, url, http_version = line.strip().split()
+ self.http_method = http_method
self.done_parsing_start = True
self.protocol.on_url(url)
self.parse()
Communication with the WSGI app
Now we can tie everything together and use our new WSGIRequest and WSGIResponse objects in the Session in server.py to facilitate the communication with the WSGI application.
#./server.py
from typing import Tuple, List
import socket
import threading
from io import BytesIO
from http_parse import HttpRequestParser
from wsgi import WSGIRequest, WSGIResponse
# our mystery app, to be implemented in a second
from app import app
class Session:
def __init__(self, client_socket, address):
self.client_socket = client_socket
self.address = address
self.parser = HttpRequestParser(self)
self.request = WSGIRequest()
self.response = WSGIResponse()
def run(self):
while True:
# this flag is now stored on the new response object
if self.response.is_sent:
break
data = self.client_socket.recv(1024)
print(f"Received {data}")
self.parser.feed_data(data)
self.client_socket.close()
print(f"Socket with {self.address} closed.")
# parser callbacks
def on_url(self, url: bytes):
print(f"Received url: {url}")
self.request.http_method = self.parser.http_method.decode("utf-8")
self.request.path = url.decode("utf-8")
def on_header(self, name: bytes, value: bytes):
print(f"Received header: ({name}, {value})")
self.request.headers.append(
(name.decode("utf-8"), value.decode("utf-8"))
)
def on_body(self, body: bytes):
print(f"Received body: {body}")
self.request.body.write(body)
self.request.body.seek(0)
def on_message_complete(self):
print("Received request completely.")
# the functionality in this function replaces the previous
# self.send_response functionality
environ = self.request.to_environ()
# the start_response callback is a method on the WSGIResponse object
# here we call the app to dynamically create a response
body_chunks = app(environ, self.response.start_response)
print("App callable has returned.")
self.response.body = b"".join(body_chunks)
self.client_socket.send(self.response.to_http())
def serve_forever(host: str, port: int):
server_socket = socket.socket()
server_socket.bind((host, port))
server_socket.listen(1)
while True:
client_socket, address = server_socket.accept()
print(f"Socket established with {address}.")
session = Session(client_socket, address)
t = threading.Thread(target=session.run)
t.start()
if __name__ == "__main__":
serve_forever("127.0.0.1", 5000)And that’s about it, really. If you go through the code closely and follow the sequence diagram earlier in this post, the logic and flow shouldn’t be hard to understand.
An application to call
Lastly, we need a new file app.py with some kind of WSGI application implementation. Let’s use flask for this for now and implement something as follows.
#./app.py
from flask import Flask, request
app = Flask(__name__)
@app.route("/", methods=["GET"])
def root():
print("Called root endpoint.")
return "hello from /"
@app.route("/create", methods=["POST"])
def create():
print(f"Called create endpoint with data {request.data}.")
return "hello from /create"At this point you can test that indeed there is something useful coming out of our pipeline. Use curl to send a request, e.g.:
$ curl localhost:5000/create -d '{"abc": 123}' -H "Content-Type: application/json"
hello from /createshould result in a server log like the following:
$ python server.py
Socket established with ('127.0.0.1', 48294).
Received b'POST /create HTTP/1.1\r\nHost: localhost:5000\r\nUser-Agent: curl/7.69.1\r\nAccept: */*\r\nContent-Type: application/json\r\nContent-Length: 12\r\n\r\n{"abc": 123}'
Received url: b'/create'
Received header: (b'Host', b'localhost:5000')
Received header: (b'User-Agent', b'curl/7.69.1')
Received header: (b'Accept', b'*/*')
Received header: (b'Content-Type', b'application/json')
Received header: (b'Content-Length', b'12')
Received body: b'{"abc": 123}'
Received request completely.
Called create endpoint with data b'{"abc": 123}'.
Start response with 200 OK [('Content-Type', 'text/html; charset=utf-8'), ('Content-Length', '18')]
App callable has returned.
Socket with ('127.0.0.1', 48294) closed.Notes
Again, we haven’t implemented any error handling at all. If you paid close attention, you will have seen that in the environ dict there is a wsgi.error entry. This should normally be used by the application to write error data back to the WSGI server. However, we just put an empty BytesIO there which we never look at again subsequently. Oops. Also, the start_response callback has an optional third parameter exc_info which can be used by the app to forward an application exception to the WSGI server. We don’t do anything with that either. Oops.
So, as before in this series we’ll just assume that a client plays nice and that the application does not cause any trouble. Our goal is to grasp the fundamental features, not to build a production-ready program.
One other big miss is how the request/response body data is currently handled. In both cases we’re buffering the entire body in memory, instead of continuously streaming in smaller chunks. Beyond that, the WSGI specification describes many features that a server should or may implement. Check out the section on “Other HTTP Features” for a taste of that, it’s all rather mushy. It’s definitely one of the parts I like least about WSGI. It seems like if you tell 100 people to implement a WSGI server, there will be 100 solutions with slight differences in behavior.
As for our implementation: basic GET or POST requests work fine, anything beyond that will probably cause some trouble somewhere. But already I find it quite amazing, that with just a few additions to our basic HTTP server we are now in a position to interface with any web application that follows the WSGI protocol.
In the next post we’ll have a look at building our own basic framework (like flask) to quickly cobble together WSGI applications.