Part 3 - WSGI to the rescue
As we’ve seen in the last post, the situation in python web programming was pretty messy in the early 2000s. There were several ways of deploying your web application that were mutually incompatible. If you e.g. adjusted your application to work with the
FastCGI protocol, you could only deploy it on webservers with
FastCGI support. If you implemented your application to work with the
mod_python protocol, you could only deploy it using the
Apache webserver, because that was the one that
mod_python was written for.
Not having a clear, uniform standard for developing python web applications balkanized the field and stifled development.
The WSGI protocol
Luckily in 2003
Web Server Gateway Interface) was proposed. It defines a simple interface between a
WSGI server and a
WSGI application. Although they are called
WSGI server and application, it is important to be aware of the fact that they are basically 2 halves of what would previously have been your web application.
So, a flow that might have looked like this:
Would now look like this:
WSGI application is written in python and is simply a callable object with exactly 2 arguments
environ input is a dictionary that contains request parameters (much like the environment variables used in
CGI). It also contains some
WSGI specific entries, e.g. a file-like object to read the (optional) request body from. The
start_response input is a callable that will be used by the application to signal to the
WSGI server that a response has been created and submit the response
HTTP status and headers.
The application callable must then return an iterable of
bytes which signify the response body. An empty return iterable would mean a response without a body.
A tiny WSGI application
So what could a
WSGI application look like? In the simplest form it’s just a python function with the correct signature, behavior and return.
def application(environ, start_response): status = "200 OK" response_headers = [("Content-type", "text/plain")] start_response(status, response_headers) return [b"Hello World!"]
This function would be called by the
WSGI server once for every request/response cycle. Behind this simple interface you could scale up your web application to arbitrary complexity e.g. by dispatching to different endpoints based on the path information in the
WSGI server/application communication sequence
We’ll look at the details of the specification much more closely in a later part of the series when we’ll implement our own
WSGI server and application. For now, this high-level overview is sufficient.
How to implement a WSGI server?
More interesting than the
WSGI application, and a source of confusion for me for a long time, is how you actually implement a
WSGI server. As we’ve seen, the application is a callable in plain python. So a
WSGI server needs to be some program that can call a python callable.
In a python webserver
It’s easy enough to see how to communicate between a
WSGI server and application if the webserver is written in python. In that case you could just import the web application and integrate it in the
HTTP request/response cycle and call it at the appropriate point to generate a response.
from my_app import application # ... parse HTTP request ... environ = ... def start_response(status, response_headers): ... response_body = application(environ, start_response) # ... send HTTP response ...
In this case, the webserver and the
WSGI server would be tightly integrated and all communication between the webserver, the
WSGI server and the
WSGI application would happen within python.
On any non-python webserver
But how would this integration and communication between the different parts happen if the webserver is not actually written in python? Especially given the fact that at that time, many of the most potent webservers were not written in python.
We’ve already looked at some protocols that facilitate communication between arbitrary webservers and scripts via environment variables and stdin/stdout (
CGI) or sockets (
FastCGI). Those protocols can be reused to put
WSGI on top of them. Here is an example flow for how this could look like using
with FastCGI] ws---static end subgraph python pool subgraph WSGI server wsgi_s[WSGI server
with FastCGI] end wsgi_app[WSGI app] wsgi_s---wsgi_app end end
At this point it starts to get really interesting. The only thing you would have to implement, compared to the example of a webserver entirely written in python, is the
WSGI server component. It would now have to translate from/to
FastCGI in order to communicate with the webserver. This wouldn’t be too hard, since there are already powerful libraries to work with
The beauty is that you don’t need to change the
WSGI application at all. It remains exactly the same as in the previous example.
This process would work the same for
CGI and in fact any other protocol that lets a webserver interact with arbitrary code, i.e. a “gateway protocol”. The
WSGI specification even has an example implementation of a
WSGI server based on
CGI here. The community would just need to write some glue code to translate between that protocol and a
WSGI server and you could immediately deploy your
WSGI application without any changes necessary.
On a C webserver
Let’s also check out how this would pan out when the webserver is written in
C (of course, apart from the fact that you could also work with
FastCGI on a
C webserver). As we’ve seen before, the python
C-API gives the
C language a somewhat elevated position for interacting with python code.
This would work very much along the same lines as
mod_python. You would need to write a specific webserver module that acts as the
WSGI server and calls your
WSGI application through the python
C-API. There actually is an
Apache module that does just that, called
WSGI application would remain completely unchanged.
What WSGI brings to the table
I hope you’ve now realized the critical innovation that
WSGI brought to the python web programming ecosystem. The crucial part is really the splitting up of the web application into a
WSGI server and application part. Where previously your web application had to be adjusted specifically for one particularly way of interacting with a webserver, this interaction part now lies entirely in the
WSGI server. The
WSGI application part is only concerned with the business logic of your app and doesn’t care about how it will get delivered via the
One thing that confused me for a long time was specifically the word
WSGI “server”. In my (admittedly, at that time, not very deep) reading of the
WSGI specification, I thought that the
WSGI server would itself actually need to be a full-fledged webserver. And this can be the case if you have a webserver written in python that also plays the role of a
WSGI server. But it can just as well be a module running in a
C-based webserver or a little python component that translates between
WSGI and a gateway protocol like
FastCGI. So, ofentimes the
WSGI server is not actually a “server” but rather a simple “translator”.
The big improvement due to
WSGI is that web application developers in python never again have to worry much about deployment. They know that, as long as their app adheres to the
WSGI application interface, there will be a way to deploy it behind pretty much all the popular webservers.
There is one other cool feature that
WSGI enables: middleware. Middleware are python programs that kind-of function as a
WSGI server and a
WSGI application at the same time. So a middleware is itself a callable that accepts the
start_response inputs and acts as an application when called. But then it actually also calls some other
WSGI application while building a response.
from webapp import application def middleware(environ, start_response): # ... do something before calling the actual application ... response_body = application(environ, start_response) # ... do something after calling the actual application ... return response_body
So the name really says where it would be used: between the
WSGI server and application. It could alter the request, or the response, or both. You could even chain middleware.
It could do jobs like only passing a request through if it is authenticated, or routing to different applications or measuring the time it takes for the response to be created and writing that in a response header. Really, you could do all kinds of interesting things with it and it would be easy to create reusable components.
I hope you’ve gained a good understanding of what
WSGI is. It definitely took me quite a while. Writing these posts forced me to inspect some gray areas where I thought I had a solid grasp of the topic, but then on closer inspection I realized I did not.
One problem is definitely that there are quite a lot of acronyms floating around and there are usually at least 3 different things called a “server” and sometimes all 3 functionalities are contained in the same program and other times they are 3 different entities that communicate in some fashion.
Also, I simply wasn’t on the internet much at the time when everything was still static pages. I didn’t witness the evolution of standards like
FastCGI or solutions like
WSGI. I never saw the problems they solved and the shortcomings they had. For me it has always been very clear that “of course you can somehow execute some python code after a request to a webserver and create and return a dynamic response”. I didn’t really comprehend that this might not be so trivial after all. The comfortable situation that we now have is really due to countless contributions and ideas from many people in the community, so let’s be grateful for that.
As a final point: here and here and here are some great posts that shed light on this topic from slightly different angles to sharpen up your understanding.
Now that we’ve got all the theory down, we can actually start building our own
WSGI server in the next post.