In the past few days I’ve had the chance to play a bit with Tornado, a non-blocking web server for Python similar to node.js. Tornado works by using something like epoll or whatever I/O event notification exists in the system. Tornado is single threaded and works like a fancy while (True) loop. As a consequence, whenever we do some blocking I/O (e.g. database call) the web server cannot process other requests. While that may not be a big deal on very low traffic sites or if your I/O subsystem is extremely fast, it starts becoming an issue as your traffic grows or if you’re hosted on something like Amazon AWS; notorious for its slow and inconsistent disk I/O performance. The biggest problems that I encountered with Tornado are: 1) the lack of robust asynchronous libraries for things like DB access and 2) Python’s own awkwardness when dealing with async code. NodeJS programmers don’t have any of these problems because the community writes software with async in mind and Javascript as a language offers anonymous blocks which make callback style code tolerable. The purpose of this post is to encourage Python programmers using Tornado to pay attention to their blocking calls or if they don’t need any of the features that Tornado offers, stick to a WSGI server such as uwsgi or gunicorn.
We’re going to start by writing a pretend service that takes half a second to return a response. Half a second is not a lot of time and doing things like calling the Facebook API for example can take much longer for that sometimes.
class PretendService(tornado.web.RequestHandler):
@asynchronous
def get(self):
""" Pretend some work is being done by sleeping for 500ms """
ioloop = tornado.ioloop.IOLoop.instance()
ioloop.add_timeout(time.time() + 0.5, self._finish_req)
def _finish_req(self):
self.finish()
Now we’re going to write two handlers, one of them will call the service in the way Python programmers are used to writing code. The other will make the same call the proper way that Tornado expects, using an asynchronous http client that supports callbacks. First the normal synchronous
class MainHandlerBlocking(tornado.web.RequestHandler):
def get(self):
req = httpclient.HTTPRequest(pretend_service_url, method='GET')
# we could use something like requests or urllib here
client = tornado.httpclient.HTTPClient()
response = client.fetch(req)
# do something with the response
And now the Tornado way
class MainHandlerAsync(tornado.web.RequestHandler):
@asynchronous
@gen.engine
def get(self):
req = httpclient.HTTPRequest(pretend_service_url, method='GET')
client = tornado.httpclient.AsyncHTTPClient()
# don't let the yield call confuse you, it's just Tornado helpers to make
# writing async code a bit easier. This is the same as doing
# client.fetch(req, callback=_some_other_helper_function)
response = yield gen.Task(client.fetch, req)
### do something with the response ###
self.finish()
and finally some plumbing code:
application = tornado.web.Application([
(r"/async", MainHandlerAsync),
(r"/external-api", PretendService),
(r"/blocking", MainHandlerBlocking)
])
if __name__ == "__main__":
define("port", default=8888, help="run on the given port", type=int)
tornado.options.parse_command_line()
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(options.port)
tornado.autoreload.start()
tornado.ioloop.IOLoop.instance().start()
Now that we have everything setup, lets fire up some benchmarks. First lets start with 5 concurrent users and try a total of 100 requests.
> ab -n 100 -c 5 http://127.0.0.1:8889/blocking
Requests per second: 1.96 [#/sec] (mean)
Time per request: 2549.939 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95% 2552ms
> ab -n 100 -c 5 http://127.0.0.1:889/async
Requests per second: 9.75 [#/sec] (mean)
Time per request: 513.067 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95% 521ms
As you can see from the results above, code written in a blocking fashion can only handle 2 requests per second and under a moderate load of 5 concurrent users takes 2.5 seconds to serve 95% of the requests. Code written async lets Tornado go on and handle more requests while we are blocked on that network call, resulting in 10 reqs/sec which 95% of the requests taking 521ms, 500 of which is due to the pretend service we wrote. But lets see how far we can take it, lets assume our service got popular and we now have 10 users at the same time.
> ab -n 500 -c 10 http://127.0.0.1:8889/blocking
Requests per second: 1.96 [#/sec] (mean)
Time per request: 5099.994 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95% 5102ms
> ab -n 500 -c 10 http://127.0.0.1:889/async
Requests per second: 19.06 [#/sec] (mean)
Time per request: 524.732 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95% 534ms
As we expected, our blocking code is still only able to handle 2 requests per second and now 95% of our users have to wait 5 seconds for a response. Our async code however lets Tornado scale and show its true power by handling 19 requests per sec with no noticeable increase in the time it takes to serve a request. Lets raise the bar one more time:
> ab -n 1000 -c 20 http://127.0.0.1:8889/blocking
Requests per second: 1.96 [#/sec] (mean)
Time per request: 10200.390 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95% 10203ms
> ab -n 1000 -c 20 http://127.0.0.1:889/async
Requests per second: 19.44 [#/sec] (mean)
Time per request: 1028.586 [ms] (mean)
Percentage of the requests served within a certain time (ms):
95% 1044ms
We are beating a dead horse here but obviously not only the async code scales better but Tornado is not kind to any sort of blocking on I/O. In that last test, we would need 10 Tornado instances to match the performance of a single async one.
Note that while I throw the word async around a lot, not any asynchronous library would work. For example, you can’t use grequests (gevent + requests), use AsyncHttpClient instead. That is because Tornado makes certain assumptions and uses a callback style API. In most cases, libraries would need to be rewritten specifically for Tornado but luckily some already have.
Finally, I recommend that if you are set on using Python and don’t need to build a real-time app stick with a WSGI server, mostly because of the lack of async libraries for Tornado and that callback-style code is not something that Python shines in. If you do plan on using Tornado, hopefully I’ve convinced you on the importance of not doing any blocking I/O.