Does Django Scale

Does Django scale?

1. "What are the largest sites built on Django today?"

   There isn't any single place that collects information about traffic on Django built sites, so I'll have to take a stab at it using data from various locations. First, we have a list of Django sites on the front page of the main Django project page and then a list of Django built sites at djangosites.org. Going through the lists and picking some that I know have decent traffic we see:

   • Instagram: What Powers Instagram: Hundreds of Instances, Dozens of Technologies.

   • Pinterest: Alexa rank 37 (21.4.2015) and 70 Million users in 2013

   • Bitbucket: 200TB of Code and 2.500.000 Users

   • Disqus: Serving 400 million people with Python.

   • curse.com: 600k daily visits.

   • tabblo.com: 44k daily visits, see Ned Batchelder's posts Infrastructure for modern web sites.

   • chesspark.com: Alexa rank about 179k.

   • pownce.com (no longer active): alexa rank about 65k.
     Mike Malone of Pownce, in his EuroDjangoCon presentation on Scaling Django Web Apps says "hundreds of hits per second". This is a very good presentation on how to scale Django, and makes some good points including (current) shortcomings in Django scalability.

   • HP had a site built with Django 1.5: ePrint center. However, as for novemer/2015 the entire website was migrated and this link is just a redirect. This website was a world-wide service attending subscription to Instant Ink and related services HP offered (*).

2. "Can Django deal with 100,000 users daily, each visiting the site for a couple of hours?"

   Yes, see above.

3. "Could a site like Stack Overflow run on Django?"

   My gut feeling is yes but, as others answered and Mike Malone mentions in his presentation, database design is critical. Strong proof might also be found at www.cnprog.com if we can find any reliable traffic stats. Anyway, it's not just something that will happen by throwing together a bunch of Django models :)

There are, of course, many more sites and bloggers of interest, but I have got to stop somewhere!

---

Blog post about Using Django to build high-traffic site michaelmoore.com described as a top 10,000 website. Quantcast stats and compete.com stats.

---

_{(*) The author of the edit, including such reference, used to work as outsourced developer in that project.}

Deployment of django app - Scaling, Static Files, Servers

1) It is completely fine to host your staticfiles on the same server as your django application however to serve said files you should use a web server such as NGINX or Apache. Django was not designed to serve static data in a production environment. Nginx and Apache on the other hand do great job at it.

2) You can definitely host your static and media files inside an S3 bucket. This will scale a lot better than hosting them on a single server as they're provided by a separate entity, meaning that no matter how many application servers you're running behind a load balancer, all of them will be requesting staticfiles from the same source. To make it more efficient you can even configure AWS' CloudFront which is Amazons CDN (content delivery network).

3) Ideally your database should be hosted on a separate server. Databases are heavy on resources therefore hosting your database on the same server as your application may lead to slowdowns and sometimes outright crashes. Scaling horizontally, you'd be connecting a lot of application servers to a single database instance; effectively increasing the load on that server.

All of the points above are relative to your use case and resources. If the application you are running doesn't deal with heavy traffic - say a few hundred hits a day - and your server has an adequate amount of resources (RAM, CPU, storage) it's acceptable to run everything off a single box.

However if you're planning to accept tens of thousands of connections every day it's better to separate the responsibilities for optimum scalability. Not only it makes your application more efficient and responsive but it also makes your life easier in the long run when you need to scale further (database clustering, nearline content delivery, etc).

TL;DR: you can run everything off a single server if it's beefy enough but in the long run it'll make your life harder.

Django: Horizontal scaling with a single database

You can enable autoscaling on Render and it will automatically scale your instances up (and down) based on your application's average CPU and/or memory utilization across all instances. You do not need to change your Django app.

How to refactor a Django site so that it can scale

Unfortunately there aren't a lot of specifics in your question. Google searches will yield a lot of information on scaling django.

http://www.morethanseven.net/2011/06/30/Django-performance-1-measuring-performance.html

This link provides about 5 different ways to measure different performance metrics. Since I can't tell you specifics on how to scale your app, here are some general good principals for any site.

1. Make sure static media is served by your web server and not by django
2. Make sure you profile everything EXPLAIN is crucial.
3. Cache everything you can. Memory is lightning fast and django makes it easy to implement.
4. Be careful of django ORM gotchas in your templates. Use django-debug toolbar to make sure you are not inadvertantly making multiple queries when iterating over result sets.
5. For databases make sure your data is normalized.
6. Even better use a CDN and make sure that your web servers and DB servers are on seperate machines.

Dont optimize prematurely!! Django can handle A LOT out of the box. I wouldn't worry about ajax or model forms or anything unless you find out that they are specific bottlenecks to you growing your site. I believe just as important as performance is making sure your site is modular, under version control and easily movable. This will facilitate in development and when you grow large enough will let you bring up new machines instantaneously.

Can you really scale up with Django...given that you can only use one database? (In the models.py and settings.py)

Django now has support for multiple databases.

Should Django be used for large, complex sites?

I found this quite interesting to read: http://softwaremaniacs.org/blog/2008/02/24/why-offline-crashed-en/

There are a lot of interesting articles in this blog, but, unfortunately, they're all in Russian.

Added: And since that post Ivan did write MySQL backend supporting master-slave replication: http://softwaremaniacs.org/soft/mysql_replicated/en/

---

Related Topics