In this article, I want to share my previous experience where I managed to increase the speed of Django admin to load more than 20 million records from MongoDB.
If you want to learn how to integrate MongoDB with Django admin then you can check my other post below:
We used to have MongoDB where it's running in Kubernetes pods and connected with Django admin to make CRUD operations easier and also for better searching with custom filtering methods.
The issue was the slow loading of records from MongoDB with sometimes timeout errors. The data used to increase on a daily basis which also makes loading even worse.
Also, we're using Djongo which translates queries for MongoDB by allowing us to use Django ORM itself.
Now, let's take a look at the general reasons for the slow performance.
Normally, MongoDB scans all documents to select if it matches the query. Since we have millions of records, it takes very long to complete scanning. By indexing the specific fields that we're using in the query, MongoDB can use the index to limit the number of documents it must inspect.
Since we're considering the precise solution for the current problem, you can check the documentation for more detailed information.
- You must create indexes for the field that you're using for filtering.
- If filtering includes multiple fields then consider creating compound indexes.
Note that, indexes are special data structures that store a small subset of the data held in a collection’s documents separately from the documents themselves. So, you have to check memory (RAM) performance and increase it if required.
Django admin uses pagination by default which counts the number of records and creates the corresponding paginator. At this point, it takes plenty of time to wait until counting is finished.
First, let's take a look
count property of
@cached_property def count(self): """Return the total number of objects, across all pages.""" c = getattr(self.object_list, 'count', None) if callable(c) and not inspect.isbuiltin(c) and method_has_no_args(c): return c() return len(self.object_list)
Now, since we don't need counting let's create a custom paginator to override default functionality by skipping the actual counting:
from django.core.paginator import Paginator class NoCountPaginator(Paginator): @property def count(self): return 999999999
Then define the paginator attribute in your admin class as below:
@admin.register(models.Mymodel) class MymodelAdmin: paginator = NoCountPaginator ...
We should also prevent the counting on the list form which disables count query on filtered pages to show the full count by setting
@admin.register(models.Mymodel) class MymodelAdmin: paginator = NoCountPaginator show_full_result_count = False
This is an extra alert to check custom filtering functions (
list_filter ) in Django admin that you might implement previously. In my case, there was a field which used to get distinct values across all documents. It extremely slows down the performance as the number of documents increases.
As a solution, you can replace it with textbox searching instead of scanning all documents and getting specific values for filtering.
Using offset in query will not change the performance but can make it even much slower than previous. Since you are declaring offset, MongoDB will go through all documents until it reaches the particular one. For the first few pages, it can be fast but the larger offset gets the slower performance will follow.
If you feel like you unlocked new skills, please share with your friends and subscribe to the youtube channel to not miss any valuable information.