Django with docker
Learn everything about deploying Django projects on docker. We will use the Doprax cloud platform. We will deploy the Django project with MySQL as the database, and also we will use Nginx as a reverse proxy. Furthermore, we will use Gunicorn as the WSGI HTTP server. The final deployed Django website is accessible here. You can find the repository of the Django project code here.
Install Django
To use Django you must install it on your machine (and also the server). You can install it system-wide or in a virtual environment. To avoid conflicts with other packages it is strongly recommended to use virtual environments to install any python requirements. We are going to use the virtualenv package to manage the virtual environments. Also to install any Python packages you need to have pip on your machine. If you use virtualenv, pip will be automatically installed and available inside your virtual environment. But for the sake of completeness, we install both of them now.
sudo apt-get install python3-pip sudo apt-get install virtualenv
You can also use venv which is a version of Virtualenv that has been integrated into the Python standard library, but Virtualenv is a better choice. You can learn more about Virtualenv here. Now create a virtual environment for your project.
virtualenv myenv -p python3
This will create a folder named myenv in your current directory. Activate your virtual environment using this command:
source ./myenv/bin/activate
Now it is time to install Django.
pip install django == 2.2
We are installing Django version 2.2 which is still the latest LTS (Long Term Support) version of Django. Although at the time of this writing, the newest version of Django is 3.1.7. You can install a specific version by using pip install django == 3.1.7. It shouldn’t be that much different. The future LTS version is going to be 3.2 and it is scheduled to be released in mid-2021.
Create a Django project
Now that we have our virtual environment activated and Django installed, it is time to create a new project. Use this command to create a project
django-admin startproject myproject
This command will create a new Django project named myproject in the current directory. Sofar the project structure should look like this:
Now let’s run this Django project to see if it is working or not. Make sure that your working directory is inside myproject where the manage.py file resides.
cd myproject python manage.py runserver
This command will run the development server provided by Django and it will be run in your localhost address on port 8000. You can now see the Django welcome page if you enter the following URL into your browser, or just click it (http://127.0.0.1:8000). You should see the following page:
You will see some complaints in your terminal about migrations. To get rid of it enter the command below to do the migrations. More on migrations later.
python manage.py migrate
Create a new Django app
To make the Django project more realistic (complex) and test some features like migrations, let’s create a Django app inside our Django project. The app will be named myapp and it is created like this:
python manage.py startapp myapp
Using the above command a new folder named myapp containing our Django app will be created. The first thing that we need to do when creating a new Django app is to add it to the installed apps list in settngs.py. So open myproject/settings.py file and add 'myapp', to the end of the INSTALLED_APPS list.
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'myapp',
]
The new project structure will look like this:
Notice the db.sqlite3 file in the project root. It has been created since we are using the default SQLite as the database for our Django project. SQLite is ok for development and test purposes but it is NOT recommended to use it in production. We will use a production-grade MySQL service as our database later.
Create a simple page
Now let’s continue by creating a simple page in our newly added Django app. Open myapp/views.py and create a new function to handle our first page like following
from django.shortcuts import render
def first_page(request):
return render(request, 'first_page.html', {})
This is the simplest views function possible! It just renders the 'first_page.html' template file (which we will create next) and returns the response to the user. Now open myproject/urls.py and import the views function that we created and also add a new line to the urlpatterns list. In the end, it should look like this:
from django.contrib import admin
from django.urls import path
from myapp.views import first_page
urlpatterns = [
path('admin/', admin.site.urls),
path('', first_page, name='index'),
]
Now create a folder named templates at the root of the project (where manage.py resides). Inside this folder create a file and name it first_page.html and then copy the following in it.
{% load static %}
<!doctype html>
<html lang="en">
<body>
<div style="background-color:blue; padding:50px">
<h1>Welcome to django deployment Doc page</h1>
</div>
<div style="padding:20px;">
<img src="{% static "doprax_logo.png" %}" alt="doprax">
<p>Powered by <a href="https://www.doprax.com/"> Doprax.com </a></p>
</div>
</body>
</html>Create a folder at the root of your project and name it static. This folder will host the static files used in your project like images, font, CSS and javascript files, and other static files. download the following image and copy it to the static folder
Three small changes are also needed in settings.py file to tell Django where to look for template files and static files. Open settings.py and find the TEMPLATES section. There should be an entry in the dictionary like this 'DIRS': [], change it to the following:
'DIRS': [os.path.join(BASE_DIR, 'templates')],
It tells Django to look for templates in a folder named templates at the root of the project. Next, add the following to the settings.py file also. You can add it anywhere in the file but I recommend fining the STATIC_URL item and paste the following below it so that static file settings would be near each other for ease of lookup. More on serving static files later. It should look like this:
STATIC_URL = '/static/'
STATIC_ROOT = '/app_data/static/'
STATICFILES_DIRS = [
os.path.join(BASE_DIR, "static"),
]Also, find ALLOWED_HOSTS = [] and add the following to it
ALLOWED_HOSTS = ['.doprax.com']
Notice that we have used a wildcard for allowed hosts, meaning that every domain name ending with doprax.com is acceptable. If you want to add a custom domain to your project, like example.com, you need to add it also to the ALLOWED_HOSTS list like this ALLOWED_HOSTS = ['.doprax.com', 'example.com'].
No open your browser and go to this link (http://127.0.0.1:8000). It should look like this:
Config Django to use MySQL
As I mentioned earlier, the default database server used in Django is SQLite, Which is only for development and test and is NOT suitable for production websites. So we are going to configure the Django application to use MySQL as the database server. If you want to have MySQL for local development you need to install it on your machine. You can use SQLite for your local development and MySQL in production. This way you can avoid the overhead of installing and configuring MySQL server locally and for production, you can use doprax MySQL service which does not need any installation. Open settings.py file and find DATABASE section. By default it should be like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}It is completely ok to use it for your local development so we are going to keep it like this for your local development but we are going to also add database configuration for production. You must comment out the local database config when you deploy your project to doprax. For production use it should look like this:
# DATABASES = {
# 'default': {
# 'ENGINE': 'django.db.backends.sqlite3',
# 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
# }
# }
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'mydb',
'USER': 'root',
'PASSWORD': os.environ['DB_PASS'],
'HOST': os.environ['DB_HOST'],
'PORT': '3306',
}
}
Notice that we have not entered the password and host of the database, instead, we are using environment variables. So when the Django project runs in production, it will read the value of DB_PASS and DB_HOST dynamically. So we must not forget to add these environment values later when we create a project in doprax.
Add Dockerfile
Docker is arguably the best way to package and deploy software these days and it is gaining huge popularity. Doprax deploys your Django application on Docker. You don’t have to use Docker for your development environment although we strongly suggest it. Docker uses a Dockerfile as the instruction for building and running your application. So we need to create a file at the root of the project and name it Dockerfile, (notice the capital D letter).
FROM ubuntu:bionic
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get -y install \
python3 python3-dev python3-dev python3-pip python3-venv python3-wheel \
mysql-client libsqlclient-dev libssl-dev default-libmysqlclient-dev
ARG USER=root
USER $USER
RUN python3 -m venv venv
WORKDIR /app
COPY requirements.txt requirements.txt
RUN /venv/bin/pip install -r requirements.txt
COPY . .
EXPOSE 5000
RUN chmod +x /app/start.sh
ENTRYPOINT ["./start.sh"]
This file is the instruction for the deployment of a production-grade Django application. It uses the official ubuntu 18.04 Docker image as the base image. Then installs all necessary packages to run Django and interact with MySQL server. Then it will create a virtual environment and install all python dependencies needed from the requirements.txt file (which we will create next). Then it will expose port number 5000, and makes start.sh (which we will shortly create ) shell script executable and runs it.
Add requirements.txt
To manage the python requirements of the project, we need to add a requirements.txt file to our project root and list every dependency in it. Since our project is very simple for now, we only need three packages. The first one is Django, the second one is mysqlclient which is needed to make connections to a MySQL database server and the last one is Gunicorn which is the WSGI web server needed for a production-grade Django application. More on Gunicorn later. If you need any additional packages, feel free to list them in the file. For now, enter the following in the requirements.txt file:
To avoid surprise bugs, it is a good idea to explicitly specify the exact version of your dependencies.
Add Start script
Docker needs one primary process to run. As It has been specified in the last line of our Dockerfile (ENTRYPOINT) our primary process is running start.sh shell script. We will now create this file and instruct it to do the necessary steps needed to run our Django web application. Create a start.sh file at the root of the project and copy the following line into it.
<strong>#!/bin/bash</strong> source /venv/bin/activate cd /app echo "----- Collect static files ------ " python manage.py collectstatic --noinput echo "-----------Apply migration--------- " python manage.py makemigrations python manage.py migrate echo "-----------Run gunicorn--------- " gunicorn -b :5000 myapp.wsgi:application
This shell script does the following:
- activates the virtual environment inside the container
- changes directory to the /app which is the main working directory of the application inside the container
- It does collect static (we have not instructed it yet to serve static files in production)
- Makes the migrations inf there is any change to the models
- Executes the migrations (migrate)
- It will run the WSGI HTTP server using Gunicorn (more on it next)
if your application needs more initialization, for example, it needs to copy some files from another server, you can instruct it to do so in this start.sh shell script.
Gunicorn as WSGI HTTP Server
Django by default comes with a built-in development HTTP server. You run it on your local machine by commanding python manage.py runserver. It will serve the request to the users. But it has several drawbacks mainly, being single-threaded, being slow, and being insecure! So nobody should use it in production AT ALL. Instead, you should use Gunicorn. As it says on the Gunicorn website, the Gunicorn server is fast, compatible with various web frameworks, and light-weight in terms of server resources. We install Gunicorn using pip (included it in requirements.txt). The basic gunicorn command that we have included in the start.sh shell script is fine for most cases but these are few important configs that you may want to use:
# basic gunicorn command, binding to port 5000 gunicorn -b :5000 myapp.wsgi:application # specify the number of workers with a positive integer. Generally, 4*(num cores) defaults to 1 gunicorn -b :5000 --workers INT myapp.wsgi:application # Outputs the access log to a file named logfile. Use - to output to stdout gunicorn -b :5000 --access-logfile logfile myapp.wsgi:application # specify log level default is info. Options for LEVEL: debug, info, warning, error, critical gunicorn -b :5000 --log-level LEVEL myapp.wsgi:application
You can learn more about Gunicorn configurations at Gunicorn documentation.
Push code to GitHub
Our demo website is ready now and we will push it to GitHub and continue the deployment of the Django application on Doprax. First, you need to create a repository on your GitHub account. Then you need to commit your code and push it to GitHub. My repository name is django-example.
git init git add . git commit -m"initial commit" git branch -M main git remote add origin git@github.com:hemenxyz/django-example.git git push -u origin main
Now you need to connect your GitHub account to doprax to continue. Go to the account section (https://www.doprax.com/account/) and click on the “connect to github” button.
You will be directed to GitHub to authorize doprax to access your repositories. If you are not logged in, you will be prompted to log in to your account, and then you will be shown the authorization page. It will look like the image below. Click on authorize dopraxcom to continue.
You can always manage authorization settings from your GitHub account settings section (https://github.com/settings/applications). Now your doprax account is connected to your GitHub account.
Now, are you ready to rock?
Create your Doprax account now – No credit card needed
Create a project in doprax
Now you need to create a project. Get more information about Doprax projects here. In your project detail page under the source sub-section, click on the import from my github account button. Then choose the repository of our demo Django application and then click import.
Add MySQL service
Under the services sub-section, click on add service button. On the next page, click on MySQL. You need to provide 4 configuration values.
- MYSQL_ROOT_PASSWORD is the root password of your MySQL server
- MYSQL_DATABASE is the name of the database we want to create. In our example, we have used
mydb, so you need to entermydb. - MYSQL_USER is a user that is created and will be assigned a super-user role for MYSQL_DATABASE mentioned above
- MYSQL_PASSWORD is the password for MYSQL_USER mentioned above.
If you do no enter any of the above values, a default value will be generated for it automatically. For passwords, it will be a random password. If you remember from when we configured Django to use MySQL as the database server we need to create two environment variables one for MySQL host and another for the password of the root user. The hostname of the MySQL service can be found in the services sub-section under MySQL name. In the image below, it has been marked by a red rectangle. You need to find your own MySQL hostname. Don’t copy mine!
Go to the source sub-section and create two environment variables:
- Key: DB_PASS
- Value: the password you chose for the root user database (MYSQL_ROOT_PASSWORD)
- Key: DB_HOST
- Value: the hostname of the Mysql service you just copied (red rectangle in the above image)
Now we need to add a volume to our project to save MySQL data. Go to the volumes sub-section and click on create volume. Give it a title. Next, choose MySQL to mount on and for the mount path, enter /var/lib/mysql/ and then click create.
Add Nginx service
Nginx is a high-performance HTTP server and reverse-proxy used by many of the world’s biggest deployments of Django (like Instagram) and other programming languages and frameworks. We will use Nginx to act as reverse proxy for our application and also serve the static files of our application. Websites need to serve images, CSS files and javascript files, and other kinds of files static files. Django development server by default serves the static files but it is highly inefficient for production use. Django has a built-in utility called collectstatic that collects all the static files in one place (STATIC_ROOT) so that they could be served by the Nginx or Apache or other webservers. So far we have configured STATIC_ROOT in settings.py file and we have used collectstatic utility in start.sh shell script to make the static files ready to be served. Next, we are going to serve the static files using Nginx.
Go to the services sub-section and click on add service. On the next page, click Nginx and then click add. The Nginx service is now added to our project. Now we need to configure it to proxy requests to our Django application and also serve static files from our static root. To make our static files available to Nginx, we need to create a volume and mount it to the main container (source), and also share it with Nginx.
Go to the volumes sub-section and click on create volume. Give it a title and then select Main to mount the volume on. For the path enter /app_data/static/ and then click create.
Now we need to share this volume with the Nginx service so that it could serve static files to our users. To do that click on the share button on the static volume that we just created. It has been marked by a red rectangle in the image below
You will then be prompted to choose the service to mount the volume on and also enter the path to mount the volume on the target service. Choose Nginx and enter /app_data/static/ exactly like mount path of static volume on main. Choose read-only as mount mode so that Nginx could only read from it and not write on it. Now we have a volume that has been shared with two services.
Now it is time to config the Nginx service to handle static files and also proxy the requests to our gunicorn process in the main container. Go to the services sub-section and find Nginx service and add a mounted config file config:
For the name of the config file enter nginx.conf and for the mount path enter /etc/nginx/nginx.conf and click create
Then click the edit button beside the name of the config file and you will be directed to the online editor.
On the editor page, on the left side of the screen click nginx.cof to open it. Copy the following configuration to this file and click save. DON’T forget to change the server_name and proxy_pass (hostname of the main source) items. Find server_name if deploy sub-section which is the URL of your project (omit HTTPS) and find proxy_pass also in deploy section.
worker_processes 8;
user root;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {}
http {
include mime.types;
server {
listen 80;
server_name django-examplelhqq.eu-ckhzajsnkv.doprax.com; # copy Your server name here
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_pass http://yc2e1q5zz8aep:5000; # copy main hostname here
}
location /static/ {
alias /app_data/static/;
}
}
}Now the incoming requests will be proxied to the main source and static files will be directly served by Nginx. Now it is time to deploy the project.
Deploy the Django project
Now open the deploy sub-section and you will see the schematic of all the components that we have so far created. It should look like this:
Click the run button and the deployment is done. It will first build the source code and then it will launch the project. The services will be run one by one. The first time that you run the project you may encounter a problem with connectivity with Mysql since it may take up to a minute to initialize it. This problem is fixable by simply restarting the project. Also, by some shell script magic, you can make sure that the deployment procedure waits until the MySQL is up and running.
Create superuser admin
To access the admin interface of your project you need to create a superuser. And to do that you need shell access to the project. When the deployment procedure is finished and the status of the main source and all services turned running, you can access the shell by clicking on the Open Shell button. Let’s do that. Click on the Open Shell button as it is shown in the image below:
A new shell window will appear on the screen and you will be logged in as the root user. This is a pseudo-shell, and not a real ssh client since doprax does not run ssh client on your container code for security reasons. The current working directory is /app since it has been defined as such in our Dockerfile. Remember that our virtual environment path is /venv/ so we need to activate our virtual environment to be able to execute any Django commands.
Now you can do everything that you need to do with the Django project including, making the migrations, do the migrations, create the superuser, and other commands. Let’s create a superuser first.
source /venv/bin/activate
python manage.py createsuperuser
It will ask you to choose a username and enter your email address. Then it will ask you to set a password and repeat the password. Now you can use this username and password to log in to your admin page. Go to your_project_url/admin and login using these credentials. Django admin will look like this. (note that we have not created any models yet and items in the admin are default Django models.)
Django migrations
Django migrations are import part of the Django deployment cycle. Migrations keep models and the database in sync. It means that when you first create a model, it will create a table in the database according to the specification of your model. Then if you change your model in future, you can apply these changes to the database tables using migrations. To do the migrations, you need to access the shell and activate the virtual environment. Open the shell (described in the previous section) and enter the following to create the migrations files.
source /venv/bin/activate
python manage.py makemigrationsThis command will create migrations files inside the app folders. Now you can apply the migrations using the following command.
python manage.py migrate