...
Just my blog

Blog about everything, mostly about tech stuff I made. Here is the list of stuff I'm using at my blog. Feel free to ask me about implementations.

Soft I recommend
Py lib I recommend

I'm using these libraries so you can ask me about them.

WSGI mod, Python, Django, Celery and a nice fella virtualenv

Old

Here I'll show how to use Python (3.8.0 in the current example, but legit for latest 3.+ versions, not for os.name == 'NT') and virtualenv to setup WGSI enabled website on Django (or any) with implemented Celery + Flower + RabbitMQ. This example is my own path used for automated test framework: Octopus (not available now, but have dev branch opened: Lobster)

 

Here only configurations and setup for Python 3.8.0, virtualenv, WSGI, Django, Celery (worker and beat services), Flower and that's it. Later I'll show Django + Celery configuration when they run together (sort of).

 

You may want to install Python at first: Read

 

When your modern and fancy python is installed, we'll make a virtual environment for our website. Never underestimate a power of virtualenv as I did, because since you start to use it, you will never understand how did you work without them before.

 

Best utility 'virtualenv'

 

Install virtualenv module using your freshly installed python, for system user, or root:

 

# https://virtualenv.pypa.io/en/latest/reference/#virtualenv-command
pip3 install virtualenv

 

It should be finished without any restrictions and errors, as usual, pip module. If you see an error while running 'virtualenv' like 'error while loading shared libraries' or something similar to 'bad interpreter' then your python installed incorrectly or system still have older links to python (not 2.7, but maybe something older than you installed) so check and uninstall all older versions (if you do not need them) and check the guide again: Here.

 

Then, you can 'cd' to the directory where you want to set up your virtual environment. Or just insert the path to that place.

 

virtualenv --python=python3 venv --system-site-packages
source venv/bin/activate

 

--system-site-packages - is useful options, because it allows virtual python to use system python site-packages. Of course, you should consider to not use trashy packages in the system to keep your enviroments secure.
--python=python3 - specify system python which will be used to build virtual environment. It's python 3.8.0 for the current topic.

 

When everything was installed correctly, you can activate virtual ens with:

 

source venv/bin/activate
(venv) /var/www/web/

 

Now you're ready to move forward:

 

Installing mod_wsgi using pip

 

Once you installed the virtual environment and activated it, you can start to install requirements. The initial requirement is mod_wsgi.

 

pip install mod_wsgi

 

There shouldn't be any issues with this installation if you run make python correctly, because it uses almost the same way to build WSGI as we used to build Python. Only issues I faced was related to improperly set of Python in the previous step.

 

WSGI Mod on virtualenv

 

When mod_wsgi installed, you could find it in a path like this:

 

/var/www/web/venv/lib/python3.8/site-packages/mod_wsgi/server/mod_wsgi-py38.cpython-38-x86_64-linux-gnu.so

 

So we'll use this path to setup Apache\HTTPD conf for your site, and Celery+Flower services later. Yes, you can setup multiple envs with different python version, so with different mod_wsgi and apache conf, so have different sites with different versions of environment and libraries.

 

Apache\https conf with mod_wsgi

 

The simplest apache conf as I use:

 

[code]
# vi /etc/httpd/conf.modules.d/10-wsgi.conf
# vi /etc/httpd/conf.d/octopus.conf
# Web site at /var/www/octopus/..
# Python scripts at /var/www/octopus/octopus/..
# /usr/lib64/python3.6/site-packages/mod_wsgi/server/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so
# "/var/www/octopus/octo/lib/python3.4/site-packages/mod_wsgi/server/mod_wsgi-py34.cpython-34m.so"
# "/var/www/octopus/octo/lib/python3.4/site-packages/mod_wsgi/server/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so"
# LoadModule wsgi_module "/var/www/octopus/octo/lib/python3.4/site-packages/mod_wsgi/server/mod_wsgi-py36.cpython-36m-x86_64-linux-gnu.so"

LoadModule wsgi_module "/var/www/octopus/venv/lib/python3.8/site-packages/mod_wsgi/server/mod_wsgi-py38.cpython-38-x86_64-linux-gnu.so"

WSGIPythonHome "/var/www/octopus/venv"
TimeOut 60

#WSGIPythonHome /var/www/octopus/octo/lib
#WSGIPythonPath /var/www/octopus/octo/lib

Alias /static/      "/var/www/octopus/static/"
Alias /octicons/    "/var/www/octopus/static/octicons/svg"

Alias /css/         "/var/www/octopus/static/css/"
Alias /js/          "/var/www/octopus/static/js/"
Alias /templates/   "/var/www/octopus/static/templates/"
Alias /fonts/       "/var/www/octopus/static/fonts/"

<Directory /var/www/octopus/static>
    Require all granted
</Directory>

<Directory /var/www/octopus/static/octicons/svg>
    Require all granted
</Directory>

<VirtualHost *:80>
    ServerName vl-aus-tkudev-37.bmc.com
    ServerAlias www.vl-aus-tkudev-37.bmc.com
    ServerAdmin oleksandr_danylchenko_cw@bmc.com
    DocumentRoot /var/www/octopus/

    #ErrorLog "|/usr/sbin/rotatelogs /var/log/octopus/cargo.error.%Y-%m-%d.log 86400"
    #CustomLog "|/usr/sbin/rotatelogs /var/log/octopus/cargo.access.%Y-%m-%d.log 86400" combined
    ServerSignature On

    <Directory /var/www/octopus/octo>
        <Files wsgi.py>
            Require all granted
        </Files>
    </Directory>
    WSGIDaemonProcess octopus python-path=/var/www/octopus/venv:/var/www/octopus/venv/lib/python3.8/site-packages
    WSGIProcessGroup octopus
    WSGIScriptAlias / /var/www/octopus/octo/wsgi.py
</VirtualHost>
[/code] 

 

And wsgi.so module:

 

# vi /etc/httpd/conf.modules.d/10-wsgi.conf
# vi /etc/httpd/conf.d/octopus.conf
# LoadModule wsgi_module modules/mod_wsgi.so
# in VIRTENV
LoadModule wsgi_module "/var/www/web/venv/lib/python3.8/site-packages/mod_wsgi/server/mod_wsgi-py38.cpython-38-x86_64-linux-gnu.so"
WSGIPythonHome "/var/www/my_site/venv"

 

LoadModule wsgi_module "/var/www/web/venv/lib/python3.8/site-packages/mod_wsgi/server/mod_wsgi-py38.cpython-38-x86_64-linux-gnu.so" as before - is the path to currently installed wsgi module in your virtual enviroment.
WSGIPythonHome "/var/www/web/venv" - actual path to your newly installed virtual enviroment.
Files wsgi.py - allowance to execute django's WSGI plugin.
WSGIDaemonProcess web python-path=/var/www/web/venv:/var/www/web/venv/lib/python3.8/site-packages - where 'web' is the user allowed to run python in directory passed
WSGIScriptAlias / /var/www/web/wsgi.py - and wsgi script\plugin of django itself.

 

File: 10-wsgi.conf seems like have a duplicate of WSGI init, I used it as a backup to switch conf to system main if something goes wrong, so historically it's also shown here.

 

More about Django on WSGI is in the official doc: here.

 

Celery + Flower and 'systemd' daemonization

 

I have seen tonnes of topics about Celery on Centos daemonization and read all Celery docs from the start till the and backward, but still had much pain to set it up properly. This is the config I use on my machines and I can recommend everybody as a pain-less approach.

 

GitHub: celery-helpers

 

You can find all config on my Github. Briefly:

 

/etc/default/celeryd.conf

 

Used to deamonize celery beat service, task planner aka cron.
Should use /var/www/web/venv/bin/celery beat as the path to your virtual environment and installed celery module (pip install celery)

 

celeryd.conf

 

This is the settings\config file for beat service.
CELERY_BIN="/var/www/web/venv/bin/celery" - also virt env path
CELERY_APP="web.web_celery:app" - path to your django's core folder where you can store additional celery configurations integrated with django. As in docs: here.

 

Then, check the repo for full info, place files as described in their heads and your celery + flower is ready!

 

Celery on Windows: Dev purposes only

 

This is another setup for celery on windows, for development purposes, because of coding with remote server synchronization is not really comfortable, so you can install a limited version of celery (which can produce unrelated errors from time to time) just to easier development on a local machine.

 

Setup is also on GitHub, but here's short description:

 

D:\Projects\PycharmProjects\lobster\venv\Scripts\activate.bat && celery -A octo.octo_celery:app worker --pool=eventlet --loglevel=ERROR --concurrency=1 -E -n alpha@tentacle
# Path D:\Projects\PycharmProjects\lobster\z_DEV\services\celery_services\windows\proj_H\alpha.bat
# Startup directory: D:\Projects\PycharmProjects\lobster\
# Output: D:\Projects\PycharmProjects\lobster\z_DEV\services\celery_services\windows\logs\alpha_out.log
# Error: D:\Projects\PycharmProjects\lobster\z_DEV\services\celery_services\windows\logs\alpha_err.log

 

Install eventlet via pip for your virtual environment.
Create a worker_name.bat somewhere in the project.
Add the path to 'venv' activate.bat file in your worker_name.bat
Add celery options:
celery -A web.web_celery:app - celery exec and your project celery all
worker --pool=eventlet - make detachable
--loglevel=ERROR - you know
--concurrency=1 - it's for run only 1 task at once
-E - can's say now, why would I use it?
-n - this too
alpha@tentacle - the name of worker\queue and host for RabbitMQ
Most settings same as in docs.

 

Use NSSM to run as service

 

Link.
It allows you to install celery service. Use that bat you create as the path.

 

Path to celery worker bat and work dir

 

Add outputs for ERR and STDOUT

 

Make logs rotate when you restart service.

 

And now you have a good local development environment. You can restart services easy in Task Manager:

 

 

That's all folks!