How to Run Celery on Windows

How to run celery on windows?

It's done the same way as in Linux. Changing directory to module containing celery task and calling "c:\python\python" -m celery -A module.celery worker worked well.

get results of running a task on Celery (Windows 10 32 bits)

How to run celery on windows?

I always had issues with Celery in Windows even though technically you can run it, by using the --pool=solo command you have alredy indicated. The document is a bit misleading in this aspect and I was always getting inconsistant functionality. I ended up getting WSL for Windows, ubuntu 20 and I run redis (or rabbitmq) all day with processes and threads and no issues. Plus it more simulates a production environemnt since you most likely wont be running just one worker.

This is my example config for testing

app = Celery ('app', broker="redis://", backend="redis://", include=['app.tasks'])

Apparently support was removed in 4.0 but re-added in 4.3. Im sure someone with a windows 10 machine, running celery on windows my be able to chime in.

How to create Celery Windows Service?

I got the answer from another website. Celeryd (daemon service for Celery) runs as a paster application, searching for 'Paster Windows Service' lead me here. It describes how to run a Pylons application as a Windows Service. Being new to paster framework and hosting python web services, it didn't cross my mind to check it at first. But that solution works for Celery with a slight change here and there in the script.

I've modified the script to make it easier for modifying Celery settings. The essential changes are:

1. Create an INI file with the settings for Celery service (shown below)
2. Create a python script to create a Windows service.

INI file settings (celeryd.ini):

[celery:service]
service_name = CeleryService
service_display_name = Celery Service
service_description = WSCGI Windows Celery Service
service_logfile = celeryd.log

Python script to create Windows Service (CeleryService.py):

"""
The most basic (working) Windows service possible.
Requires Mark Hammond's pywin32 package.  
Most of the code was taken from a  CherryPy 2.2 example of how to set up a service
"""
import pkg_resources
import win32serviceutil
from paste.script.serve import ServeCommand as Server
import os, sys
import ConfigParser

import win32service
import win32event

SCRIPT_DIR          = os.path.abspath(os.path.dirname(__file__))
INI_FILE            = 'celeryd.ini'
SERV_SECTION        = 'celery:service'
SERV_NAME           = 'service_name'
SERV_DISPLAY_NAME   = 'service_display_name'
SERV_DESC           = 'service_description'
SERV_LOG_FILE       = 'service_logfile'
SERV_APPLICATION    = 'celeryd'
SERV_LOG_FILE_VAR   = 'CELERYD_LOG_FILE'

# Default Values
SERV_NAME_DEFAULT           = 'CeleryService'
SERV_DISPLAY_NAME_DEFAULT   = 'Celery Service'
SERV_DESC_DEFAULT           = 'WSCGI Windows Celery Service'
SERV_LOG_FILE_DEFAULT       = r'D:\logs\celery.log'

class DefaultSettings(object):
    def __init__(self):
        if SCRIPT_DIR:
            os.chdir(SCRIPT_DIR)
        # find the ini file
        self.ini = os.path.join(SCRIPT_DIR,INI_FILE)
        # create a config parser opject and populate it with the ini file
        c = ConfigParser.SafeConfigParser()
        c.read(self.ini)
        self.c = c

    def getDefaults(self):
        '''
        Check for and get the default settings
        '''
        if (
            (not self.c.has_section(SERV_SECTION)) or
            (not self.c.has_option(SERV_SECTION, SERV_NAME)) or
            (not self.c.has_option(SERV_SECTION, SERV_DISPLAY_NAME)) or
            (not self.c.has_option(SERV_SECTION, SERV_DESC)) or
            (not self.c.has_option(SERV_SECTION, SERV_LOG_FILE))
            ):
            print 'setting defaults'
            self.setDefaults()
        service_name = self.c.get(SERV_SECTION, SERV_NAME)
        service_display_name = self.c.get(SERV_SECTION, SERV_DISPLAY_NAME)
        service_description = self.c.get(SERV_SECTION, SERV_DESC)
        iniFile = self.ini
        service_logfile = self.c.get(SERV_SECTION, SERV_LOG_FILE)
        return service_name, service_display_name, service_description, iniFile, service_logfile

    def setDefaults(self):
        '''
        set and add the default setting to the ini file
        '''
        if not self.c.has_section(SERV_SECTION):
            self.c.add_section(SERV_SECTION)
        self.c.set(SERV_SECTION, SERV_NAME, SERV_NAME_DEFAULT)
        self.c.set(SERV_SECTION, SERV_DISPLAY_NAME, SERV_DISPLAY_NAME_DEFAULT)
        self.c.set(SERV_SECTION, SERV_DESC, SERV_DESC_DEFAULT)
        self.c.set(SERV_SECTION, SERV_LOG_FILE, SERV_LOG_FILE_DEFAULT)
        cfg = file(self.ini, 'wr')
        self.c.write(cfg)
        cfg.close()
        print '''
you must set the celery:service section service_name, service_display_name,
and service_description options to define the service 
in the %s file
''' % self.ini
        sys.exit()

class CeleryService(win32serviceutil.ServiceFramework):
    """NT Service."""

    d = DefaultSettings()
    service_name, service_display_name, service_description, iniFile, logFile = d.getDefaults()

    _svc_name_ = service_name
    _svc_display_name_ = service_display_name
    _svc_description_ = service_description

    def __init__(self, args):
        win32serviceutil.ServiceFramework.__init__(self, args)
        # create an event that SvcDoRun can wait on and SvcStop
        # can set.
        self.stop_event = win32event.CreateEvent(None, 0, 0, None)

    def SvcDoRun(self):
        os.chdir(SCRIPT_DIR)
        s = Server(SERV_APPLICATION)
        os.environ[SERV_LOG_FILE_VAR] = self.logFile
        s.run([self.iniFile])
        win32event.WaitForSingleObject(self.stop_event, win32event.INFINITE)

    def SvcStop(self):
        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
        #win32event.SetEvent(self.stop_event)
        self.ReportServiceStatus(win32service.SERVICE_STOPPED)
        sys.exit()

if __name__ == '__main__':
    win32serviceutil.HandleCommandLine(CeleryService)

To install the service run python CeleryService.py install and then python CeleryService.py start to start the service. NOTE: These commands should be run in command-line with administrator rights.

If the service needs to be removed, run python CeleryService.py remove.

I was trying to host Celery as part of enhancing my RhodeCode installation. This solution seems to work. Hope this will help someone.

How to test celery with django on a windows machine

Celery (since version 4 as pointed out by another answer) does not support Windows (source: http://docs.celeryproject.org/en/latest/faq.html#does-celery-support-windows). Even so, you have some options:

1) Use task_always_eager=True. This will run your tasks synchronously – with this, you can verify that your code is doing what it's supposed to do. Running tasks synchronously works even on Windows.

More info: http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-task_always_eager

2) Use WSL (Windows Subsystem for Linux).

More info: https://learn.microsoft.com/en-us/windows/wsl/install-win10#install-the-windows-subsystem-for-linux

3) Use Docker for Windows. You can run Celery Worker(s) and Celery Beat inside the container.

More info: https://docs.docker.com/docker-for-windows/

Personally, I use option 1) for unit testing and option 2) for development.

---

Related Topics