How to Get Beanstalkd Queue to Work for PHP

Unable to get Beanstalkd Queue to work for PHP

SOLUTION FOUND:

After some more research, I have managed to get it to work! A decent amount was missing to get to that point. The process was the following:

1. Execute sudo apt-get install beanstalkd in the linux terminal to install beanstalkd.
2. Execute sudo apt install composer to install composer, which is the program recommended to be used to install pheanstalk.

3. Create a composer.json file that will let composer know what library to download and what version of said library. For instance:

   {
     "require": {
       "pda\pheanstalk": "2.1.1"
     }
   }
   

4. Execute composer install in the linux terminal. This has to be done in the same folder as the composer.json file.

5. Include the necessary code that will initiate the Pheanstalk class, and use it as documented. And that is it! Sample code would be as follows:

   <?php
   
   require_once('vendor/autoload.php');//require the autoload file provided by
                                       //composer
   
   //Initiate an instance of the Pheanstalk class
   $pheanstalk = new Pheanstalk_Pheanstalk('127.0.0.1');
   
   //adding a job to queue/tube testtube:
   $pheanstalk->useTube('testtube')->put('message');
   
   //obtaining the job by a worker:
   $job = $pheanstalk->watch('testtube')->ignore('default')->reserve();
   
   echo $job->getData;//outputting the message
   
   $pheanstalk->delete($job);//deleting the job from the queue.
   
   ?>
   

Laravel + Beanstalkd: How to run queue:listen as a service

You should use something like Supervisor to run the queue in production. This will allow you to run the process in the background, specify the number of workers you want processing queued jobs and restart the queue should the process fail.

As for the queue you choose to use, that's up to you. In the past I've used Beanstalkd locally installed on an instance and Amazon SQS. The local instance was fine for basic email sending and other async tasks, SQS was great for when the message volume was massive and needed to scale. There are other SaaS products too such as IronMQ, but the usual reason people run into issues in production are because they're not using Supervisor.

You can install Supervisor with apt-get. The following configuration is a good place to start:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /home/forge/app.com/artisan queue:work --sleep=3 --tries=3
autostart=true
autorestart=true
numprocs=8
stdout_logfile=/home/user/app.com/worker.log

This will do the following:

• Give the queue worker a unique name
• Run the php artisan queue:work command
• Automatically start the queue worker on system restart and automatically restart the queue workers should they fail
• Run the queue worker across eight processes (this can be increased or reduced depending on your needs)
• Log any output to /home/user/app.com/worker.log

To start Supervisor, you'd run the following (after re-reading the configuration/restarting):

sudo supervisorctl start laravel-worker:*

The documentation gives you some more in-depth information about using Supervisor to run Laravel's queue processes.

How to fire Laravel Queues with beanstalkd

This is actually a really badly documented feature in Laravel.

What you actually need to do is have the JobClass.php in a folder that is auto-loaded, I use app/commands, but they can also be in app/controllers or app/models if you like. And this function needs to have a fire event that takes the $job and $data argument.

To run these, simply execute php artisan queue:listen --timeout=60 in your terminal, and it will be busy emptying the queue, until it's empty, or it's been running for longer then 60 seconds. (Small note: The timeout is the time-limit to start a queue, so it may run for 69 seconds if 1 job takes 10 seconds.

If you only want to run 1 job (perfect for testing), run php artisan queue:work

There are tools like Supervisord that make sure your job handlers keep running, but I recommend to just make a Cron task that starts every X minutes based on how fast the data needs to be processed, and on how much data comes in.

How to set up Beanstalkd with PHP

1. If the worker isn't taking too long to fetch the feed, it will be fine. You can run multiple workers if required to process more than one at a time. I've got a system (currently using Amazon SQS, but I've done similar with BeanstalkD before), with up to 200 (or more) workers pulling from the queue.
2. A single worker script (the same script running multiple times) should be fine - the script can watch multiple tubes at the same time, and the first one available will be reserved. You can also use the job-stat command to see where a particular $job came from (which tube), or put some meta-information into the message if you need to tell each type from another.
3. A good example of running a worker is described here. I've also added supervisord (also, a useful post to get started) to easily start and keep running a number of workers per machine (I run shell scripts, as in the first link). I would limit the number of times it loops, and also put a number into the reserve() to have it wait for a few seconds, or more, for the next job the become available without spinning out of control in a tight loop that does not pause at all - even if there was nothing to do.

Addendum:

1. The shell script would be run as many times as you need. (the link show how to have it re-run as required with exec $@). Whenever the php script exits, it re-runs the PHP.
2. Apparently there's a Djanjo app to show some stats, but it's trivial enough to connect to the daemon, get a list of tubes, and then get the stats for each tube - or just counts.

---

Related Topics