How to setup Laravel queue on a shared hosting

Laravel provides a queue functionality out of the box. That’s really great. And your application is up and running pretty fast and decent. But it's the server side of things that can make it difficult: queue server setup and monitoring for instance. The challenge is using queues on a shared hosting where you cannot control the server environment.

In this post, I’d like to share you our way of achieving queue functionality on shared hosting.

Why do we use queues?

A queue is a programming concept which stores a collection of scripts, waiting to be processed. At Think Tomorrow we use queues for time-consuming jobs or large reads from a third-party API. Like requesting a sitemap generation or a backup, subscribing form submissions to Mailchimp, or syncing Hubspot contacts.
In short, you don’t want to let your user wait for these side effects to complete during their request. Putting these scripts on a queue will offload the script to another process and allows the visitor to continue his request flow immediately. Queues give your users a much nicer experience. 

Worth the wait. 

With a queue system in place, time becomes an important aspect in the request/response development. There is no initial script response during the user's request. 
This means that the result is unknown and the communication towards the user should reflect that. 

Let's try to visualize a simple request flow with a queued script:

Consider this example: Without a queue, when a user requests a large excel export of their product catalog, they’ll have to wait in the browser for the server to finish and return the response. This can easily take seconds, even minutes. When utilizing a queue, the user gets an immediate response saying that the export is being generated. Once the export is ready, the export result will be mailed to the user.

Shared hosting monitoring

A queue should always be ready to line up new jobs. So it needs to be a long-running process. Ideally, the queue works with a process manager such as supervisor. This takes care of starting the queue process and restarts it when necessary. 
On a shared hosting, there is no such monitor available. There is usually also no Redis or beanstalk available as well. Since we really need to use queues in most of our projects, we went out to find a second to the best alternative: We use database queues which are available by default in Laravel and set up a cron task for the monitoring. 
The scripts can be viewed here: 
https://gist.github.com/BenCavens/810758e74718a981c4cd2d2cf532407e

Queue process 

Every minute, the cron schedule runs the php artisan queue:work —stop-when-empty command. This command runs any available queued jobs. If there aren’t any, this script stops immediately. You might be wondering: why not leaving the --stop-when-empty option out and let the worker stay active? This way the jobs will get processed instantly no? Yes but remember that this command gets run every minute. And since the cronjob triggers this command every minute, it is essential that we don’t cause any overlapping processes. With the previous command still running, you quickly reach the end of the available PHP processes.

And just for a precaution, we have added an hourly queue:restart as well. 

Failing queues

We have a queue monitoring command that checks if there are any job records still present in the database table. If there are still records lingering after five minutes, this means that the records have not been processed and we consider the queue to have stalled. In that case, an error message will be reported. 
Besides that, we listen to the Laravel Queue::failing event and report to our error tracking in case it gets triggered. 

Not fast though

The major downside of cron is that the highest frequency of polling of every minute. With a queue driver like redis, the job is immediately processed if possible. But with our cron based solution, there is only a queue check every minute. If you’re okay with this constraint, then this solution might be good for you as well.

Conclusion

This was a brief introduction on how we set up queues on our shared hostings. I hope this was somewhat interesting to you and maybe even can convince you to use queues, when one of your projects lives on a shared hosting.