How to run PHP scripts in parallel

Like many others, PHP is my language of choice for backend development. It's easy to learn and is a great starting point if you're just getting started with web development.

PHP is designed to be single threaded so each request / page will use a single CPU core and a single thread. If another request comes along from your web server at the same time PHP will use a different thread for this request. Single threaded languages have their advantages and disadvantages, one pro is that it's easy to write (e.g you don't have to monitor tasks running parallel) but the downside of this is that you always have to wait for tasks to finish before you can start another one.

I run Textreel a service for creating videos from audio files either through the web interface or the API. As part of my application I have a PHP script that runs on a cron to generates an image for each frame and then stiches them together to produce the finished video. Ideally I would have used a language that supports mutli-threading like Go or Rust for this part of my application but I don't fancy nor have the time to learn another language from scratch.

We can see this limitation in action by using the htop application which allows you to monitor usages on all of your CPU cores. My development machine has 8 cores but you can see below that when generating a video only 1 core is being used at nearly full capacity.

I've done my best to optimise the video generation code to be as fast as possible and generally speaking converting a 30 seconds audio clip into a 25 frames per second video takes X seconds which isn't great, particulary for API users.

To make use of all of my servers processing power I need to change the application logic so I can run multiple PHP scripts in parallel. Because there's no way to communicate between threads in PHP you'll need to use a database or the file system to do this (if you need too). I decided to keep it simple and allocate a specific range of frames for each thread to generate, then after they've all completed just check I have the correct number of frames.