Accident.com
July 2023

Using Pub/Sub With Google Cloud Function as Task Queue Processor

How we solved scalability, reliability and asynchronous communication for a client using Google Cloud's Pub/Sub and Cloud functions

Services

Job Queues

Platforms

Web

Using Pub/Sub With Google Cloud Function as Task Queue Processor

Project Overview

Job Queues- are used to offload some of those time-consuming, load intensive tasks processing away from your main threads.

Earlier, when we were working on a project with a tech stack fully based on the CodeIgniter PHP framework, we found that CodeIgniter does not offer built-in support for queues like Laravel. Therefore, we came up with a solution to implement an in-house job queue system.

The Approach

To avoid synchronous calls for processing long waiting tasks, we implemented a solution where tasks are stored independently and processed in the background at a later time.

Approach Workflow

As previously mentioned, we started storing jobs independently, and decided to use a table to hold each task record's payload - the input for the task to be processed. We added fields to track each job, such as the creation time, status, and task response. This table serves as a queue for all the tasks. To process the tasks, we configured a cron job to run in the background, which picks up a set number of tasks to process, and executes them in a first-in, first-out (FIFO) manner.

Initially, this approach was effective. However, we soon encountered a problem - as we had the application and the cron processor running on the same instance the tasks queue table began to bloat, and the jobs were being executed too slowly.

Google Cloud Function & Pub/Sub To Rescue 🚀

Pub/Sub Overview
Pub/Sub
Is a messaging service that allows different applications to communicate with each other in a reliable and scalable way. It works by creating "topics" where messages can be published, and "subscriptions" that receive those messages.

The following are several well-known frameworks that support pub/sub:

  • Redis
  • Apache Kafka
  • RabbitMQ

Why Google Pub/Sub?

Compared to other pub/sub systems out there, Google Pub/Sub offers several key advantages. Firstly, it is fully managed, which means that Google handles the underlying infrastructure, so you don't have to worry about maintaining and scaling your own messaging system. Secondly, it provides a highly reliable and scalable messaging service that can handle millions of messages per second, making it suitable for use cases with high throughput requirements. Finally, it integrates seamlessly with other Google Cloud Platform services, making it easy to build powerful and scalable applications that can take advantage of Google's entire ecosystem.

Solution Workflow

We decided to use Google Cloud Functions, a serverless computing service that allows you to run your code in response to different events. By combining it with Pub/Sub, we were able to use Cloud Functions to process messages that were published to a topic. This approach solved our problem in a unique way. We wanted to eliminate the task queue table and track jobs in a different location, but migrating the task queue storage wasn't feasible. Instead, we opted to keep using the same task queue table to hold all the tasks.

To implement this solution, we first created the task-queue topic in Google Pub/Sub, which listens for any incoming messages. We then removed our cronJob processor, which was running every minute to execute jobs in batches, as it was too slow. Instead, we wrote a Google Cloud Function that had the same implementation as the cronJob script but would only run when we published messages to our task-queue topic. The function would pick up the task using the ID from the message payload, process it, and update the task record.

Overall, this solution allowed us to improve our job execution performance significantly. The use of Google Cloud Functions eliminated the need for a dedicated processor to execute jobs, resulting in a more scalable and efficient system. Additionally, the Pub/Sub messaging service enabled us to decouple the task queue from the application, leading to a cleaner and more manageable codebase.

Queueing messages into Pub/Sub
New Workflow with Pub/Sub and Google Cloud Function

That’s It! Thanks for reading! If you ever want to chat on building awesome cloud stuff, contact us at team@intuio.io

References:

https://cloud.google.com/pubsub

https://cloud.google.com/pubsub/docs/overview

https://cloud.google.com/functions/docs/calling/pubsub

Testimonials

Don’t take our word for it.
See what our past clients say.

“Best Development Agency”

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Non dolor urna, ullamcorper feugiat elementum non et erile tortor. Dignissim eu faucibus et, congue. Tempus commodo vitae consequat eget et quis non tincidunt idert.

John Carter - Code Webflow Template
John Carter
Head of Product at Twitter
Company Logo - Code Webflow Template

“One of the best agencies out there”

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Non dolor urna, ullamcorper feugiat elementum non et erile tortor. Dignissim eu faucibus et, congue. Tempus commodo vitae consequat eget et quis non tincidunt idert.

Will Spark - Code Webflow Template
Will Spark
Engineering VP at Google
Company Logo - Code Webflow Template

“True Experts in Web Development”

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Non dolor urna, ullamcorper feugiat elementum non et erile tortor. Dignissim eu faucibus et, congue. Tempus commodo vitae consequat eget et quis non tincidunt idert.

Sam Parker - Code Webflow Template
Sam Parker
VP of Marketing at Twitter
Company Logo - Code Webflow Template

“Best Development Agency”

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Non dolor urna, ullamcorper feugiat elementum non et erile tortor. Dignissim eu faucibus et, congue. Tempus commodo vitae consequat eget et quis non tincidunt idert.

John Carter - Code Webflow Template
John Carter
CEO at Facebook
Company Logo - Code Webflow Template

“One of the best agencies out there”

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Non dolor urna, ullamcorper feugiat elementum non et erile tortor. Dignissim eu faucibus et, congue. Tempus commodo vitae consequat eget et quis non tincidunt idert.

Will Spark - Code Webflow Template
Will Spark
Engineering VP at Google
Company Logo - Code Webflow Template

“True Experts in Web Development”

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Non dolor urna, ullamcorper feugiat elementum non et erile tortor. Dignissim eu faucibus et, congue. Tempus commodo vitae consequat eget et quis non tincidunt idert.

Sam Parker - Code Webflow Template
Sam Parker
VP of Marketing at Twitter
Company Logo - Code Webflow Template
   Previous
Next   
Let’s get in touch

Ready to start working together with us?

 Email Icon - Code Webflow Template

Send us a message!

Don't miss out on the opportunity to take your business to new heights. Contact us now and let's embark on a journey of success together.

Contact us
Development Execution - Code Webflow Template

Browser our packages

From startups seeking a strong foundation to established enterprises aiming to optimize operations, our scalable and flexible packages cater to businesses of all sizes.

Browse packages