Using Pub/Sub With Google Cloud Function as Task Queue Processor
How we solved scalability, reliability and asynchronous communication for a client using Google Cloud's Pub/Sub and Cloud functions
Job Queues
Web
Improvement in Background Event Handling
Project Overview
Job Queues- are used to offload some of those time-consuming, load intensive tasks processing away from your main threads.
Earlier, when we were working on a project with a tech stack fully based on the CodeIgniter PHP framework, we found that CodeIgniter does not offer built-in support for queues like Laravel. Therefore, we came up with a solution to implement an in-house job queue system.
The Approach
To avoid synchronous calls for processing long waiting tasks, we implemented a solution where tasks are stored independently and processed in the background at a later time.
Approach Workflow
As previously mentioned, we started storing jobs independently, and decided to use a table to hold each task record's payload - the input for the task to be processed. We added fields to track each job, such as the creation time, status, and task response. This table serves as a queue for all the tasks. To process the tasks, we configured a cron job to run in the background, which picks up a set number of tasks to process, and executes them in a first-in, first-out (FIFO) manner.
Initially, this approach was effective. However, we soon encountered a problem - as we had the application and the cron processor running on the same instance the tasks queue table began to bloat, and the jobs were being executed too slowly.
Google Cloud Function & Pub/Sub To Rescue 🚀
Pub/Sub Overview
Pub/Sub Is a messaging service that allows different applications to communicate with each other in a reliable and scalable way. It works by creating "topics" where messages can be published, and "subscriptions" that receive those messages.
The following are several well-known frameworks that support pub/sub:
- Redis
- Apache Kafka
- RabbitMQ
Why Google Pub/Sub?
Compared to other pub/sub systems out there, Google Pub/Sub offers several key advantages. Firstly, it is fully managed, which means that Google handles the underlying infrastructure, so you don't have to worry about maintaining and scaling your own messaging system. Secondly, it provides a highly reliable and scalable messaging service that can handle millions of messages per second, making it suitable for use cases with high throughput requirements. Finally, it integrates seamlessly with other Google Cloud Platform services, making it easy to build powerful and scalable applications that can take advantage of Google's entire ecosystem.
Solution Workflow
We decided to use Google Cloud Functions, a serverless computing service that allows you to run your code in response to different events. By combining it with Pub/Sub, we were able to use Cloud Functions to process messages that were published to a topic. This approach solved our problem in a unique way. We wanted to eliminate the task queue table and track jobs in a different location, but migrating the task queue storage wasn't feasible. Instead, we opted to keep using the same task queue table to hold all the tasks.
To implement this solution, we first created the task-queue topic in Google Pub/Sub, which listens for any incoming messages. We then removed our cronJob processor, which was running every minute to execute jobs in batches, as it was too slow. Instead, we wrote a Google Cloud Function that had the same implementation as the cronJob script but would only run when we published messages to our task-queue topic. The function would pick up the task using the ID from the message payload, process it, and update the task record.
Overall, this solution allowed us to improve our job execution performance significantly. The use of Google Cloud Functions eliminated the need for a dedicated processor to execute jobs, resulting in a more scalable and efficient system. Additionally, the Pub/Sub messaging service enabled us to decouple the task queue from the application, leading to a cleaner and more manageable codebase.
That’s It! Thanks for reading! If you ever want to chat on building awesome cloud stuff, contact us at team@intuio.io
References:
https://cloud.google.com/pubsub
https://cloud.google.com/pubsub/docs/overview
https://cloud.google.com/functions/docs/calling/pubsub
Don’t take our word for it.
See what our past clients say.
“Best Development Agency”
I'm impressed with the cloud and web development work done by Intuio! Their proficiency and effectiveness beyond my expectations. Strongly advised!
“One of the best agencies out there”
Outstanding assistance from Intuio! They have excellent cloud and mobile development services. delivered precisely and on schedule.
“True Experts in Cloud Development”
Happy with the outcome! With regard to cloud development services, Intuio excelled. dependable and competent service!
“Best Development Agency”
Intuio's rapid POC implementation exceeded expectations, delivering quality results within the stipulated deadline. Highly recommend their expertise!
“Best Professional Team”
I am extremely impressed Intuio's level of expertise and professionalism. Intuio is the best software consulting agency I have worked with.
“True Experts in Web Development”
When it comes to web development, Intuio is unmatched. Great experience always producing outstanding outcomes. Highly recommend Intuio !
Ready to start working together with us?
Send us a message!
Don't miss out on the opportunity to take your business to new heights. Contact us now and let's embark on a journey of success together.
Contact us
Browser our packages
From startups seeking a strong foundation to established enterprises aiming to optimize operations, our scalable and flexible packages cater to businesses of all sizes.
Browse packages