While using Postgres as a message queue might seem like an attractive and simple solution, it often leads to serious performance issues, database bloat, and eventually system failures, making it a critical single point of failure that will cause headaches down the line.
Thinking of using Postgres for your message queue? You might want to reconsider. While it seems like a simple solution, relying on Postgres for this task can create a critical single point of failure that will haunt you later, especially when your system is under stress. What this means for you is that apparent simplicity can cost you a lot of sleep and effort when things go wrong.
Many developers are initially drawn to the idea of using Postgres for all their needs. The advice, 'Just use Postgres,' sounds great at first because it avoids complex setups with many different services. You can deploy a single monolith and sleep soundly, or so it seems. The appeal comes from features like 'SELECT ... FOR UPDATE SKIP LOCKED,' which allows you to reserve jobs with no extra infrastructure, no separate brokers, just one line of code. It looks like you're simplifying things, but you're actually hiding a critical component inside your most important, least replaceable database.
However, this convenience comes at a cost. Postgres uses a system called MVCC, which means it creates a new row for every update. A message queue constantly modifies its table: claiming a job, completing it, or retrying it — each action creates more data. This constant modification leads to 'table bloat' and 'index fragmentation.' Your database's automatic cleanup process (autovacuum) struggles to keep up with the changes. Experts like Brandur Leach (formerly of Stripe) and Gunnar Morling have highlighted these issues. They point out that long-running consumer transactions and high change rates inevitably lead to MVCC bloat and a build-up of transaction logs (WAL pile-up), with vacuum often losing the race.
Essentially, your message queue slowly damages the very database it's stored in. This setup also limits your system's throughput. Benchmarks show that Postgres, when used as a queue, struggles significantly. For example, a test on May 15, 2023, found it maxed out at only about 660 messages per second with a 1KB payload, and a publish latency of 38ms. In comparison, dedicated message brokers like RabbitMQ can handle significantly more messages.
So, while the simplicity of 'just using Postgres' is attractive, it often creates a bottleneck and a single point of failure within your most vital system. When your queue gets backed up, your entire database is on fire – and they are the same problem. For anything critical or high-volume, it's better to use dedicated tools for message queuing, saving you from a painful 3 AM pager alert.
Many developers are initially drawn to the idea of using Postgres for all their needs. The advice, 'Just use Postgres,' sounds great at first because it avoids complex setups with many different services. You can deploy a single monolith and sleep soundly, or so it seems. The appeal comes from features like 'SELECT ... FOR UPDATE SKIP LOCKED,' which allows you to reserve jobs with no extra infrastructure, no separate brokers, just one line of code. It looks like you're simplifying things, but you're actually hiding a critical component inside your most important, least replaceable database.
However, this convenience comes at a cost. Postgres uses a system called MVCC, which means it creates a new row for every update. A message queue constantly modifies its table: claiming a job, completing it, or retrying it — each action creates more data. This constant modification leads to 'table bloat' and 'index fragmentation.' Your database's automatic cleanup process (autovacuum) struggles to keep up with the changes. Experts like Brandur Leach (formerly of Stripe) and Gunnar Morling have highlighted these issues. They point out that long-running consumer transactions and high change rates inevitably lead to MVCC bloat and a build-up of transaction logs (WAL pile-up), with vacuum often losing the race.
Essentially, your message queue slowly damages the very database it's stored in. This setup also limits your system's throughput. Benchmarks show that Postgres, when used as a queue, struggles significantly. For example, a test on May 15, 2023, found it maxed out at only about 660 messages per second with a 1KB payload, and a publish latency of 38ms. In comparison, dedicated message brokers like RabbitMQ can handle significantly more messages.
So, while the simplicity of 'just using Postgres' is attractive, it often creates a bottleneck and a single point of failure within your most vital system. When your queue gets backed up, your entire database is on fire – and they are the same problem. For anything critical or high-volume, it's better to use dedicated tools for message queuing, saving you from a painful 3 AM pager alert.
