Apache Pulsar Ready for Prime Time

(Jurik Peter/Shutterstock)

A streaming and messaging system aimed at stream processing has been elevated from incubator to “top-level project” status by the Apache Foundation.

Apache Pulsar was released as open source by Yahoo in 2016 and has since been promoted by its creators who went on to form Streamlio, the fast data platform startup. The Apache Foundation released the 2.0 version of Apache Pulsar in June, elevating it to operational status this week. Promoters note that the publish-and-subscribe messaging system has run in production at Yahoo for more than three years, delivering millions of messages per second across a myriad of topics.

“The last year has seen the evolution of Pulsar from a its original messaging core into an integrated platform for data in motion,” said Matteo Merli, co-founder of Streamlio and champion of the open source project.

Apache Pulsar is touted as a highly scalable, low-latency messaging platform running on commodity hardware. Besides Yahoo (NASDAQ: AABA), current enterprise users include Zhaopin Ltd., the Chinese online recruitment service. Zhaopin said Apache Pulsar addresses “the shortcomings of existing messaging systems, such as message durability, low latency.”

Other early enterprise users said they are using the messaging system as a bridge between public and private clouds as they roll out hybrid cloud strategies. Other early uses include stream processing and analysis of industrial Internet of Things sensor data. Most emerging use cases seek to move beyond slow batch processing, Pulsar supporters said.

Steamlio and other Pulsar backers note that streaming and messaging platforms help connect fast-moving and diverse data with applications. Steamlio, Palo Alto, Calif., said Pulsar recently scored highest for performance and scaling in OpenMessaging benchmark testing. (Streamlio’s Merli is also a founding member of the OpenMessaging community.) The startup further claimed a seven-fold throughput increase when testing Pulsar against Apache Kafka.

“We saw that making Pulsar an open source project would not only enable broader adoption, but also accelerate innovation based on Pulsar’s core architecture,” Merli noted in a blog post.

The messaging service also is being positioned as a data fabric that connects the network edge to the cloud or to the data center on a common platform. That capability, Merli claimed, would enable real-time business transactions that require low-latency processing and data analytics.

With its quick rise within the open source community, Pulsar is now set to challenge industry de facto standard Apache Kafka, the distributed, open-source software that emerged from LinkedIn and is backed by Confluent. Kafka has garnered millions of downloads since it was released more than seven years ago.

Recent items:

Streamlio Claims Pulsar Performance Advantages Over Kafka

Yahoo’s New Pulsar: A Kafka Competitor