What's new in Apache Pulsar 4.0 LTS since Pulsar 3.0 LTS?​

Pulsar 4.0 includes numerous Pulsar Improvement Proposals (PIPs) since the previous LTS release Pulsar 3.0 in May 2023. These improvements have enhanced the platform's capabilities across multiple areas.

Since there's a lot of new features and improvements, this blog post will only highlight a few of the recent updates. Apache Pulsar contributors are invited to contribute additional blog posts with more details about the new features and improvements.

Enhanced Key_Shared Subscription: Scale Without Compromising Message Order​

Key_Shared subscription is one of Pulsar's most valuable features, enabling organizations to scale their message processing capacity by adding multiple consumers while maintaining strict message ordering based on keys. This capability is crucial for applications requiring both high throughput and ordered processing, such as financial transactions, event processing, and real-time analytics.

In Pulsar 4.0, we've improved the Key_Shared subscription implementation through a significant enhancement with PIP-379 Key_Shared Draining Hashes for Improved Message Ordering. The new design ensures messages with the same key are handled by only one consumer at a time, while eliminating unnecessary message blocking that previously impacted system performance during consumer changes and application restarts.

Operations teams can quickly identify and resolve any Key_Shared ordered message delivery issues through comprehensive troubleshooting metrics in Pulsar topic stats. Future improvements will introduce a REST API that will further simplify troubleshooting by providing direct access to unacknowledged message details and powerful key-based search capabilities for resolving message delivery issues where typically the root cause is in an application that doesn't acknowledge a message and due to message ordering constraints, further messages for the key are blocked. Web based user interfaces and CLI tools can build upon this REST API, allowing also automation for resolving or alerting in operations. Related Key_shared troubleshooting metrics will also be exposed via Prometheus and OTel interfaces in future updates.

You can learn more about the Key_Shared subscription preserving order of message delivery by key and related troubleshooting capabilities in the Pulsar documentation.

Enhanced Secure Docker Image Runtime Based on Alpine and Java 21​

Pulsar 4.0 contains enhancements to its Docker runtime environment, combining the security benefits of Alpine Linux with the performance improvements of Java 21's runtime. PIP-324 introduced in Pulsar 3.3.0 aligns with our commitment to providing a secure, efficient, and resource-optimized platform for messaging workloads.

The new Docker images are now based on Alpine Linux instead of Ubuntu, reducing the image size while improving the security posture.

A key security enhancement is the elimination of CVEs in the base image. While the previous Ubuntu-based images carried 12 Medium/Low CVEs with no available resolution, the new Alpine-based images start with zero CVEs, providing a more secure foundation for production deployments. This improvement is particularly valuable for organizations with strict security requirements and compliance needs.

The Docker images now include Java 21 with Generational ZGC, bringing significant improvements in garbage collection performance. Generational ZGC provides sub-millisecond pause times, better CPU utilization, and improved memory efficiency compared to previous garbage collectors. This translates to more predictable latencies and better resource utilization for Pulsar deployments.

These improvements make Pulsar 4.0's Docker runtime an even more compelling choice for organizations requiring both security and performance in their messaging infrastructure. The combination of Alpine Linux's minimal attack surface and Java 21's advanced garbage collection provides a robust foundation for running Pulsar in containerized environments.

Enhanced Quality of Service (QoS) Controls​

Multi-tenancy in messaging systems presents unique architectural challenges, particularly around resource isolation and predictable performance. Apache Pulsar's approach to this has centered on providing granular control over system resources while maintaining consistency across tenant workloads.

The core advancement in Pulsar 4.0 comes through PIP-322 Pulsar Rate Limiting Refactoring. At its foundation is a token bucket algorithm implementation that unifies rate limiting across broker, topic, and resource group levels. This eliminates the previous split between "default" and "precise" rate limiters—a design choice that had introduced unnecessary CPU overhead and lock contention in IO threads.

The practical impact is straightforward: more predictable performance in multi-tenant deployments, especially when multiple rate limiting conditions intersect. While Pulsar already supports producer rate limiting, the community is building on this foundation with PIP-385 to improve producer flow control — a key piece in completing Pulsar's end-to-end QoS capabilities.

Rate Limiting and Capacity Management in Modern Messaging Platforms​

Rate limiting in Pulsar 4.0 sets the groundwork for a more comprehensive approach to capacity management. The goal is simple: enable dynamic resource allocation while maintaining strict performance guarantees, particularly in environments where workload patterns are unpredictable.

This matters because traditional auto-scaling approaches, which typically respond to CPU or memory metrics, often fall short in messaging systems. Andy Warfield's analysis of S3 storage workloads highlighted a pattern that's equally relevant in messaging: workloads are inherently bursty, characterized by sharp peaks amid periods of relative quiet. Traditional auto-scaling can't effectively handle these patterns without significant overprovisioning.

The industry has been converging on solutions to this challenge. Amazon DynamoDB's admission control system and TiDB's resource control framework demonstrate practical approaches to managing capacity in multi-tenant environments. Both systems use token bucket algorithms for base rate limiting but extend beyond this to dynamic quota management.

The solution lies in sophisticated capacity management systems that can smooth out these peaks across a platform's total capacity. This approach aligns perfectly with Marc Brooker's insights into the economics of load-balanced systems, where the efficiency gains of cloud-native architectures become most apparent at scale.

Pulsar's resource group concept, introduced with PIP 82: Tenant and namespace level rate limiting in Pulsar 2.8 in June 2021, laid initial groundwork for similar capabilities. While the documentation needs expansion, the technical foundation is solid. The token bucket implementation from PIP-322 provides the basis for evolution toward dynamic quota management and admission control, working in concert with Pulsar's load balancing to manage capacity effectively.

Looking ahead, this creates a path toward more sophisticated capacity management: dynamic limit adjustment based on real-time load, cross-tenant prioritization, and precise capacity allocation to specific tenants or namespaces. The focus is on handling demanding workloads while maintaining performance predictability—without the traditional answer of overprovisioning.

The development of these capabilities remains open and community-driven. Technical discussions and decisions happen in the open on the Apache Pulsar dev mailing list and get documented in Pulsar Improvement Proposals (PIPs).

Start using Pulsar 4.0​

Pulsar 4.0.0 is now available for download. The download page also includes details about the Docker image. To get started with Pulsar, you can run a Pulsar cluster on your local machine, Docker, or Kubernetes.

Upgrading existing clusters to Pulsar 4.0​

Starting from version 3.0, users have the option to perform live upgrades or downgrades between two consecutive LTS versions or two consecutive feature versions (which also include LTS versions).

Pulsar changes in releases are designed in a way that allows upgrading an existing Pulsar cluster to a newer release and then rolling back to the original release version in case of issues. This is considered when changes are made in Pulsar and the default BookKeeper configuration for Pulsar. Since Apache Pulsar is an open-source project, there is no guarantee that a specific configuration can be upgraded and downgraded.

In many cases, the ability to downgrade won't work if the upgrade happens from an older version than the latest released version in a particular release series. For example, upgrading to Pulsar 3.0.x works between 2.10.6 and 3.0.7, but not between 2.10.5 and 3.0.7.

Each Pulsar user is responsible for operating their cluster, and Pulsar cluster upgrades should be tested in testing and staging environments to ensure that a specific configuration can be upgraded and downgraded. When something is supported in the Apache Project, it means that the project is committed to addressing reported issues. This also applies to release upgrade compatibility.

According to this policy, users should first upgrade clusters to Pulsar 3.0.x or 3.3.x before upgrading to Pulsar 4.0.0. If you are upgrading from Pulsar 2.x, you should first upgrade to Pulsar 2.10.6 or 2.11.3, then upgrade to Pulsar 3.0.7, and finally upgrade to Pulsar 4.0.0.

Upgrading clients to use Pulsar 4.0​

Older Pulsar clients are compatible with Pulsar 4.0.0 and Pulsar Java client 4.0.0 is compatible with older Pulsar clusters. When upgrading clients, you can upgrade directly to the latest supported version. It is recommended to keep the clients up-to-date with latest security patches and bug fixes.

There's a critical security vulnerability CVE-2024-47561 in all Pulsar Java clients before 3.0.7, 3.3.2 and 4.0.0. Please check the Apache Pulsar Security page for more details. All Pulsar Java client users are recommended to upgrade to 3.0.7, 3.3.2 or 4.0.0.

When upgrading Java clients, it is a common problem that client module library versions are not aligned. It is recommended to use the Pulsar BOM to manage the Pulsar Java client version in Maven and Gradle builds. There's also specific instructions for selecting the Pulsar client version in Spring Boot projects.

Thank You to Apache Pulsar Contributors​

Apache Pulsar 4.0 represents the collaborative effort of a vibrant open-source community. This release showcases the dedication of developers, organizations, and users worldwide who have contributed to making data streaming more accessible and scalable.

We extend our deepest gratitude to:

• The individual contributors who developed new features, reported bugs, fixed bugs, and improved documentation
• The committers and PMC members who participated in Apache Pulsar project security issue handling, PIP decision making and voting for the releases.
• The organizations that have deployed Pulsar in production and shared their valuable feedback
• The users who participated in testing and provided invaluable input during the release process
• The broader Apache Software Foundation community for their continued support

The project faces typical open-source challenges — we're operating with a volunteer workforce and currently experiencing a backlog in pull request reviews. While this is a common scenario in successful Apache projects, we're actively working on processes to handle contributions more efficiently.

Apache Pulsar's strength lies in its community-driven development. Whether you're running Pulsar in production or evaluating it for your technology stack, we encourage you to join the conversation on the dev@pulsar.apache.org mailing list or the Pulsar Slack community. Your experience and feedback help shape the future of this platform.

What's new in Apache Pulsar 3.2?​

This release introduces 180+ improvements, optimizations, and fixes on the broker, Pulsar IO/Pulsar Functions, CLI, and others.

It also unveils key new features, highlighted below.

Rate limiting​

An increasing number of messaging-as-a-service platform teams are adopting Apache Pulsar as their main building block for providing messaging services across their organizations. This is clear validation that the value of Apache Pulsar’s truly multi-tenant architecture is delivering results, making Apache Pulsar a cost-efficient and reliable solution for messaging-as-a-service platform teams in very demanding application environments.

In the Apache Pulsar project, we are committed to delivering further improvements to the existing multi-tenancy features. One area of improvement is the service level management and capacity management of a large Pulsar system. This is also a key concern of messaging-as-a-service platform teams.

Apache Pulsar 3.2.0 features the implementation of PIP-322 Pulsar Rate Limiting Refactoring. Rate limiters act as a conduit to more extensive capacity management and Quality of Service (QoS) controls in Pulsar. They are integral to Pulsar's core multi-tenancy features. This refactoring paves the way for future enhancements in this area.

Null-key messages deletion during compaction​

Apache Pulsar supports Topic Compaction.

In versions 3.1.0 and earlier of Pulsar, topic compaction retains messages that have null keys.

Starting with Pulsar 3.2.0, the default behavior has been changed to delete messages with null keys during topic compaction, which helps in reducing storage costs. However, if necessary, you can revert to the previous behavior by setting topicCompactionRetainNullKey=true in broker.conf.

For more details, refer to the description in PIP-318.

WebSockets new features​

The Pulsar WebSocket API now supports:

• Consuming multiple topics as described in PIP-307.
• End-to-end encryption, as described in PIP-290.

Enhanced security for Pulsar Functions & Pulsar IO​

Pulsar 3.2.0 provides increased security for Pulsar Functions & Pulsar IO by allowing users to configure Pulsar Connectors with non-plaintext secrets. More details in PIP-289 & the following pull requests:

• Make connectors load sensitive fields from secrets
• Support configuration secret interpolation

CLI user experience improvements​

This release introduces several handy features to the CLI, including:

• Add command line option for configuring the memory limit
• Allow for deleting topics in bulk, using a regex or a file
• Display the current cluster in pulsar-admin clusters list

Build improvement​

Pulsar 3.2.0 also introduces enhancements to the build process and reliability, including a Bill of Materials (BOM) to simplify dependency management, as outlined in PIP-326.

More​

Feel free to explore the full release notes for a detailed list of all improvements and fixes.

Compatibility between releases​

When upgrading an existing Pulsar installation, it's crucial to perform component upgrades in a sequential manner.

Starting from version 3.0, users have the option to perform live upgrades or downgrades between two consecutive LTS versions or two consecutive feature versions (which also include LTS versions).

For the 3.2 series, you should be able to upgrade from version 3.x.

Please refer to the release policy to learn more about the compatibility between releases.

Getting started​

Pulsar 3.2.0 is now available for download. To get started with Pulsar, you can run a Pulsar cluster on your local machine, Docker, or Kubernetes.

Getting involved​

Apache Pulsar is one of the fastest-growing open-source projects, recognized by the Apache Software Foundation as a Top 5 Project based on engagement. The vitality of Pulsar relies on continued community growth, which would not be possible without each and every contributor to the project. The Pulsar community welcomes contributions from anyone with a passion for open source, messaging, and streaming, as well as distributed systems! Looking for more ways to stay connected with the Pulsar community? Check out the following resources:

• Pulsar Summit North America 2023 took place on Wednesday, October 25, 2023! Watch the sessions recordings. Follow @PulsarSummit on Twitter/X for updates and details on the upcoming Pulsar Summit events.
• Read the Apache Pulsar Contribution Guide to start your first contribution.
• Visit the Pulsar GitHub repository, follow @apache_pulsar on Twitter/X , and join the Pulsar community on Slack.

Since Pulsar’s graduation as a Top-Level Project (TLP) in September 2018, it has been driven by an active global community, with 12K+ commits from 639 contributors, 12.2K+ stars, 3.5K forks, and 10K+ Slack users.

We are grateful to all of our community members and those in the broader open-source community who contributed to the Apache Pulsar project. They are the reasons behind every step Apache Pulsar has made over the past years.

Now, let’s take a look at some of the highlights in 2023.

Apache Pulsar 3.0 LTS release: a Big Milestone for the Community​

The Apache Pulsar community announced the release of Apache Pulsar 3.0, the first Long-Term Support (LTS) version! Starting from Pulsar 3.0, the Pulsar community plans to release LTS versions to meet the needs of different users for stability and new features, as well as to reduce the burden of maintaining historical releases.

The previous release process has short maintenance cycles of approximately 3 to 4 months, while many users are still using old versions. To keep up with new updates and features, they may be forced to perform upgrades within a short timeframe, for which they are not prepared in terms of available time and required efforts.

Therefore, the Pulsar community introduces LTS versions with feature releases between them. The project follows a variant of Semantic Versioning, replacing major.minor.patch with LTS.feature.patch. For example:

• 2.11.0 is a feature release
• 3.0.0 is the first LTS release
• 3.0.1 is a patch release of the LTS release
• 3.1.0 is a feature release
• 3.2.0 is a feature release
• 3.2.1 is a patch release
• 4.0.0 is an LTS release

This pattern provides version support for users seeking stability and those seeking new features. Users who want a more stable release can use versions 3.0.x, while those seeking new features can use versions 3.x. This new release model is a major step for the Pulsar community because it:

• Allows users to choose between different releases based on their needs for stability or new features;
• Clarifies the release cycle for both maintainers and users;
• Frees maintainers from spending too much time maintaining a long list of old releases.

With the new release model, the Pulsar community looks to release LTS versions every 18 months, with bug fixes continuing for 24 months and security vulnerability patches supported for 36 months. See the image below for details.

For more information, see PIP-175 and the Release policy.

New Website​

The Apache Pulsar website has a fresh look! Thanks to Emidio Cardeira, Asaf Mesika, Tison Chen and Kiryl Valkovich for creating an engaging design that captures the future feel of our thriving community and next-gen solution.

Pulsar Admin Go Library​

Pulsar Admin Go Library provides a unified Go API for managing Pulsar resources such as tenants, namespaces, topics, etc.

Enhanced OTel-based metric system​

PIP-264 was completed, approved by the community, and started development. It will solve a big pain point for Pulsar users with a large number of topics - 50k up to 1M topics: observability. The Apache Pulsar community has taken a large undertaking to make OpenTelemetry Java SDK ready for very low latency systems such as Pulsar with two big features it currently develops: Near-zero memory allocations and metric filtering upon collection, which was also added to OpenTelemetry specifications.

Key events recap​

In 2023, the Apache Pulsar community put together a number of meetups and events across the globe to share the latest messaging and streaming technologies. Among others, three summits received the most attention from community members.

• Pulsar Summit Europe 2023: This event witnessed a remarkable milestone as over 400 attendees from 20+ countries joined the virtual stage to explore the cutting-edge advancements in Apache Pulsar and the real-world success stories of Pulsar-powered companies. This record-breaking turnout at the Pulsar Summit not only demonstrates the surging adoption of Pulsar but also highlights the ever-growing enthusiasm and curiosity surrounding this game-changing technology. It featured 5 keynotes on Apache Pulsar and 12 breakout sessions on tech deep dives, use cases, and ecosystem talks. They came from companies like Lego, VMWare, DataStax, StreamNative, RisingWave, Axon, Zafin and others. Watch the sessions.

• CommunityOverCode Asia 2023 conference (previously known as ApacheCon Asia) has been held from August 18th to August 20th. The conference gathers adopters, developers, engineers, and technologists from some of the most influential open-source communities in the world.

• CommunityOverCode NA 2023 conference (previously known as ApacheCon NA) has been held from October 7th to October 10th. The conference gathers adopters, developers, engineers, and technologists from some of the most influential open-source communities in the world. In 2023, CommunityOverCode introduced a Streaming track featuring three talks on Pulsar. If you missed the conference, you can still check out the slide decks!

• Pulsar Summit NA 2023: Hosted in person at the famous Hotel Nikko (with after parties overlooking the city!) in San Francisco, the summit featured nearly 200 attendees and showcased 20 carefully curated sessions, each a testament to the vibrancy and innovation within the Pulsar ecosystem. They came from companies like Cisco, Discord, Iterable, Attentive, VMware, Flipkart, Boomi and others. We are so grateful for the opportunity to spend a full day sharing knowledge and witnessing the community members connecting and inspiring each other. Watch the sessions.

Community growth​

The development of the Pulsar community would not be possible without our contributors. Among them, Pulsar Committers and PMC members have taken the lead in contributing to and promoting the project. In 2023, many new faces joined the community while we also welcomed old friends taking more responsibility. Let’s take a look at the Pulsar community by numbers.

• 639 global contributors for the Pulsar main GitHub repository
• 13.4K+ GitHub stars
• 3.5K+ forks
• 8 new Committers
• 6 new PMC members
• 10k+ Pulsar Slack members
• 20M+ Docker pulls

The Pulsar community welcomes all kinds of contributions. For more information, see the Apache Pulsar Contribution Guide.

Project releases​

In 2023, the Pulsar community worked hard to improve the project’s capabilities and fix existing bugs with 2 major versions and 12 minor versions.

The Apache Pulsar community released version 2.11 with 61 contributors providing feature enhancements and fixes that delivered 1617 commits.

A Big Milestone for the Apache Pulsar community was the release of Apache Pulsar 3.0, the first Long-Term Support (LTS) version! The community is getting bigger! Over 140 contributors submitted about 1500 commits to the Pulsar 3.0 release, which is the largest contribution yet for a project that is fast becoming one of the biggest open-source projects. It includes support for LTS, which delivers the predictability and stability that larger enterprise teams need to deliver a solid and reliable messaging and streaming service.

Thanks for all your contributions!

Many important Pulsar capabilities were delivered in these releases, such as Extensible Load Balancer and Large-scale delayed message support. For more information, see the Release Notes page.

Updates about clients, Pulsar Manager, and Pulsar Helm Chart are listed below:

• Pulsar C++ Client 3.4.2
• Pulsar Go Client 0.11.1
• Pulsar Node.js Client 1.9.0
• Pulsar Python Client 3.3.0
• Pulsar Manager 0.4.0
• Pulsar Helm Chart 3.1.0
• Pulsar dotnet Client 3.1.1
• Reactive Client for Apache Pulsar 0.1.0

For more information, see the Clients Release Notes page.

Pulsar ecosystem​

In 2023, the Pulsar community worked with other open-source communities to add more integrations to the Pulsar ecosystem. Notable integrations include:

• Quarkus Extension for Apache Pulsar provides support for Apache Pulsar through SmallRye Reactive Messaging framework. Based on Eclipse MicroProfile Reactive Messaging specification 3.0, it proposes a flexible programming model bridging CDI and event-driven.

• Spring for Apache Pulsar provides a PulsarTemplate for publishing to a Pulsar topic and a PulsarListener annotation for consuming from a Pulsar topic, as well as various convenience APIs for Spring developers to ramp up their development journey into Apache Pulsar. Support is also included in Spring Boot via auto-configuration.

• Oxia: currently, the practical limit for the number of topics manageable by a single Pulsar cluster is around 1 million topics. The lack of horizontal scalability in Zookeeper is one of the reasons for this limit. Oxia, released this year, is a scalable metadata store and coordination system. Replacing Zookeeper with Oxia in a Pulsar cluster allows for exceeding this 1M topics limit, although it is not the only prerequisite. This contributes to the goal of reaching 100M topics.

For more information, see the Ecosystem page.

What’s next in 2024

The Pulsar community seeks to further improve the project from different aspects.

Enhanced OTel-based metric system​

As stated above, PIP-264 is designed to enhance observability in scenarios where a single cluster manages a large number of topics, ranging from 50k up to 1M topics. Currently, there are two major features under development to support this functionality:

• Near-zero memory allocations
• Metric filtering upon collection, which was also added to OpenTelemetry specifications.

Pulsar Rate Limiting​

An increasing number of messaging-as-a-service platform teams are adopting Apache Pulsar as their main building block for providing messaging services across their organizations. This is clear validation that the value of Apache Pulsar’s truly multi-tenant architecture is delivering results, making Apache Pulsar a cost-efficient and reliable solution for messaging-as-a-service platform teams in very demanding application environments.

In the Apache Pulsar project, we are committed to delivering further improvements to the existing multi-tenancy features. One area of improvement is the service level management and capacity management of a large Pulsar system. This is also a key concern of messaging-as-a-service platform teams.

In December 2023, PIP-322 Pulsar Rate Limiting Refactoring was accepted and completed and will be release as part of Pulsar 3.2.0 release. Rate limiters act as a conduit to more extensive capacity management and Quality of Service (QoS) controls in Pulsar. They are integral to Pulsar's core multi-tenancy features. This refactoring will pave the way for future enhancements in this area.

Pulsar SQL removal from the main repo​

Pulsar SQL (Trino/Presto) will be moved from the main repository to a separate repository. This change will offer the following benefits:

• A significant reduction in the size of the TGZ and Docker image, saving approximately 400MB.
• Reduced build time.

Security​

Docker image vulnerability scans will start soon.

Events​

We will also have more events coming for 2024, including Pulsar Summit North America and Pulsar Summit APAC. If you missed or want to relive the Pulsar Summit North America 2023, you can check out the videos of our amazing speakers here!

Stay in touch!​

To stay up to date with community news and discuss hot topics with other members, you can subscribe to the Pulsar mailing lists for users and developers, follow us on X, and join the Pulsar Slack workspace and Pulsar community meetings that happen online.

What's new in Apache Pulsar 3.1?​

Pluggable topic compaction service​

Pulsar's Topic Compaction feature provides a key-based data retention mechanism that allows users to keep only the most recent message associated with a specific key. This helps reduce storage space and improve system efficiency.

Data in topics can be stored in various formats. For example, KoP (Kafka protocol handler) can store data in Kafka format.

Previously, Pulsar always compacted topic data, assuming that messages were in the Pulsar data format. However, this approach had limitations, as it prevented protocol handlers from utilizing the topic compaction feature with customized data formats such as the Kafka format used by KoP.

That's why PIP-278 introduced a pluggable topic compaction service interface to support customization of the actual compaction logic. This customization can be done while the major compaction task is still controlled by the Pulsar broker. This change primarily benefits protocol handlers developers.

Pluggable partition assignment strategy​

Pulsar offers robust support for load balancing to ensure efficient resource utilization across Pulsar clusters.

The fundamental unit for load balancing is the topic bundle, which refers to a group of topics within the same namespace.

Previously, the only strategy for assigning a topic to a topic bundle was consistent hashing. However, this strategy doesn't fit all scenarios.

PIP-255 introduced a pluggable topic bundle (partition) assignment interface to allow customization of the assignment algorithm. This enables users to adjust the strategy according to their specific scenarios.

Metadata size threshold for compression​

Previously, even if the metadata was small, we had to apply compression. Now, we support a size-based threshold.

Starting from version 2.9, Pulsar supports compressing managed ledger information and managed cursor information stored in the metadata store. This feature can significantly reduce the size of large metadata.

However, for small metadata, compression doesn't provide significant benefits and may consume unnecessary computational resources.

PIP-270 introduces two configuration options: managedLedgerInfoCompressionThresholdInBytes and managedCursorInfoCompressionThresholdInBytes. These options allow users to customize the size threshold for compressing metadata, with the default value set to 16 KB.

Lazy creation of offload resources​

Tiered storage is an essential technology that enables the migration of old topic data from BookKeeper to long-term and more cost-effective storage while maintaining transparent client access to the topic data.

Tiered storage operates through offloaders. Previously, when a topic was created, the offloader immediately generated the associated offload resources, even though these resources remained unused until the actual offloading task was triggered.

PR-20775 modifies this behavior by lazily creating the offload blob store. This means that the actual allocation occurs only when the offloading task is triggered, preventing excessive preallocation of resources.

Compatibility between releases​

When upgrading an existing Pulsar installation, it's crucial to perform component upgrades in a sequential manner.

Starting from version 3.0, users have the option to perform live upgrades or downgrades between two consecutive LTS versions or two consecutive feature versions (which also include LTS versions).

For the 3.1 series, you should be able to upgrade directly from version 3.0 or downgrade from the subsequently released version 3.2. If you are currently using an earlier version, please ensure that you upgrade to version 3.0 before proceeding further.

Getting started​

Pulsar 3.1.0 is now available for download. To get started with Pulsar, you can run a Pulsar cluster on your local machine, Docker, or Kubernetes.

Getting involved​

Apache Pulsar is one of the fastest-growing open-source projects, recognized by the Apache Software Foundation as a Top 5 Project based on engagement. The vitality of Pulsar relies on continued community growth, which would not be possible without each and every contributor to the project. The Pulsar community welcomes contributions from anyone with a passion for open source, messaging, and streaming, as well as distributed systems! Looking for more ways to stay connected with the Pulsar community? Check out the following resources:

• Pulsar Summit North America 2023 will take place on Wednesday, October 25, 2023! Register now and follow @PulsarSummit on Twitter/X for updates and details of this much-anticipated one-day event.
• Read the Apache Pulsar Contribution Guide to start your first contribution.
• Visit the Pulsar GitHub repository, follow @apache_pulsar on Twitter/X , and join the Pulsar community on Slack.

The conference gathers adopters, developers, engineers, and technologists from some of the most influential open-source communities in the world. To date, there has been a total of over 100 proposals submitted by presenters from AWS, Huawei, Tencent Cloud, StreamNative, WeBank, and many more.

We are excited to have various Apache Pulsar-related sessions on this conference, including the milestone achieved by our first LTS release, integrations with applications on cloud, the Pulsar community development, and so on.

Featured sessions​

Let's have a quick look at some of the featured sessions about Apache Pulsar.

Apache Pulsar 3.0: First LTS release and new features​

Zike Yang, Software Engineer, StreamNative

The Apache Pulsar community recently launched Apache Pulsar 3.0, the first LTS release of Pulsar. In this talk, we will delve into the importance of Pulsar's LTS version. We'll also cover the major features introduced in Pulsar 3.0, including a new load balancer, support for large-scale delayed messages, and Direct IO optimization.

Huawei Terminal Cloud optimizes Apache Pulsar in container scenarios​

Lin Lin, SDE Expert, Huawei

Apache Pulsar is a cloud-native message queue that, based on its storage separation architecture, can often shrink the compute layer to save resources during periods of low traffic. We made a lot of optimizations for Apache Pulsar in containerization scenarios. For example, at present, the Pulsar load balancing algorithm relies on the past load data of nodes, and the process of achieving balance is relatively slow. When HPA is enabled, node capacity expansion may be triggered during load balancing, and capacity expansion triggers new load balancing. How can we optimize to make Pulsar more cloud-native?

Apache Pulsar current-limiting principle and application practice​

Jialing Wang, Software development engineer, China Mobile Cloud Capability Center

Combined with the current limiting practice of mobile cloud Pulsar and Kafka, this speech analyzes the working principle of current limiting at all levels of production and consumption in Apache Pulsar and the implementation scheme of ResourceGroup, and introduces the mobile cloud Kafka based on Pulsar and KoP. How to manage traffic between multiple clusters in cloud native scenarios based on ResourceGroup and load balancer?

The Secret to Great Developer Experience is Killer Content​

Yu Liu, Apache Pulsar PMC member

In recent years, the software industry has entered the cloud-native 2.0 era. The emergence of various new technologies has brought a sweet burden to developers. On the one hand, they have more choices; on the other hand, they face greater complexities. With the rise of the "developer-first" culture and the increasing decision-making weight of developers on product selection, the world's major technology giants are changing from traditional sales ideas to developer-centric ideas (B2D) to promote products, and high-quality content is the most effective marketing strategy. To satisfy all stakeholders' needs and accelerate the flywheel effect, how to design content that makes developers fall in love at first sight? To improve the development experience, how to create the "Aha moment" of the content? To increase user stickiness, how to make developers love the product forever through content? To strengthen brand competitiveness, how to differentiate the content strategies for open-source projects and commercial products? ——This session will share the practice in the Apache Pulsar community and dive deeper into how to design killer content that developers will love.

More resources​

As we can see from topics submitted to CommunityOverCode Asia 2023, Apache Pulsar has become one of the most active Apache projects over the past few years, with a vibrant community that continues to drive innovation and improvements to the project.

1. Save your spot at the Pulsar Summit North America 2023. It will be taking place in San Francisco，October 25, 2023. Sign up today to join the Pulsar community and the messaging and event streaming community.
2. Join the Apache Pulsar community. Subscribe to the Pulsar mailing lists for user-related or Pulsar development discussions. You can also join the Pulsar Slack to ask quick questions or discuss specialized topics.

We're excited to announce that Pulsar Summit North America 2023 will take place on Wednesday, October 25, 2023!

We welcome your participation to help make the event a success, by submitting a talk for the event or offering sponsorship.

What is Pulsar Summit?​

Pulsar Summit is the conference dedicated to Apache Pulsar and the messaging and event streaming community. The conference gathers an international audience of developers, data architects, data scientists, Apache Pulsar committers and contributors, and friends within the streaming and messaging ecosystem. Together, they share experiences, exchange ideas and knowledge, and receive hands-on training led by Pulsar experts.

Since 2020, seven global Pulsar Summit Events have featured 170+ interactive sessions by tech leads, open-source developers, software engineers, and software architects from Google, AWS, Splunk, Tencent, Verizon Media, Iterable, Yahoo, Nutanix, BIGO, TIBCO, OVHcloud, Clever Cloud, and more. The conferences have garnered 2,200 attendees representing 700 companies, including individuals from leading organizations such as Google, Microsoft, AMEX, Salesforce, TikTok, Alibaba, Tencent, Disney, and Paypal.

Speak at Pulsar Summit North America 2023!​

Submit your session abstract

Share your Pulsar story and speak at the summit! Pulsar Summit offers a unique opportunity to connect with your peers and raise your profile in the rapidly growing Apache Pulsar community.

Our theme this year is "Why Pulsar?" and we are looking for stories that are innovative, informative, or thought-provoking. Join us to speak at the summit! You will be on stage with all the top Pulsar thought-leaders. It is a great way to participate and raise your profile in the rapidly growing Apache Pulsar community.

As a speaker, you will receive:​

• Free conference pass.
• Your headshot, bio, and session featured on the Pulsar Summit website.
• Your session will be promoted on YouTube, Twitter, and LinkedIn.
• The opportunity to share your knowledge and engage with the vibrant Pulsar community!

To speak at the summit, please submit an abstract about your presentation. Remember to keep your proposal short, relevant, and engaging. All levels of talks (beginner, intermediate, and advanced) are welcome. We invite you to submit your talk proposals and share your unique experience in one of the following themes:

• “Why Pulsar?” - Your journey is inspiring, and we want to hear it! Share your stories of why you chose Pulsar, the challenges you overcame, and the solutions you implemented. Let others learn from your experiences and gain valuable insights from your use cases.
• “Learning Pulsar” - Are you an educator at heart? Submit an entry-level talk introducing Pulsar, its remarkable features, and best practices. Help beginners navigate their Pulsar journey and inspire them with your expert knowledge.
• “Deep Dive” - For all the experts out there, we invite you to share deep technical knowledge about Pulsar. Whether it's about the inner workings, optimization techniques, or advanced features, we'd love to hear it all.
• “Around Pulsar” - Have you built exceptional tools to work with Pulsar? Or have you integrated Pulsar with other technologies to create a superior solution? We'd love to hear about your innovations.

Important Dates:​

• CFP opens: Wednesday, June 21st, 2023
• CFP closes: Friday, July 7th, 2023
• Speaker notifications sent: Friday, July 28th, 2023
• Schedule announcement: Friday, August 4th, 2023

Submissions are open until Friday, July 7th, 2023. If you want some advice or feedback on your proposal or have any questions about the summit, please do not hesitate to contact us at organizers@pulsar-summit.org. We are happy to help!

Help us make #PulsarSummit North America 2023 successful by spreading the word and submitting your proposal and sponsorship! Follow us on Twitter (@pulsarsummit) to receive the latest updates on the summit.

About the Organizer​

StreamNative is built by the original creators of Apache Pulsar and Apache BookKeeper, and is one of the leading contributors to the open source Apache Pulsar project. As the core developers of Pulsar, the StreamNative team is deeply versed in the technology, the community, and the use cases. Today, StreamNative is focusing on growing the Apache Pulsar and BookKeeper communities and bringing its deep experience across diverse Pulsar use cases to companies across the globe.

Introducing Long-Term Support releases​

Starting from Pulsar 3.0, the Pulsar community plans to release LTS versions to meet the needs of different users for stability and new features, as well as to reduce the burden of maintaining historical releases.

The previous release process has short maintenance cycles of approximately 3 to 4 months, while many users are still using old versions. To keep up with new updates and features, they may be forced to perform upgrades within a short timeframe, for which they are not prepared in terms of available time and required efforts.

Therefore, the Pulsar community introduces LTS versions with feature releases between them. The project follows a variant of Semantic Versioning, replacing major.minor.patch with LTS.feature.patch. For example:

• 2.11.0 is a feature release;
• 3.0.0 is the first LTS release;
• 3.0.1 is a patch release of the LTS release;
• 3.1.0 is a feature release;
• 3.2.0 is a feature release;
• 3.2.1 is a patch release;
• 4.0.0 is an LTS release.

This pattern provides version support for users seeking stability and those seeking new features. Users who want a more stable release can use versions 3.0.x, while those seeking new features can use versions 3.x. This new release model is a major step for the Pulsar community because it:

• Allows users to choose between different releases based on their needs for stability or new features;
• Clarifies the release cycle for both maintainers and users;
• Frees maintainers from spending too much time maintaining a long list of old releases.

With the new release model, the Pulsar community looks to release LTS versions every 18 months, with bug fixes continuing for 24 months and security vulnerability patches supported for 36 months. See the image below for details.

For more information, see PIP-175 and the Release policy.

What’s new in Apache Pulsar 3.0?​

Apache Pulsar 3.0 is an LTS release with a number of important features, enhancements, and fixes. Here are some of the highlighted ones.

New Pulsar broker load balancer​

PIP-192 introduces a new load manager implementation, aiming to balance cluster utilization as evenly as possible while minimizing latency and reducing reliance on ZooKeeper.

Issues with the previous load manager include:

• Load data for all brokers and bundles is replicated to all brokers via ZK watchers. This N-replication poses scalability issues when Pulsar clusters grow to thousands of brokers and millions of topics.
• Lookup requires redirection to the leader broker.

In Pulsar 3.0, the broker and bundle load data have been re-evaluated. The new load manager stores this load data in non-persistent topics, while topic ownership information is saved in system topics and maintained with a state machine for eventual consistency. Clients can now connect to any broker for lookup without redirection.

Large-scale delayed message support​

Scheduled and delayed message delivery is a common feature in messaging systems. PIP-195 aims to address limitations in this feature, which has been supported since version 2.4.0. The issues with the current implementation include:

• Memory constraints: The delayed message index is typically maintained in memory, leading to high memory overhead when there are many delayed messages. Although it is possible to utilize multiple brokers’ memory by creating multiple partitions for a topic and distributing them across multiple brokers, the overall memory consumption remains unchanged.
• Expensive index rebuilding: In the case of a large number of delayed messages (e.g. hundreds of millions), when migrating a topic to a different broker or recovering from broker downtime, rebuilding the index requires replaying logs. This process consumes significant resources and affects both the client and catch-up reads.

The objectives of the new delayed message mechanism introduced in Pulsar 3.0 are:

• Supporting delayed message index snapshots to minimize the costs of rebuilding the index;
• Reducing memory usage for maintaining the delayed message index.

Build multi-arch docker images​

With 3.0, Pulsar will start publishing Docker images with versions both for Intel x86-64 and Arm64 architectures. 

Users trying to use Pulsar standalone, or running TestContainer tests on a Mac M1/M2 laptop will now see much improved performance and avoid all the issues with the Docker container engine when it emulates x86-64 CPU withing an Arm64 host.

At the same time, this image will make it possible to run Pulsar in a Docker/Kubernetes production environment on Arm64 machines.

From a user perspective, no change is required: when pulling the image, Docker will select the appropriate version based on the host architecture.

For more information, see PR-19432.

BookKeeper direct IO logic optimization​

The current ledger read/write logic in BookKeeper involves multiple buffering and caching layers, which cause limitations in read and write throughput due to memory consumption, inefficient cache utilization, and eviction problems.

More specifically, when writing an entry, data is buffered in both the memory table and the OS PageCache (after the memory table is full). When reading an entry, data may be fetched from the memory table, read cache, or read from the entry log file based on cache hits, potentially causing memory waste and reduced cache hit rate. Since the OS PageCache is shared by all read and write files, and the OS will prefetch a fixed amount of data when reading files, it is easy to cause OS PageCache pollution and affect the read and write performance.

BP-47 introduces optional support to bypass the OS PageCache using the O_DIRECT flag for open(2) and fallocate(2) syscalls on supported systems (Linux and macOS). The new implementation uses JNI for direct I/O and incorporates write buffer pools and read buffer management. It helps reduce memory consumption and achieve better control over cache utilization. It modifies the existing ledger read/write logic while maintaining the original organization of the entry log file.

Transaction Buffer segmented snapshot optimization​

The current Transaction Buffer in Pulsar involves handling messages sent with transactions and taking periodic snapshots to avoid replaying all messages from the original topic. However, when a topic has long-term data retention and many aborted transactions, a snapshot may become a bottleneck, causing increased costs as the snapshot size grows.

PIP-196 introduces a segmented transaction buffer snapshot, which allows splitting the snapshot into multiple parts, each with a fixed number of aborted transactions and a maxReadPosition identity. This approach aims to support a large number of aborted transactions, improve transaction buffer recovery speed, and address the write amplification issue in system topic snapshots.

In Pulsar 3.0, the new design incorporates multiple snapshot segments through a secondary index, with index and snapshot segments stored in different compact topics. The snapshot segment is an immutable segment that can be configured in size. A new system topic is used to store snapshot segments, while a separate index topic stores the snapshot segment indexes, allowing for more efficient recovery and reduced memory overhead.

The proposal modifies the existing transaction buffer logic while maintaining the original organization of the transaction buffer, with the goal of improving overall performance and resource utilization.

Blue-green cluster deployment support​

Blue-green deployment is a widely-used solution for migrating live traffic from one cluster to another. In this model, traffic is gradually transferred from the blue cluster to the green cluster, allowing for smooth upgrades and the possibility of rollbacks if necessary.

PIP-188 implements blue-green deployment in Pulsar by introducing changes to the broker, client, and managedLedger. The broker will support an admin API that allows marking a cluster for migration and specifying redirection URLs. The migration state and redirection URLs are persisted in the cluster metadata. The broker asynchronously marks each topic owned by that broker as migrated and notifies all the producers and consumers (which have drained the backlog for their subscriptions) with a new client-protocol command called “Migrated-Topic”, which has redirection URLs to the green cluster. Producers and consumers for those topics cache the redirection URLs and retry to connect to the broker with that URL which redirects them to the green cluster.

With such capability introduced in Pulsar 3.0, users can achieve seamless traffic migration between clusters without causing downtime for topics.

For more information about the Pulsar 3.0 release, see the release notes.

Compatibility between releases​

When upgrading an existing Pulsar version, it is important to upgrade components linearly.

Before Pulsar 3.0, upgrades should be performed linearly through each feature version. For example, when upgrading from 2.8 to 2.10, it is important to upgrade to 2.9 before going to 2.10.

Starting from 3.0, users can perform live upgrades or downgrades between two consecutive LTS versions. For example:

• ✅ 3.0 -> 4.0 -> 3.0;
• ✅ 3.2 -> 4.0 -> 3.2;
• ✅ 3.2 -> 4.4 -> 3.2;
• ❌ 3.2 -> 5.0.

Getting started​

Pulsar 3.0.0 is now available for download on GitHub. To get started with Pulsar, you can run a Pulsar cluster on your local machine, Docker, or Kubernetes.

Looking forward​

With Pulsar 3.0 LTS model, it will be easier for users to choose to stay on LTS stable releases or newer releases with the latest feature and improvements, all with the guarantee of longer support and a path to a live upgrade to the next LTS version.

At the same time, several features introduced in 3.0 sets the stage for further improvements. One such example is the new load balancer (PIP-192): while immediately useful and a marked improvement as it is, it also paves the way for many more optimizations in how topics are transferred from one broker to another, with the goal of minimizing the latency impact of such operations.

Another exciting area of development is the proposal to rehaul the metric collection and aggregation system described in PIP-264.

Finally, Pulsar 3.0 is benefitting from the performance improvements introduced in Apache BookKeeper 4.16. We see a lot of potential to further improve BookKeeper performance for an even faster Pulsar!

Getting involved​

Apache Pulsar is one of the fastest-growing open-source projects, recognized by the Apache Software Foundation as a Top 5 Project based on engagement. The vitality of Pulsar relies on continued community growth, which would not be possible without each and every contributor to the project. The Pulsar community welcomes contributions from anyone with a passion for open source, messaging and streaming, as well as distributed systems! Looking for more ways to stay connected with the Pulsar community? Check out the following resources:

• Pulsar Virtual Summit Europe 2023 will take place on Tuesday, May 23rd, 2023! Register now for free and follow @PulsarSummit on Twitter for updates and details of this much-anticipated one-day event.
• Read the Apache Pulsar Contribution Guide to start your first contribution.
• Visit the Pulsar GitHub repository, follow the project on Twitter @apache_pulsar, and join the Pulsar community on Slack.

This blog walks through the most noteworthy changes. For the complete list, including all feature enhancements and bug fixes, check out the Pulsar 2.10.4 Release Notes.

Improve performance in Key_Shared subscriptions (PR-19167)​

Issue​

In earlier versions, Key_Shared subscriptions suffered from performance issues due to the lack of a ref count for sticky hash.

Resolution​

A significant performance improvement has been made to the Key_Shared subscriptions by adding a ref count for sticky hash. This optimization reduces the time required to check if messages contain the hash from a given sticky hash set, resulting in improved performance and efficiency.

AbstractBatchedMetadataStore - Use AlreadyClosedException instead of IllegalStateException (PR-19284)​

Issue​

In earlier versions, when the broker was shutting down, the AbstractBatchedMetadataStore would complete pending operations with a generic IllegalStateException. However, code dependent on the MetadataStore usually expected instances of MetadataStoreException and might not react properly to this error.

Resolution​

This PR improves the error handling during broker shutdown by completing pending operations with an AlreadyClosedException instead of IllegalStateException. This ensures that the dependent code responds more appropriately to the error, even if the broker is shutting down.

Fix open cursor with null-initialPosition, resulting in the earliest position (PR-18416)​

Issue​

In earlier versions, called ledger.openCursor("xxx", null) with a null initialPosition parameter would result in the cursor being set to the earliest position, causing unexpected behavior. The root cause of this issue was due to incorrect handling of the initialPosition in ManagedLedgerImpl.java.

Resolution​

This PR fixes the issue by ensuring that the correct initial position is used when opening a cursor with a null initialPosition. The code now sets the cursor to the latest position, as expected.

Add isActive in ManagedCursorImpl (PR-19341)​

Issue​

In previous versions, when there were many concurrent subscriptions in a topic, broker performance would degrade due to many io-threads waiting for the lock, synchronized (activeCursors) while calling checkBackloggedCursors.

Resolution​

This PR adds an isActive variable in ManagedCursorImpl to minimize the access to activeCursors in ManagedLedgerImpl, which reduces the contention on the lock and improves the broker performance with many concurrent subscriptions.

Support deleting partitioned topics with the keyword -partition- (PR-19230)​

Issue​

In earlier versions, although users could create partitioned topics using the client when partitioned type auto-creation was enabled, there was no support for deleting these partitioned topics.

Resolution​

This PR adds support for deleting partitioned topics using the keyword -partition-, making it easier for users to manage their partitioned topics.

What’s Next?

If you are interested in learning more about Pulsar 2.10.4, you can download and try it out now!

For more information about the Apache Pulsar project and current progress, visit the Pulsar website, follow the project on Twitter @apache_pulsar, and join Pulsar Slack!

We’re excited to invite you to Pulsar Virtual Summit Europe 2023! Join the Apache Pulsar community online on Tuesday, May 23rd for this exciting one-day event. Don’t miss 5 keynotes and 14 breakout sessions, and the opportunity to network with fellow attendees in this free online event. Not in the timezone? No problem! Register today to ensure you get the recorded sessions a week before general public release.

Global Pulsar Summits gather developers, architects, and data engineers to discuss the latest in real-time data streaming and message queuing. Past Pulsar Summits have featured more than 200 interactive sessions presented by tech leaders from WeChat, Blizzard, Intuit, Micro Focus, Salesforce, Splunk, Verizon Media, Tencent, Uber, and more. The Summits garnered 2,200+ global attendees representing top technology, fintech, and media companies, such as Google, Amazon, eBay, Microsoft, American Express, LEGO, Athena Health, Paypal, just to name a few.

Don't miss this one-day event! Register now for free!

Featured sessions​

This year, Pulsar Virtual Summit Europe will include tech deep dives, use cases, best practices, and insights into Pulsar’s global adoption and thriving community. Take a sneak peek below at a few of the featured sessions.

Challenges of Hosting a Pulsar-as-a-Service Platform Under a Shared Responsibility Model​

Edgaras Petovradzius, Senior Engineer, the LEGO Group

Mathias Ravn Tversted, Engineer, the LEGO Group

This talk will explore the challenges the LEGO Group encountered in hosting and managing Pulsar-as-a-Service across multiple domains, and how they collaborated with StreamNative in the process. It will highlight how the organization used OAuth2 for authentication and its self-service management platform for access control of Pulsar resources. The speakers will also discuss their observability tooling, incorporating cloud-native tools like Prometheus and OTel collectors, as well as internal tools for metrics and an ELK stack. Finally, the presentation will share best practices for using Pulsar clients in various programming languages and explore alternative methods of producing and consuming messages.

Pulsar Observability in High-Topic Cardinality Deployments for Telco​

Habip Kenan Üsküdar, DevGitOps Engineer, Axon Networks

Don't miss speaker Habip Kenan Üsküda from Axon Networks, as he shares the experience of building an observability stack using Grafana and Prometheus for their cloud-native platform based on Apache Pulsar. As the number of topic pairings scaled beyond 50,000, with the goal of exceeding 1 million, the team encountered major challenges in scaling their monitoring stack. The presentation will cover their innovative approach to partitioning labels and tenancy management, as well as their efforts to extend Grafana Agent Operators’ Helm Charts to overcome these bottlenecks.

Building a Full Lifecycle Streaming Data Pipeline​

Timothy Spann, Principal Developer Advocate for Data in Motion, Cloudera

David Kjerrumgaard, Systems Engineer, Developer Advocate and Author of "Pulsar in Action", StreamNative

Julien Jakubowski, Developer Advocate, StreamNative

Join Tim, David, and Julien as they delve into the process of building a full lifecycle streaming data pipeline using Apache Pulsar, Spring, Java, Apache Pinot, Trino, and Apache Iceberg. This session will provide an overview of each tool’s key features and capabilities, demonstrating their integration for a robust and efficient real-time streaming data pipeline. The talk will also cover best practices for using these tools together, as well as case studies and real-world examples of successful pipeline implementations.

The Future of Metrics in Pulsar​

‍Asaf Mesika, Principal Engineer, StreamNative

In this session, Asaf Mesika will discuss the challenges of using observability metrics in Pulsar from both user and committer perspectives. He will highlight issues such as high topic count limitations, improper histogram use in Grafana, and implementation difficulties. The talk will also present a proposal to address these problems by adopting the OpenTelemetry Java SDK, offering insights for leveraging metrics in Pulsar.

Pulsar in Finance - A Tale of a Migration​

George Orban, Tech Lead, Senior Architect, Quant Developer, Daiwa Capital Markets

In this talk, George Orban will share the experience of migrating a pricing engine and trading system from TIBCO Rendezvous and other messaging solutions to Apache Pulsar. He will discuss the reasons for choosing Pulsar, its suitability for enterprise applications and finance, and how it improved their stack’s resilience, robustness, and speed. The presentation will also cover new functionalities gained through Pulsar’s richer semantics, lessons learned, tools developed and open-sourced during the migration, and the future of Pulsar at Daiwa.

Oxia: Scaling Pulsar’s Metadata to 100x​

Matteo Merli, Apache Pulsar PMC Chair, CTO, StreamNative

Join Apache Pulsar PMC Chair and StreamNative CTO, Matteo Merli, for the much-anticipated introduction of Oxia, a metadata store and coordination system designed to overcome the limitations of ZooKeeper in scaling Pulsar clusters. Learn first-hand the design goals, architecture, and development journey of Oxia. Matteo will also explain how Oxia’s design leverages modern cloud-native environments to provide a highly flexible and dynamic operational environment. Watch this keynote to learn more about scaling Pulsar’s metadata with Oxia and how it can improve the performance and scalability of Pulsar clusters.

Check out the full agenda and details at pulsar-summit.org.

How to participate​

Register now for free! Contact us at organizers@pulsar-summit.org if you have any questions.

About the organizer​

StreamNative is the organizer of Pulsar Virtual Summit Europe 2023. Founded by the original developers of Apache Pulsar and Apache BookKeeper, StreamNative builds a cloud-native event streaming platform that enables enterprises to easily access data as real-time event streams. As the core developers of Pulsar, the StreamNative team is deeply versed in the technology, the community, and the use cases. Today, StreamNative is focusing on growing the Apache Pulsar and BookKeeper communities and bringing its deep experience across diverse Pulsar use cases to companies across the globe.

The Apache Pulsar community embraced a significant milestone last month as the project witnessed its 600th contributor to the Pulsar main GitHub repository. We would like to thank everyone in the Pulsar community who contributed to this remarkable achievement.

Since Pulsar’s graduation as a Top-Level Project (TLP) in September 2018, it has been driven by an active global community, with 160+ releases, 11K+ commits from 600 contributors, 12.2K+ stars, 3.2K+ forks, and 9600+ Slack users.

Strong community growth​

600 contributors​

The number of contributors is an important metric to measure the health of an open-source project. In the last year alone, we added almost 130 contributors to the project, a 28% increase from the previous year. The image below shows the number of Pulsar contributors over the past 6 years.

Monthly active contributors of Pulsar and Kafka​

Both Pulsar and Kafka are popular streaming systems with contributors across the globe and are adopted by organizations spanning different industries. Although Kafka outnumbers Pulsar in the total number of contributors, the latter surpassed the former in terms of monthly active contributors about 2 years ago and has maintained a strong momentum since then.

12K+ GitHub stars​

GitHub stars are another key metric for open-source projects. Figure 3 displays the star history of Pulsar since its inception.

Pulsar adoption​

As the project achieves strong growth in contributors, it also sees widespread adoption by companies across industries. Their success stories speak volumes about a more stable and secure project capable of powering different use cases in the messaging and streaming space.

Pulsar has played an essential role in handling mission-critical workloads for both existing and new users. Tencent, one of the earliest companies to adopt Pulsar in production, has been consistently working to improve the project for better stability and shared their experience of handling 100 billion messages per day. New adopters include Nippon Telegraph and Telephone Corporation (NTT) Software Innovation Center, which uses a single Pulsar topic to handle 100K consumers for its IoT use case. Some organizations migrated from systems like Kafka to Pulsar, such as Sina Weibo and mParticle. They select Pulsar not just for its flexibility, scalability, high availability, and unique architecture. More importantly, Pulsar solves the problems and pain points where other systems fall short.

For more information, see this list of companies using or contributing to Pulsar and check out how different organizations are using Pulsar.

Get involved​

Backed by a diverse community of contributors, we believe that each and every pull request counts and would like to see more contributors join the journey. To start making your contribution to the project, see the Apache Pulsar Contribution Guide.

To stay up to date with community news and discuss hot topics with other members, you can subscribe to the Pulsar mailing lists for users and developers, follow us on Twitter, and join the Pulsar Slack workspace.

The Pulsar community hosts events, meetups, and webinars for Pulsar users of all experience levels. You can check out the Events page and join different user groups to stay tuned.

What is Pulsar Summit?​

Pulsar Summit is the conference dedicated to Apache Pulsar and the messaging and event streaming community. The conference gathers an international audience of developers, data architects, data scientists, Apache Pulsar committers and contributors, and friends within the streaming and messaging ecosystem. Together, they share experiences, exchange ideas and knowledge, and receive hands-on training led by Pulsar experts.

Since 2020, seven global Pulsar Summit Events have featured 170+ interactive sessions by tech leads, open-source developers, software engineers, and software architects from Google, AWS, Splunk, Tencent, Verizon Media, Iterable, Yahoo, Nutanix, BIGO, TIBCO, OVHcloud, Clever Cloud, and more. The conferences have garnered 2,000 attendees representing 700 companies, including individuals from leading organizations such as Google, Microsoft, AMEX, Salesforce, TikTok, Alibaba, Tencent, Disney, and Paypal.

Join us and speak at Pulsar Virtual Summit Europe 2023​

Share your Pulsar story and speak at the summit! Pulsar Summit offers a unique opportunity to connect with your peers and raise your profile in the rapidly growing Apache Pulsar community. 

We’re looking for Pulsar stories that are innovative, informative, or thought-provoking. Here are some suggestions based on previous events:

• A Pulsar success story or case study
• Use cases, operations, tools, techniques, or the Pulsar ecosystem
• A deep dive into technologies
• Best practices and lessons learned
• Anything else related to Pulsar that inspires the audience

Interested in speaking at the summit? Submit your session abstract here. 

All levels of talks (beginner, intermediate, and advanced) are welcome. Remember to keep your proposal short, relevant, and engaging. The following session formats are acceptable:

• Session Presentation: 30-minute presentation, maximum of 2 speakers, can include demo
• Lighting Talk: 20-minute presentation, maximum of 2 speakers

All accepted submissions will be pre-recorded. Due to time zone and network limitations, we do not recommend speakers present their talk live.

Event dates to remember​

• CFP opens: February 1st, 2023
• CFP closes: March 3rd, 2023
• CFP notifications: March 24th, 2023
• Schedule announcement: March 31st, 2023
• Event date: May 23rd, 2023

Community sponsorships available​

We invite you to participate as a Community Sponsor for Pulsar Virtual Summit Europe 2023. Sponsoring this event provides an excellent opportunity for your organization to further engage and connect with the quickly growing Pulsar and streaming communities.

Community Sponsorships for Pulsar Virtual Summit Europe 2023 include your company logo on the pulsar-summit.org website and on-screen during the welcome introduction presentation, as well as opportunities for you to help promote the event. 

To secure your no-fee Community Sponsorship, please submit your response here. The full Sponsorship Prospectus will be made available with your inquiry. For more information on becoming a sponsor, please contact the Pulsar Summit event organizers at organizers@pulsar-summit.org.

Help us make #PulsarSummit Europe 2023 successful by spreading the word and submitting your proposal and sponsorship! Follow us on Twitter (@pulsarsummit) to receive the latest updates on the summit.

About Apache Pulsar​

Apache Pulsar is a cloud-native, distributed messaging and streaming platform that empowers companies around the world to manage trillions of events per day. The Pulsar community has witnessed rapid growth since it became a top-level Apache Software Foundation project in 2018. Over the past four years, the vibrant community keeps driving innovation and improvements to the project.

About the organizer​

StreamNative is proud to host Pulsar Virtual Summit Europe 2023. Founded by the original developers of Apache Pulsar and Apache BookKeeper, StreamNative builds a cloud-native event streaming platform that enables enterprises to easily access data as real-time event streams. As the core developers of Pulsar, the StreamNative team is deeply versed in the technology, the community, and the use cases. Today, StreamNative is focusing on growing the Apache Pulsar and BookKeeper communities and bringing its deep experience across diverse Pulsar use cases to companies across the globe.

Want to stay informed of the latest developments regarding Pulsar Virtual Summit Europe 2023? Sign up here to be the first to hear about open registration, exciting speaker announcements, and details for all things Pulsar Summit.

Highlights of the 2.11 release​

• Upgrade Pulsar server Java build version to 17 (PIP-156)
• Support chunking with shared subscriptions (PR-16202)
• Support pluggable topic factory (PR-12235)
• Support configurable compression type for ManagedCursorInfo to balance storage resource usage (PR-14542)

This blog documents the most noteworthy changes in this release. For the complete list, including all feature enhancements and bug fixes, check out the Pulsar 2.11 Release Notes.

Notable feature enhancements and fixes​

Broker​

Store subscription properties within metadata service (PR-15757)​

Issue

Restarting the broker or unloading topics may lead to the loss of subscription properties since they are not stored in the metadata service.

Resolution

Store the subscription properties within the metadata service (ManagedCursorInfo).

Support configurable compression type for ManagedCursorInfo (PR-14542)​

Issue

When the cursor data expands, the data size increases and it takes a lot of time to pull the data.

Resolution

Introduce various compression algorithms to support cursor compression.

Reference documentation: managedCursorInfoCompressionType

Sync Pulsar metadata across multiple clouds (PR-16425)​

Issue

Synchronizing configuration metadata (policies) is a critical path to sharing tenant/namespace/topic policies among clusters and administering policies uniformly across all clusters. However, syncing the metadata store (global zookeeper) among Pulsar clusters deployed on separate cloud platforms is not supported in earlier versions.

Resolution

Use two system topics metadataSyncEventTopic and configurationmetadataSyncEventTopic along with a metadata synchronizer to sync metadata among clusters deployed on different cloud platforms and persist local topic policies with it.

Reference documentation

• metadataSyncEventTopic
• configurationmetadataSyncEventTopic

Producer​

Fix client memory limit currentUsage leak in ProducerImpl (PR-16837)​

Issue

The client memory limit currentUsage leaks in ProducerImpl and causes the producer’s send rate to be slow.

Resolution

Release the memory when there are failed pending batch messages.

Release semaphore before discarding messages in batchMessageContainer (PR-17019)​

Issue

There is a race condition in batchMessageContainer when discarding the messages and releasing the semaphore, which causes the semaphore to be held.

Resolution

Release the semaphore before discarding the messages in batchMessageContainer.

Consumer​

Support consumer client memory limit (PR-15216)​

Issue

It takes time for an application with a large number of producers/consumers to select an appropriate value for its queue size. The same applies to topics with many partitions.

Resolution

1. Block the expansion of the consumer receiver’s queue size if the memory usage exceeds 75%.
2. Trigger the shrinking of the consumer receiver’s queue size if the memory usage exceeds 95%.

Support adding interceptors for readers (PR-14729)​

Issue

Pulsar supports adding interceptors for producers and consumers to implement message tracing, but there is no way to add interceptors for readers.

Resolution

1. Add a new interface ReaderInterceptor to customize the reader interceptor.
2. Support setting reader interceptor and auto-update partition configurations by using ReaderBuilder.

User documentation: Create a reader with an interceptor

Function​

Support Record as a new output type of Functions (PR-16041)​

Issue

In earlier versions, if you want to dynamically set an output topic, message properties, or change the output schema in a Pulsar function, the only way is to create a function that returns Void. It would be more intuitive to return a structure like Record that carries this information.

Resolution

Add a utility method newOutputRecordBuilder to Context that returns a FunctionRecord builder initialized with the information from the source record. The builder methods can be used to override these values as needed.

User documentation: Use Record as function output

Tiered storage​

Add a universal S3 provider for the offloader (PR-15710)​

Issue

Pulsar supports cloud storage compatible with S3 APIs, such as AWS and Aliyun, through existing offloaders, but the forced registration of specific metadata limits their use.

Resolution

Provide a more general offloader S3 to serve more S3-compatible storage, which uses pure JClouds S3 metadata and allows overriding the default JClouds properties through system properties.

User documentation: Use S3 offloader with Pulsar

Proxy​

Support PrometheusRawMetricsProvider for Pulsar proxies (PR-14681)​

Issue

The metrics of plugins could not be exposed to Prometheus for monitoring.

Resolution

Add PrometheusRawMetricsProvider in Pulsar proxies so that plugins can add their metrics to Prometheus.

What’s Next?​

If you are interested in learning more about Pulsar 2.11, you can download and try it out now!

For more information about the Apache Pulsar project and current progress, visit the Pulsar website, follow the project on Twitter @apache_pulsar, and join Pulsar Slack!

We are grateful to all of our community members and those in the wider open-source community who contributed to the project. They are the reasons behind every step Pulsar has made over the past year.

Now, let’s take a look at some of the highlights in 2022.

New Pulsar website​

In May 2022, the Apache Pulsar community launched its new website. The new appearance has a light blue background with the homepage structure redesigned. You can see PIP-87 and PIP-113 for more details. Following is a comparison of the websites.

Before:

After:

Many community members participated in the discussion and provided valuable feedback. Currently, the Pulsar community is still working to improve the reader experience by making key information more visible and accessible.

Key events recap​

In 2022, the Apache Pulsar community put together a number of meetups and events across the globe to share the latest messaging and streaming technologies. Among others, two Pulsar Summits received the most attention from community members.

• Pulsar Summit San Francisco 2022: This event marked the first-ever, in-person Pulsar Summit in North America with 200+ attendees. It featured 20 speakers from companies like Google, AWS, and Yahoo!, 5 keynotes on Apache Pulsar and other streaming technologies, and 12 breakout sessions on tech deep dives, use cases, and ecosystem talks. Watch the sessions.
• Pulsar Summit Asia 2022: With 1,500+ registrations and 40,000+ views globally, this two-day virtual event gathered 41 speakers giving 3 keynote addresses and 36 sessions. They came from companies like ByteDance, Huawei, Tencent, Nippon Telegraph and Telephone Corporation (NTT), Yum China, Netease, vivo, Nutanix, and StreamNative. Watch the sessions.

To stay up to date with the latest events, you can join the following meetups or groups:

• New York City Apache Pulsar Meetup
• SF Bay Area Apache Pulsar Meetup
• Netherlands Apache Pulsar Meetup
• Japan Pulsar User Group

Community growth​

The development of the Pulsar community would not be possible without our contributors. Among them, Pulsar Committers and PMC members have taken the lead in contributing to and promoting the project. In 2022, many new faces joined the community while we also welcomed old friends taking more responsibility. Let’s take a look at the Pulsar community by numbers.

• 598 global contributors for the Pulsar main GitHub repository
• 12.1K+ GitHub stars
• 3.2K+ forks
• 16 new Committers
• 5 new PMC members
• 9560 Pulsar Slack members
• 4.3K+ downloads
• 14M+ Docker pulls

The Pulsar community welcomes all kinds of contributions. For more information, see the Apache Pulsar Contribution Guide.

Project releases​

In 2022, the Pulsar community worked hard to improve the project’s capabilities and fix existing bugs with 1 major version and 8 minor versions. Many important Pulsar capabilities were delivered in these releases, such as cluster-level automatic failure recovery and lazy-loading added to PartitionedProducer. Check out each release below to see details:

• 2.10.2
• 2.10.1
• 2.10.0
• 2.9.3
• 2.9.2
• 2.8.4
• 2.8.3
• 2.8.2
• 2.7.5

For more information, see the Release Notes page.

Updates about clients, Pulsar Manager, and Pulsar Helm Chart are listed below:

• Pulsar C++ Client 3.1.0
• Pulsar C++ Client 3.0.0
• Pulsar Go Client 0.9.0
• Pulsar Go Client 0.8.1
• Pulsar Go Client 0.8.0
• Pulsar Node.js Client 1.7.0
• Pulsar Node.js Client 1.6.2
• Pulsar Python Client 3.0.0
• Reactive Client for Apache Pulsar 0.1.0
• Pulsar Manager 0.3.0
• Pulsar Helm Chart 3.0.0

For more information, see the Clients Release Notes page.

Pulsar ecosystem​

In 2022, the Pulsar community worked with other open-source communities to add more integrations to the Pulsar ecosystem. Notable integrations include:

• Spring for Apache Pulsar: This project aims to provide Spring-friendly APIs, building blocks, and programming models for writing Java applications that interact with Apache Pulsar. For more information, read the blog Spring for Apache Pulsar 0.1.0 available now.
• Flink-Pulsar Sink Connector: The Flink-Pulsar Sink Connector is part of the Flink-Pulsar DataStream Connector. It implements Flink’s new SinkV2 API and allows you to write Flink job results back to Pulsar topics seamlessly.
• HTTP Sink Connector: The HTTP Sink Connector pushes records to any HTTP server with the record value in the body of POST method. The body of the HTTP request is the JSON representation of the record value.

For more information, see the Ecosystem page.

What’s next in 2023​

The Pulsar community seeks to further improve the project from different aspects. For example, we plan on releasing the first long-term support (LTS) version of Pulsar to meet the needs of different users. For those who want a more stable Pulsar version, they can choose the LTS release. The Pulsar community will help improve and upgrade the LTS versions with bug fixes and security patches.

Additionally, we will also have more events coming for 2023, including Pulsar Summit North America and Pulsar Summit APAC. To stay up to date with community news and discuss hot topics with other members, you can subscribe to the Pulsar mailing lists for users and developers, follow us on Twitter, and join the Pulsar Slack workspace.

This blog walks through the most noteworthy changes. For the complete list including all feature enhancements and bug fixes, check out the Pulsar 2.10.3 Release Notes.

Add getState in transactions for client APIs (PR-17423)​

Issue​

In earlier versions, the state of transactions on the client side could not be obtained with the client APIs. Users have no way to check the state of transactions before ending them.

Resolution​

Add an interface to get the state of transactions.

Enable delayed transaction messages (PR-17548)​

Issue​

In earlier versions, delayed message delivery and transaction messages could not be used simultaneously. When sending a transaction message with a certain delay and committing this transaction, the messages sent by transactions are immediately received by consumers without any expected delay.

Resolution​

Allow clients to send delayed messages with transactions using trackDelayedDelivery.

Allow configuring and disabling the size of lookahead for detecting fixed delays in messages (PR-17907)​

Issue​

When detecting that all messages have a fixed delay time, it’s helpful that users can configure the size of lookahead to detect and pause the reads from cursors. But in earlier versions, the size of `lookahead is pre-defined and fixed, which lacks flexibility and limits the usage.

Resolution​

Allow configuring and disabling the size of lookahead for detecting fixed delays in messages.

Fix memory leak while offloading ledgers (PR-18500)​

Issue​

In earlier versions, BlockAwareSegmentInputStreamImpl never releases the BookKeeper entries in the close method, leading to OutOfDirectMemory errors on brokers that frequently run offloading activities. This PR fixes the incorrect if conditions.

Fix namespace cannot be deleted by force (PR-18307)​

Issue​

In earlier versions, when you delete a namespace by force, if __transaction_buffer_snapshot and __change_events (system topic for topic policies) have been deleted, then the deletion of regular topics will fail because it cannot clear the snapshots and topic policies.

Resolution​

Delete regular topics before deleting system topics when deleting a namespace. And deleting the system topic __change_events does not clear its topic policy.

What’s Next?

If you are interested in learning more about Pulsar 2.10.3, you can download and try it out now!

For more information about the Apache Pulsar project and current progress, visit the Pulsar website, follow the project on Twitter @apache_pulsar, and join Pulsar Slack!

Since its inception in 2020, Pulsar Summit Asia has received increasing attention from both Asia and beyond. For Pulsar Summit Asia 2022, more than 1400 people from companies like Amazon, Tencent, IBM, Huawei, Dell, ByteDance, and Splunk registered for this online event to discuss the latest messaging and streaming technologies powering a wide variety of industries like education, food, gaming, e-commerce, and social media.

Before we talk about some of the Summit highlights, we would like to thank the entire Apache Pulsar community and all the friends from other open-source communities for making this event a great success! Additionally, this event would not have drawn the attention of such a broad audience without our speakers, Program Committee members, as well as community and media partners. Thank you all for your help and energy!

And now, let’s look at some of the highlights and a round-up of this online virtual event:

• 1400+ registrations and 40,000+ views globally
• 41 speakers from companies like ByteDance, Huawei, Tencent, Nippon Telegraph and Telephone Corporation (NTT), Yum China, Netease, vivo, Nutanix, and StreamNative
• 3 keynotes on Apache Pulsar and event-driven applications
• 36 sessions on use cases, technical deep dives, and ecosystem talks
• 14 Program Committee members
• 18 community and media partners

Keynotes and sessions at a glance​

This two-day virtual event brought together engineers, architects, and data scientists from the messaging and streaming communities. They talked about Pulsar adoption for different use cases, event-driven platforms, technical details, and even Pulsar integration with other ecosystems. The following is a quick recap of some of the keynotes and sessions.

Keynotes​

• A Cloud-Native, Unified Messaging and Streaming System for Modern Data Infrastructure (Mandarin): Jia Zhai, an Apache Pulsar PMC member, gave a high-level overview of Apache Pulsar and explained how it meets the requirements for messaging and streaming with its cloud-native features.
• What You Should Know about Apache Pulsar in 2022 (Mandarin): Penghui Li, an Apache Pulsar PMC member, talked about some of the existing problems in Pulsar and how the Pulsar community would work to solve them going forward.
• Event-Driven Applications Done Right (English): Matteo Merli, Apache Pulsar PMC Chair, provided his insights on the fundamentals of modern event-driven applications.

Use cases​

• Awesome Pulsar in Yum China (English): Chauncey Yan from Yum China explained why Yum China selected Pulsar for production and shared their experience of performance tuning.
• Pulsar + Envoy: Building an OTO Marketing Platform for Different Business Scenarios on Microservices (Mandarin): Jason Jiang from Tencent shared their experience of using Pulsar and Envoy to create an OTO marketing platform built on microservices for different business scenarios.
• Pulsar in Smart Education: How NetEase Youdao Put Pulsar into Practice for Complex Business Scenarios (Mandarin): Jiaqi Shen from NetEase introduced NetEase Youdao’s practices of using Apache Pulsar to support complex scenarios in smart education.
• Streaming Wars and How Apache Pulsar is Acing the Battle (English): Shivji Kumar Jha and Sachidananda Maharana from Nutanix talked about how they adopted Pulsar for different use cases and migrated applications from other messaging solutions to Pulsar.
• Tens of Trillions of Messages: How Apache Pulsar Supports Big Data Business at Tencent (Mandarin): Dawei Zhang from Tencent discussed how they used Apache Pulsar for big data business to support scenarios requiring high availability and strong consistency.

Technical deep dives​

• Taking Jakarta JMS to New Generation Messaging Systems - Apache Pulsar (English): Enrico Olivelli and Mary Grygleski from DataStax explained how Pulsar concepts map to the Jakarta Messaging Specifications and demonstrated how to connect a Jakarta EE application to Pulsar.
• **A New Way of Managing Pulsar with Infrastructure as Code (Mandarin): **Max Xu and Fushu Wang from StreamNative discussed how to leverage the Terraform Provider for Pulsar and the Pulsar Resources Operator to help better manage Pulsar.
• A Deep Dive into Pulsar's Geo-replication for High Availability (Mandarin): Jialing Wang from China Mobile talked about the asynchronous and synchronous data replication mechanisms, and explained how they deployed Pulsar across multiple regions and improved its performance at China Mobile.
• Apache Pulsar in Volcano Engine E-MapReduce: Integration and Scenarios (Mandarin): Xin Liang from ByteDance introduced Volcano Engine E-MapReduce, a stateless, open-source big data platform, and how Pulsar fits into the platform’s ecosystem supporting different use cases.
• **Handling 100K Consumers with One Topic: Practices and Technical Details (English): **Hongjie Zhai from NTT Software Innovation Center shared their practices and technical details of handling 100K consumers with a single Pulsar topic.

Ecosystem​

• Pulsar + Flink + Camel: How Vertice Built its CMDB-based Real-time Data Platform (Mandarin): Wei Wang from Vertice offered his insights on how to build a CMDB-based Real-time Data platform with Pulsar, Flink, and Camel.
• Simplify Pulsar Functions Development with SQL (Mandarin): Rui Fu from StreamNative discussed how SQL syntax, Pulsar Functions, and Function Mesh can work together to deliver a unique user development experience for real-time data jobs in the cloud environment.
• Apache Pulsar + KubeEdge: Managing Edge Devices with Low Latency and Persistent Storage (Mandarin): Ryan Zhao from Huawei Cloud introduced a management solution for edge devices implemented through the Device Management Interface of KubeEdge and Apache Pulsar.
• Make Querying from Pulsar Easier: Introduce Flink Pulsar SQL Connector (English): Yufei Zhang from StreamNative walked through the basic concepts and examples of using Pulsar SQL Connector and discussed PulsarCatalog’s two different modes of using Pulsar as a metadata store.
• Migrating from RabbitMQ to Apache Pulsar: Using AMQP-on-Pulsar (AoP) in E-commerce Industry (Mandarin): Yifei Ming from Access Corporate Group talked about their experience of using the AMQP-on-Pulsar project to migrate RabbitMQ workloads to AoP.

What’s new in the Pulsar community​

In addition to the keynotes and sessions, we also shared some exciting news at the Summit.

Apache Pulsar has been adopted by organizations and users across the globe since it graduated as a Top Level Project in September 2018. Recently, the project witnessed its 580th contributor, almost hitting the 600 milestone!

So far this year, we have welcomed 16 new Apache Pulsar Committers to the Pulsar family. They have made continuous contributions to the Pulsar community and as Pulsar Committers, they now have write access to the Pulsar repository. They are:

• @RobertIndie
• @yuruguo
• @gaozhangmin
• @nodece
• @Shoothzj
• @hqebupt
• @StevenLuMT
• @lordcheng10
• @tisonkun
• @aloyszhang
• @mattisonchao
• @urfreespace
• @dlg99
• @nicoloboschi
• @liudezhi2098
• @cbornet

We also have 4 new members joining the Apache Pulsar Project Management Committee (PMC) for their merit for the evolution of the project. They are:

• @lhotari
• @michaeljmarshall
• @Technoboy-
• @Jason918

Congratulations to them all 🎉 ! And we are looking forward to more contributions from more friends in the broader open-source community.

More on Pulsar Summit Asia 2022​

All the sessions in Pulsar Summit Asia 2022 were pre-recorded and they will be uploaded to this YouTube account soon. You can also find the complete list of Summit sessions on this page.

At the same time, we will be working with some of the speakers to convert their speeches into blogs and case studies, which will be published soon.

Feel free to contact us at organizers@pulsar-summit.org if you have any questions and see you in the next Summit!

This August, we concluded Pulsar Summit SF, our first-ever, in-person event in North America. It witnessed over 12 breakout sessions and 5 keynotes with over 200 attendees from Apple, Blizzard, IBM, Optum, Iterable, Twitter, Uber, and many more. As we could see from the conference, there is an increase in the adoption of Pulsar and growing interest in messaging and streaming. And now, we are excited to invite you to Pulsar Summit Aisa 2022 to explore the latest messaging and streaming technologies!

Held on November 19th and 20th, this two-day virtual event will feature 36 sessions by developers, engineers, architects, and technologists from ByteDance, Huawei, Tencent, Nippon Telegraph and Telephone Corporation (NTT) Software Innovation Center, Yum China, Netease, vivo, WeChat, Nutanix, StreamNative, and many more. It will include sessions on Pulsar use cases, its ecosystem, operations, and technology deep dives.

Featured sessions​

Let’s have a quick look at some of the featured sessions.

Handling 100K Consumers with One Topic: Practices and Technical Details (English)​

Hongjie Zhai, Researcher, NTT Software Innovation Center

With the development of smart factories and connected vehicles, exchanging messages between a considerable number of devices is required for monitoring and controlling systems. In this connection, Apache Pulsar is one of the best solutions to keeping message pipelines simple, real-time, and safe. As the current message brokers are mainly designed for cloud services, users may face performance problems when they have too many consumers. This session will share the practices and technical details of handling 100K consumers with a single Pulsar Topic.

Awesome Pulsar in Yum China (English)​

Chauncey Yan, Backend Engineer, Yum China

Yum China Holdings, Inc. is the largest restaurant company in China with a vision of becoming the world’s most innovative pioneer in the restaurant industry. After its technical research on next-generation message queue solutions, it ultimately selected Pulsar as the implementation standard of message middleware for its technical middle platform. To date, Pulsar has been widely applied in different scenarios within Yum China such as business middle platform and system observability. This talk will be focused on why Yum China selected Pulsar for production use and its experience of performance tuning.

Streaming Wars and How Apache Pulsar is Acing the Battle (English)​

Shivji Kumar Jha, Staff Engineer, Nutanix

Sachidananda Maharana, MTS IV, Nutanix

This session will cover the operational challenges Nutanix has faced over the past 4 years when running Pulsar and how Pulsar fits into different use cases given its multi-tenancy and configurability. It will also talk about how Nutanix has aced these challenges to stick to Pulsar and even moved applications from other messaging solutions to Pulsar. It will end with the challenges and learnings on migrating from Kafka and Kinesis to Pulsar.

Pulsar + Envoy: Building an OTO Marketing Platform for Different Business Scenarios on Microservices (Mandarin)​

Jason Jiang, Senior Engineer, Tencent

In game marketing, the OTO (One-time Offer) model provides an effective way to improve user experience. A very common OTO scenario is to recommend certain material when the player has met the required conditions. To offer such a capability with low cost and high efficiency, you can use a customized Envoy plugin to support the Pulsar protocol. With Envoy’s flexible routing configurations and various functional plugins, you can provide solutions to different OTO business scenarios with microservices. In this session, Jason Jiang from Tencent will share their experience of using Pulsar and Envoy to create an OTO marketing platform built on microservices for different business scenarios.

Apache Pulsar in Volcano Engine E-MapReduce (Mandarin)​

Xin Liang, Senior Engineer, ByteDance

This session will introduce Volcano Engine E-MapReduce, a stateless, open-source big data platform, as well as the motivation for integrating Pulsar as a new cluster type into the Volcano Engine E-MapReduce ecosystem. It will cover some use cases of Pulsar in Volcano E-MapReduce, especially in real-time data warehouse and stream processing. It will also discuss typical real-time scenarios and common problems, and provide possible solutions powered by Pulsar with related services in Volcano Engine E-MapReduce.

A New Way of Managing Pulsar with Infrastructure as Code (Mandarin)​

Max Xu, Software Engineer, StreamNative

Fushu Wang, Cloud Engineer, StreamNative

Infrastructure as Code (IaC) is the process of managing and provisioning infrastructure resources through code instead of manual configurations. IaC offers the benefits of understandable code, idempotence, and consistency over traditional manual configurations. StreamNative developed the Terraform Provider for Pulsar and the Pulsar Resources Operator, which utilize Terraform and Kubernetes CRDs respectively to provide declarative management of Pulsar resources, such as tenants, namespaces, and topics. In this talk, two engineers from StreamNative will discuss how to leverage these two IaC tools to help you better manage Pulsar.

Stability Optimization of Apache Pulsar at Huawei Mobile Services (Mandarin)​

Lin Lin, Apache Pulsar PMC Member, SDE Expert, Huawei

HUAWEI Mobile Services is dedicated to enriching users’ lives with next-level content and services that meet every conceivable need and span diverse fields, including smart home, health and fitness, mobile office, apps, smart travel, and entertainment. Currently, HUAWEI Health, AppGallery, HUAWEI Video, and HUAWEI Mobile Cloud are all running on top of HUAWEI Mobile Services. In this session, Lin from Huawei will share their practices of using Apache Pulsar in complex business scenarios at Huawei Mobile Services and propose some of their enhancements for better stability.

To learn more about how companies and organizations today leverage Apache Pulsar for streaming and messaging, especially for mission-critical deployments in production, see the complete list of sessions in Pulsar Summit Asia 2022.

How to participate​

Pulsar Summit Asia 2022 is a virtual conference gathering speakers and audiences from different regions. As such, we arranged the conference schedule with time zones, regions, and languages all taken into consideration to provide the best possible experience.

All sessions on Day 1 will be presented in Mandarin, and all talks on Day 2 will be in English. For the Chinese audience, you can register on Huodongxing.com with your WeChat account. For non-Chinese or English audiences, you can register on this page to watch the English sessions with Zoom.

Register now for free! Contact us at organizers@pulsar-summit.org if you have any questions.

About the organizer​

StreamNative is the organizer of Pulsar Summit Asia 2022. Founded by the original developers of Apache Pulsar and Apache BookKeeper, StreamNative builds a cloud-native event streaming platform that enables enterprises to easily access data as real-time event streams. As the core developers of Pulsar, the StreamNative team is deeply versed in the technology, the community, and the use cases. Today, StreamNative is focusing on growing the Apache Pulsar and BookKeeper communities and bringing its deep experience across diverse Pulsar use cases to companies across the globe.

The highlight of the 2.7.5 release is that it fixes some critical bugs on broker, proxy, and storage, including message/data loss, broker deadlock, and connection leak. Note that 2.7.5 is the last release of 2.7.x.

This blog walks through the most noteworthy changes. For the complete list, including all feature enhancements and bug fixes, check out the Pulsar 2.7.5 Release Notes.

Fixed the deadlock on metadata cache missing while checking replications. PR-16889​

Issue​

After the changes in #12340, there are still a couple of places making blocking calls. These calls occupy all the ordered scheduler threads preventing the callbacks from completing until the 30 seconds timeout expires.

Resolution​

Change the blocking calls to async mode on the metadata callback thread.

Fixed the deadlock when using the key_shared mode. PR-11965​

Issue​

When the key_shared mode is used in consumers, deadlock may happen in the broker due to some race conditions and result in a lot of CLOSE_WAIT status connections.

Resolution​

Change unlock before the callback in the asyncDelete function of ManagedCursorImpl.

Fixed the message loss issue due to ledger rollover. PR-14703​

Issue​

If users config managedLedgerMaxLedgerRolloverTimeMinutes > 0, and the rollover happens when the ManagedLedger state is CreatingLedger, the messages written during that time are lost.

Resolution​

Rollover only when the ledger state is LedgerOpened.

Fixed the port exhaustion and connection issues in Pulsar Proxy. PR-14078​

Issue​

Pulsar Proxy can get into a state where it stops proxying Broker connections while Admin API proxying keeps working.

Resolution​

Optimize the proxy connection to fail-fast when the target broker isn't active. Fix the race conditions in Pulsar Proxy when establishing a connection, leading to invalid states and hanging connections. Add connection timeout handling to proxy connections. Add read timeout handling to incoming connections and proxied connections.

Fixed the compaction data loss due to missed compaction properties during cursor reset. PR-16404​

Issue​

The compaction reader seeks the earliest position to read data from the topic, but the compaction properties are missed during cursor reset, which leads to the initialized compaction subscribe without a compaction horizon, so the compaction reader skips the last compacted data. It only happens when initializing the compaction subscription and can be introduced by the load balance or topic unloading manually.

Resolution​

Keep the properties for resetting the cursor while the cursor is for data compaction. Copy the properties to the new mark delete entry while advancing the cursor, which is triggered by the managed ledger internal. It's not only for the compacted topic, and the internal task should not lose the properties when trimming the cursor.

What’s Next?

If you are interested in learning more about Pulsar 2.7.5, you can download and try it out now!

For more information about the Apache Pulsar project and current progress, visit the Pulsar website, follow the project on Twitter @apache_pulsar, and join Pulsar Slack!

In January this year, Pulsar Summit Asia 2021 (delayed due to the COVID-19 pandemic) featured more than 25 interactive sessions by technologists, developers, software engineers, and software architects from StreamNative, BIGO, China Mobile, Nutanix, Tencent, and more. The conference drew over 1000 attendees around the world, including attendees from top technology, internet, and media companies, such as Tencent, TikTok, Alibaba, and Microsoft.

Pulsar Summit Asia 2022 will be hosted virtually on November 19th and 20th, 2022. It is expected to cover the pivotal topics and technologies at the core of Apache Pulsar.

Join us and speak at Pulsar Summit Asia 2022​

Share your Pulsar story and speak at the summit! It is a great opportunity to participate and raise your profile in the rapidly growing Apache Pulsar community. We are looking for Pulsar stories that are innovative, informative, or thought-provoking. Here are some suggestions:

• Use cases, operations, tools, techniques, or the Pulsar ecosystem
• A deep dive into technologies
• Best practices and lessons learned
• A Pulsar success story or case study
• Anything else related to Pulsar that inspires the audience

To speak at the summit, submit an abstract about your session. All levels of talks (beginner, intermediate, and advanced) are welcome. Remember to keep your proposal short, relevant, and engaging. The following session formats are acceptable:

• Presentation: 40-minute presentation, maximum of 2 speakers
• Panel: 40-minute of discussion amongst 3 to 5 speakers
• Workshop: 60-90 minutes, in-depth, hands-on tutorials, maximum of 2 speakers
• Lightning talk: 10-minute presentation, maximum of 2 speakers

You need to pre-record your session after the proposal is approved. For time zone and network reasons, we do not recommend speakers present their talk live.

Dates to remember​

• CFP opens: August 22nd, 2022
• CFP closes: October 9th, 2022
• CFP notifications: October 19th, 2022
• Schedule announcement: November 4th, 2022
• Event dates: November 19th, 2022 - November 20th, 2022

Sponsor Pulsar Summit​

Pulsar Summit is a conference for the community and your support is needed. Sponsoring this event provides a great opportunity for your organization to further engage with the Apache Pulsar community. For more information on becoming a sponsor, contact us at organizers@pulsar-summit.org.

Help us make #PulsarSummit Asia 2022 a big success by spreading the word and submitting your proposal! Follow us on Twitter (@pulsarsummit) to receive the latest updates on the summit!

About Apache Pulsar​

Apache Pulsar is a cloud-native, distributed messaging and streaming platform that empowers companies around the world to manage trillions of events per day. The Pulsar community has witnessed rapid growth since it became a top-level Apache Software Foundation project in 2018. Over the past four years, the vibrant community keeps driving innovation and improvements to the project.

About the organizer​

StreamNative is the organizer of Pulsar Summit Asia 2022. Founded by the original developers of Apache Pulsar and Apache BookKeeper, StreamNative builds a cloud-native event streaming platform that enables enterprises to easily access data as real-time event streams. As the core developers of Pulsar, the StreamNative team is deeply versed in the technology, the community, and the use cases. Today, StreamNative is focusing on growing the Apache Pulsar and BookKeeper communities and bringing its deep experience across diverse Pulsar use cases to companies across the globe.

The highlight of the 2.9.3 release is introducing 30+ transaction fixes and improvements. Earlier-adoption users of Pulsar transactions have documented long-term use in their production environments and reported valuable findings in real applications. This provides the Pulsar community with the opportunity to make a difference.

This blog walks through the most noteworthy changes. For the complete list including all feature enhancements and bug fixes, check out the Pulsar 2.9.3 Release Notes.

Enabled cursor data compression to reduce persistent cursor data size. 14542​

Issue​

The cursor data is managed by the ZooKeeper/Etcd metadata store. When the data size increases, it may take too much time to pull the data, and brokers may end up writing large chunks of data to the ZooKeeper/Etcd metadata store.

Resolution​

Provide the ability to enable compression mechanisms to reduce cursor data size and the pulling time.

Reduced the memory occupied by metadataPositions and avoid OOM. 15137​

Issue​

The map metadataPositions in MLPendingAckStore is used to clear useless data in PendingAck, where the key is the position that is persistent in PendingAck and the value is the max position acked by an operation. It judges whether the max subscription cursor position is smaller than the subscription cursor’s markDeletePosition. If the max position is smaller, then the log cursor will mark to delete the position. It causes two main issues:

• In normal cases, this map stores all transaction ack operations. This is a waste of memory and CPU.
• If a transaction that has not been committed for a long time acks a message in a later position, the map will not be cleaned up, which finally leads to OOM (out-of-memory).

Resolution​

Regularly store a small amount of data according to certain rules. For more detailed implementation, refer to PIP-153.

Checked lowWaterMark before appending transaction entries to Transaction Buffer. 15424​

Issue​

When a client sends messages using a previously committed transaction, these messages are visible to consumers unexpectedly.

Resolution​

Add a map to store the lowWaterMark of Transaction Coordinator in Trasanction Buffer, and check lowWaterMark before appending transaction entries to Trasanction Buffer. So when sending messages using an invalid transaction, clients will receive NotAllowedException.

Fixed the consumption performance regression. PR-15162​

Issue​

This performance regression was introduced in 2.10.0, 2.9.1, and 2.8.3. You may find a significant performance drop with message listeners while using Java Client. The root cause is each message will introduce the thread switching from the external thread pool to the internal thread poll and then to the external thread pool.

Resolution​

Avoid the thread switching for each message to improve consumption throughput.

Fixed a deadlock issue of topic creation. PR-15570​

Issue​

This deadlock issue occurred during topic creation by trying to re-acquire the same StampedLock from the same thread when removing it. This will cause the topic to stop service for a long time, and ultimately with a failure in the deduplication or geo-replication check. The workaround is restarting the broker.

Optimized the memory usage of brokers.​

Issue​

Pulsar has some internal data structures, such as ConcurrentLongLongPairHashMap, and ConcurrentLongPairHashMap, which can reduce the memory usage rather than using the Boxing type. However, in earlier versions, the data structures were not supported for shrinking even if the data was removed, which wasted a certain amount of memory in certain situations.

Pull requests

• https://github.com/apache/pulsar/pull/15354
• https://github.com/apache/pulsar/pull/15342
• https://github.com/apache/pulsar/pull/14663
• https://github.com/apache/pulsar/pull/14515
• https://github.com/apache/pulsar/pull/14497

Resolution​

Support the shrinking of the internal data structures, such as ConcurrentSortedLongPairSet, ConcurrentOpenHashMap, and so on.

What’s Next?

If you are interested in learning more about Pulsar 2.9.3, you can download and try it out now!

Pulsar Summit San Francisco 2022 will take place on August 18th, 2022. Register now and help us make it an even bigger success by spreading the word on social media!

For more information about the Apache Pulsar project and current progress, visit the Pulsar website, follow the project on Twitter @apache_pulsar, and join Pulsar Slack!

The conference gathers adopters, developers, engineers, and technologists from some of the most influential open source communities in the world. To date, there has been a total of over 200 proposals submitted by presenters from Intel, Huawei, Tencent Cloud, StreamNative, Sina Weibo, vivo, and many more. Nearly 50 of these sessions are related to streaming and messaging, 30 of which are focused on Apache Pulsar-based technologies.

Featured sessions​

Let’s have a quick look at some of the featured sessions in messaging and streaming, ranging from technical deep dives, best practices, to tutorials, and insights.

FLiPN Awesome Streaming with Open Source (English)​

Timothy Spann, Developer Advocate, StreamNative

In this talk, Tim will walk through how to build different types of streaming applications by using Apache NiFi, Apache Flink, Apache Spark, and Apache Pulsar together. The session will demonstrate how to ingest various data and REST feeds to enrich data and send them to Apache Pulsar. Applications will be built on top of the live streaming data with Web socket dashboards, Apache Spark SQL ETL, and Apache Flink continuous SQL.

Introducing TableView: Pulsar's Database Table Abstraction (English)​

David Kjerrumgaard, Apache Pulsar Committer, Developer Advocate, StreamNative

In many use cases, applications are using Pulsar consumers or readers to fetch all the updates from a topic and construct a map with the latest value of each key for the messages that were received. The new TableView consumer offers support for this access pattern directly in the Pulsar client API itself and encapsulates the complexities of manually constructing such local caches manually. This talk will demonstrate how to use the new TableView consumer using a simple application and discuss best practices and patterns for using the TableView consumer.

Route to the Next-Generation Message Middleware: How vivo Migrated to Pulsar (Mandarin)​

Limin Quan, Jianbo Chen, Big Data Engineer, vivo

vivo has used Kafka to support its business with over 1000 billion messages per day. Now, it has migrated to Apache Pulsar as its next-generation message middleware to handle an even larger amount of data. In this talk, Quan and Chen will share the reasons behind vivo’s choice of Apache Pulsar and how vivo has worked to put Pulsar into practice (for example, migrating plans and troubleshooting tips).

Practice and Optimization: Apache Pulsar in Tencent Cloud (Mandarin)​

Xiaolong Ran, Apache Pulsar Committer, Senior R&D Engineer, Tencent Cloud

As Apache Pulsar has been put into production at scale in Tencent Cloud, it is widely used in different scenarios supporting companies and organizations across industries. To bring the experience to another level, Tencent has worked out a series of strategies in terms of optimization and stability. In this talk, Ran will focus on Tencent Cloud's work on performance optimization and shed light upon some of the best practices and troubleshooting tips for using Apache Pulsar.

Apache Pulsar as Lakehouse: Introducing the Lakehouse Tiered Storage Integration for Apache Pulsar (Mandarin)​

Hang Chen, Apache Pulsar PMC member, Software Engineer, StreamNative

Currently, tiered storage is introduced to offload cold data, while they are managed by Apache Pulsar in a non-open format. Therefore, it is very difficult to integrate the data into other big data components, such as Flink SQL and Spark SQL. In this talk, Chen will explain how to use Lakehouse to manage offloaded data and integrate it with the cold data offloading mechanism.

Build High-Performance Apache Pulsar with Intel Optane Persistent Memory (Mandarin)​

Fenghua Hu, Cloud Software Architect, Intel

Intel Optane persistent memory (PMem) is a revolutionary memory product, which features high performance, large capacity, storage persistence, and more. In this talk, Hu will demonstrate how to use Intel Optane PMem to bring Apache Pulsar’s ability of high throughput and low latency to another level and effectively cope with performance-demanding scenarios.

The Evolution of Apache Pulsar as A Message Queue in Huawei Device (Mandarin)​

Lin Lin, Apache Pulsar PMC member, SDE Expert, Huawei Device

Xiaotong Wang, Senior Engineer, Huawei Device

In the cloud-native era, Huawei Device is faced with many challenges in message queue infrastructure, such as maintenance difficulties in different message queue solutions, high overheads, and disaster tolerance ability building. This talk will cover Huawei Device’s experience in redesigning its message queue architecture and present some solutions to these problems.

To learn more about how companies and organizations today leverage Apache Pulsar for streaming and messaging, serverless computing, and mission-critical deployments in production, see other Apache Pulsar-related sessions in streaming and messaging tracks respectively.

How to participate​

Register now for free.

More resources​

As we can see from topics submitted to ApacheCon Asia 2022, Apache Pulsar has become one of the most active Apache projects over the past few years, with a vibrant community that continues to drive innovation and improvements to the project.

1. Save your spot at the Pulsar Summit San Francisco. The first in-person Pulsar Summit is taking place this August! Sign up today to join the Pulsar community and the messaging and event streaming community.
2. Join the Apache Pulsar community. Subscribe to the Pulsar mailing lists for user-related or Pulsar development discussions. You can also join the Pulsar Slack to ask quick questions or discuss specialized topics.