Walrus is a modern distributed message streaming platform that provides high performance, fault tolerance, and scalability. It is ideal for microservices, logging systems, real-time analytics, and any applications where reliable data delivery is critical.
Project repository: Walrus on GitHub
Why Walrus is Needed
Walrus is perfect for applications that require high reliability and scalable data streams. Examples of use:
- Processing large streams of events in real-time.
- Logging and monitoring systems.
- Microservices architecture with message exchange.
- Building ETL processes and real-time data analytics.
Main advantages:
- Fault Tolerance - automatic leader rotation and data replication.
- High Performance - the log-based storage structure provides fast access to messages.
- Scalability - segmentation and load distribution across nodes.
Real Use Cases for Walrus
Walrus is suitable for a wide range of tasks where high performance and reliable message streaming are important. For example, web applications can use it to stream user events and logs in real-time, helping to quickly detect errors and analyze user behavior. In microservices architecture, Walrus acts as a reliable intermediary for message exchange between services, ensuring consistency and fault tolerance. Additionally, the platform is excellent for building ETL processes and analytics systems, where data from various sources is collected, processed, and delivered in real-time for further processing and visualization.
How Walrus Works
Walrus is designed as a distributed message streaming system that combines a high-performance log engine with Raft consensus for coordinating metadata:
- Segmentation: Topics are divided into segments (approximately ~1,000,000 messages each). Each segment has a leading node where writes are directed. When a segment fills up, leadership automatically switches to another node, balancing the load across the cluster.
- Raft Consensus: The Raft protocol is used only for metadata — for example, to agree on which node is responsible for which segment. Message data does not go through the Raft layer, making the critical write path very fast.
- Write Fencing: Only the leader node of a segment can perform writes to it. This prevents “split-brain” situations where multiple nodes might write to the same segment simultaneously.
- Reading: As segments fill up, they are “sealed.” Reading can be done from any node that has a copy of the sealed segment without moving the data.
- Client Protocol: Clients (producers and consumers) can connect to any node in the cluster — Walrus automatically redirects the request to the correct segment leader.
The Walrus architecture looks like this:
Installing Walrus
To run Walrus, you will need Go and Docker (optional).
Installation via Go
Clone the repository
git clone https://github.com/nubskr/walrus.gitcd walrusBuild the project
go build -o walrus main.goRun
./walrusInstallation via Docker
docker pull nubskr/walrusdocker run -d --name walrus -p 8080:8080 nubskr/walrusAfter starting, Walrus will be available on port 8080 by default.
Using Walrus
Walrus provides an API for publishing and subscribing to messages. Main use cases:
1. Publishing Messages
curl -X POST http://localhost:8080/topics/my-topic/messages \
-H "Content-Type: application/json" \
-d '{"key": "user1", "value": "Hello, Walrus!"}'2. Subscribing to Messages
curl http://localhost:8080/topics/my-topic/subscribe3. Creating Topics
curl -X POST http://localhost:8080/topics \
-H "Content-Type: application/json" \
-d '{"name": "my-topic", "partitions": 3}'Walrus automatically distributes data across segments and ensures consistency using Raft.
Conclusion
Walrus is a powerful and fault-tolerant platform for working with data streams. Start by running it locally via Docker or Go, create a topic, and try publishing your first messages. Thanks to its performance and scalability, Walrus will become a reliable tool for your project.
FAQ
- 1. Is Walrus suitable for small projects?
Yes, Walrus can be used locally for testing and small projects, but its strengths are fully realized in distributed systems. - 2. How is fault tolerance ensured?
Through automatic leader rotation and data replication between nodes using Raft. - 3. Can it be integrated with other systems?
Yes, Walrus can work with any service that supports HTTP API for publishing and subscribing to messages. - 4. Is there official documentation?
Full documentation is available on GitHub Walrus.
