Streaming data in AWS? Reducing complexity and cost is key.
Managed streaming data services on AWS can remove a lot of the pain in handling high throughput, low latency data streams. But not all services can deliver on your performance, low complexity, and low cost requirements. Redpanda Cloud helps you support mission critical, high performance data streams without running up those migraine-inducing AWS bills.
How to choose the right fully managed streaming data service
AWS offers a marketplace of options to support your real-time data streams. Two popular choices for fully managed data streaming—with data sovereignty in your own AWS account—are Redpanda Cloud and Amazon MSK. When evaluating these services, its critical to think about your specific needs for simplicity, performance, cost, and reliability.
Redpanda Cloud
|
Amazon MSK
|
|
Management Complexity
|
Managed service for Redpanda, a streaming data platform with brokers, HTTP proxy, schema registry, and Raft consensus plane built-in
Native Kafka API; no need to rewrite existing apps
Fully managed upgrades and patching
Single-AZ and Multi-AZ deployment options
Automatic data and partition balancing
Uses tiered storage to support retention limits and compaction for topics
|
Managed service for Apache Kafka brokers and Apache ZooKeeper nodes; HTTP proxy and schema registry are self-managed by the customer
Native Kafka API; no need to rewrite existing apps
Automatic hardware patches; version upgrades are self-managed by the customer
Multi-AZ only; Must deploy across 2 or 3 AZs
Need 3rd party tool (Cruise Control) for partition and data balancing
Due to Kafka limitation, compacted topics cannot
|
AWS integration
|
Uses S3 natively for cloud storage
Integrates with AWS services like Kinesis Data Analytics and AWS Lambda via native Kafka APIs
Available on the AWS Marketplace or Redpanda.com
|
Integrates with S3 using Kinesis Data Firehose
Integrates with AWS services like Kinesis Data Analytics and AWS Lambda via native Kafka APIs
Available directly through the AWS console and CLI
|
Performance
|
Achieves 10x lower average latencies vs Apache Kafka
Uses C++, thread-per-core architecture to maximize hardware utilization
|
Performance is comparable to Apache Kafka
Dependent on Apache Kafka, which experiences performance degradation due to JVM page caching, garbage collection
|
Cost of compute and storage
|
Uses 3-6x fewer compute resources than Apache Kafka when running the same workload
|
Resource consumption is comparable to Apache Kafka
|
Reliability
|
Raft-native architecture for replication is Jepsen- verified to be safe for data
99.9% uptime SLA
|
Uses Apache Kafka ISR for replication, which can be subject to unsafe leader election and data loss
99.9% uptime SLA
|
Connectors
|
Fully managed connectors for popular databases, storage, and analytics tools
|
Only self-managed connectors available (no 1st party support from AWS)
|
Data sovereignty
|
With BYOC clusters, data doesn’t leave your VPC, whether in AWS or GCP
|
Data doesn’t leave your VPC (limited to AWS)
|
Monitoring
|
Available natively via Prometheus
|
Available via CloudWatch (charged separately)
|
User interface
|
Redpanda Console is a fully integrated user interface, also compatible with Kafka
|
No first party UI; must use 3rd party tools like Redpanda Console
|
Multi-cloud
|
Support for AWS and Google Cloud, plus Azure for self-hosted Redpanda (Azure support coming soon for Redpanda Cloud)
|
No multi-cloud support; available only on AWS
|
Here's what you get with Redpanda Cloud
Redpanda Cloud is a fully managed streaming data service that provides everything you need to get from 0 to 1 with data streams in AWS.
Redpanda Cloud offers a complete streaming data platform in one fully managed service, including brokers, HTTP proxy, and schema registry. That leaves zero external dependencies for you to manage. And it's easy to get going with Redpanda's developer-friendly console, CLI, and API interfaces.
Redpanda's streaming data engine was designed to maximize the performance potential of modern infrastructure, resulting in 10x lower average latencies vs. Apache Kafka and Kafka-based services. This is due to innovations like Thread-per-Core architecture, C++ design, and async processing.
Redpanda's Bring Your Own Cloud (BYOC) offers a fully managed service, while keeping data in your own VPC environment to satisfy compliance and regulatory requirements. No more tradeoffs on operational agility and data sovereignty. Who says you can't have your cake and eat it too?
Redpanda Cloud helps reduce your infrastructure spend because its efficient streaming data engine runs on 3x fewer compute resources on average vs Apache Kafka. Redpanda's native S3 storage delivers up to 8-9x savings in long term data retention costs.
Redpanda Cloud offers reliable data streaming at scale with data safety, thanks to its Raft-based replication. It provides enterprise stability with atuomatic data and partition balancing, fully managed upgrades and patching, and high availability with a 99.9% uptime SLA.
Redpanda Cloud offers managed connectors for the most popular data platforms including MongoDB, Snowflake, S3, MySQL, PostgreSQL, SQL Server, and more. Additionally, Redpanda's native Kafka API integrates with the full range of Kafka ecosystem tools.
It's simple to deploy Redpanda in whatever way your business needs
Redpanda is available in deployment options that suit your unique needs, including fully managed Redpanda Cloud Services, a free-to-use Community Edition, and self-hosted Enterprise Edition.
"Redpanda was able to handle the same load on far less resources than Kafka required. It was 5 to 6 times more efficient, greatly reducing our hardware costs as well as support and licensing costs."
- Michael Pearce, Sr. Architect at StoneX
"Redpanda BYOC gives us a fully managed Kafka service running on our own cloud servers, balancing our internal compliance requirements with ease of use, and without compromising performance and compatibility."
- Kannan D.R., Enterprise Data Architect @ LiveRamp
"Integrating Redpanda into our CQRS architecture has not only improved the performance and reliability of our platform, but also made our lives easier by simplifying deployment and maintenance processes!"
- Juxhin Dyrmishi Brigjaj, Head of Engineering @ Exein