Kinesis Data Stream & Kinesis Firehose common use case

Amazon Kinesis Data Streams

You can use Amazon Kinesis Data Streams to collect and process a large number of stream data records in real-time/low latency. You can create data-processing applications, known as Kinesis Data Streams applications. A typical Kinesis Data Streams application reads data from a data stream as data records. These applications can use the Kinesis Client Library, and they can run on Amazon EC2 instances. You can send the processed records to dashboards, use them to generate alerts, dynamically change pricing and advertising strategies, or send data to various other AWS services. For information about Kinesis Data Streams features and pricing, see Amazon Kinesis Data Streams.

The following are typical scenarios for using Kinesis Data Streams:

1. Accelerated log and data feed intake and processing
2. Real-time metrics and reporting or data analytics
3. Complex stream processing like Directed Acyclic Graphs (DAGs) and many more

Kinesis Data Streams High-Level Architecture

https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html

How it works at a high level

Amazon Kinesis Data Firehose

Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service, Amazon OpenSearch Serverless, Splunk, and any custom HTTP endpoint or HTTP endpoints owned by supported third-party service providers, including Datadog, Dynatrace, LogicMonitor, MongoDB, New Relic, Coralogix, and Elastic. Kinesis Data Firehose is part of the Kinesis streaming data platform, along with Kinesis Data Streams, Kinesis Video Streams, and Amazon Kinesis Data Analytics. With Kinesis Data Firehose, you don’t need to write applications or manage resources. You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Kinesis Data Firehose to transform your data before delivering it.

High-level architecture and How it works at a high level

Building an application with common use case

The main goal of this application is to data receive using a kinesis data stream and store data in s3. Because the future data analytics team will analyze the data for s3 to their analyzing tool. To build this application, I used

• Serverless framework
• NodeJS
• TypeScript

Application Build With Kinesis Data Stream and Lambda consumer

AWS Service used

1. Kinesis Data Stream
2. Lambda
3. SQL
4. S3
5. IAM
6. CloudFormation

High-level Architecture

Data Stream, Lambda consumer, SQS for lambda event stream error and s3 for storing data

Serverless source code

GitHub - Rafat97/aws-kinesis-data-stream-lambda-s3: AWS kinesis data stream consume with lambda and…

Application Build With Kinesis Firehose

AWS Service used

1. Kinesis Data Stream
2. Kinesis Firehose
3. Lambda
4. SQL
5. S3
6. IAM
7. CloudFormation

High-level Architecture

Data Stream, Firehose , Lambda transformer and s3 for storing data and error

Serverless source code

GitHub - Rafat97/aws-kinesis-firehose-lambda-s3

References

What Is Amazon Kinesis Data Firehose?

What Is Amazon Kinesis Data Streams?