In this step, you create Kinesis Data Stream and Delivery Stream which can ingest data, transform data from JSON to Parquet format using Glue Data Catalog Schema and then write to the S3 bucket.
-
Goto the Kinesis Management console, select Kinesis Data Streams as the option and click on the Create data stream button.
-
On the next screen, type in the stream name as dojostream. Type in 1 for the Number of open shards. Click on the Create data stream button.
-
The data stream is created in no time. Next click on the Kinesis Firehose in the left menu and click on the Create Delivery Stream button.
-
On the next screen, type in dojodeliverystream as the delivery stream name. Select Kinesis Data Stream option as the source. Select dojostream as the kinesis data stream and then click on the Next button.
-
On the next screen, in the Convert record format section, select Enabled for the Record format conversion. Select Apache Parquet for the output format. Select Ireland as the AWS Glue Region. Select dojodatabase as the Glue Database. Select dojotable as the Glue Table. Select Latest for the table version and then click on the Next button. You configured Parquet as the destination format for the transformation.
-
On the next screen, select Amazon S3 as the destination. Select dojo-kinesis-destination as the bucket. If you created bucket with a different name, then select that one. Click on the Next button.
-
On the next screen, type in 60 for the Buffer interval. Select Choose existing IAM role option and select dojokinesisrole as the IAM Role. Keep rest of the fields to the default and click on the Next button.
-
On the next Review screen, click on the Create delivery stream button. The delivery stream is created in no time.
-
The next step is to create client application which will publish data to the Kinesis stream.