Serverless Data Stream with AWS

This is a template for AWS IoT data stream using Serverless framework.

Overview

The data flow of this template is below.

1. Send data from devices via MQTT
2. IoT Core subscribes data and send it to Kinesis Firehose
3. Kinesis executes batch (transform data, Putting data together)
4. Send batched data to S3

If you have a device, you can test with it. Without a device, you can imitate device behavior from AWS IoT core.

Setup

Installation

1. You need Serverless cli if you don't have it

$ npm install -g severless

2. AWS creadential setup if you need it

$ brew install awscli
$ aws configure

3. Clone this repo

$ git clone https://github.com/egurinko/serverless-data-stream.git
$ cd serverless-data-stream

4. deploy

$ sls deploy --stage <stage name>

5. remove

$ sls remove --stage <stage name>

Test

Send data to AWS IoT core

The topic name IoT core subscribe is iot/topic . If you have a device, please send data to it.

If you don't have a device, please go to IoT Core page Test tab. And publish data to iot/topic like below.

Check Kinesis batch

To check if kinesis is actually working, you have two ways.

1. Go to transformation Lambda function
   
   • go to lambda Monitoring tab
   • Check cloud watch logs below

If you can see cloud watch log, at least you data reaches to Lambda.

2. Check destination S3 bucket
   • Go to the destination bucket. If your setup succeeds, you can seed some object inside of th bucket

If you select data, you can see data you sent.

Links

• AWS
• AWS Documentation
• Serverless