Merge pull request #1 from JoshKCarroll/initial-commit

Initial migration

.gitignore

@@ -5,6 +5,9 @@
 *.so
 *.dylib
 
+# dependency stuff
+/vendor/
+
 # Test binary, build with `go test -c`
 *.test
 

.travis.yml

@@ -0,0 +1,27 @@
+language: go
+go:
+- 1.x
+services: docker
+
+env:
+  global:
+  - KINESIS_ENDPOINT="localhost:4568"
+
+before_install:
+# Initialize localstack
+- docker-compose up -d
+# Download the dependencies
+- export GO111MODULE=on
+- go mod download
+- go mod vendor
+- go mod verify
+- go get github.com/mattn/goveralls
+- go get golang.org/x/tools/cmd/cover
+
+script:
+- curl --silent -OL https://raw.githubusercontent.com/vishnubob/wait-for-it/8ed92e8cab83cfed76ff012ed4a36cef74b28096/wait-for-it.sh
+- chmod a+x wait-for-it.sh && ./wait-for-it.sh $KINESIS_ENDPOINT
+- "$HOME/gopath/bin/goveralls -service=travis-ci -v"
+
+after_script:
+- docker-compose down

README.md

@@ -1,2 +1,43 @@
 # frinesis
-Frizzle for AWS Kinesis
+
+An AWS Kinesis implementation of a [Frizzle](https://github.com/qntfy/frizzle) Sink.
+
+In addition to the AWS Kinesis SDK for Go, Frinesis uses a modified version of
+[sendgridlabs/go-kinesis/batchproducer](https://github.com/sendgridlabs/go-kinesis) (under separate MIT license).
+
+Frizzle is a magic message (`Msg`) bus designed for parallel processing w many goroutines.
+  * `Receive()` messages from a configured `Source`
+  * Do your processing, possibly `Send()` each `Msg` on to one or more `Sink` destinations
+  * `Ack()` (or `Fail()`) the `Msg`  to notify the `Source` that processing completed
+
+## Running the tests
+
+`go test -v --cover ./...`
+
+## Configuration
+Frinesis Sinks are configured using [Viper](https://godoc.org/github.com/spf13/viper).
+```
+func InitSink(config *viper.Viper) (*Sink, error)
+
+InitSinkWithLogger(config *viper.Viper, logger *zap.Logger)
+```
+
+We typically initialize Viper through environment variables (but client can do whatever it wants,
+just needs to provide the configured Viper object with relevant values). The application might
+use a prefix before the below values.
+
+| Variable | Required | Description | Default |
+|---------------------------|:--------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------:|
+| AWS_REGION_NAME | required | region being used e.g. `us-east-1` |  |
+| KINESIS_ENDPOINT | optional | if using a custom endpoint e.g. for local testing. Defaults to AWS standard internal and retrieving credentials from IAM if not set. `http://` prefixed if no scheme set |  |
+| KINESIS_FLUSH_TIMEOUT | sink (optional) | how long to wait for Kinesis Sink to flush remaining messages on close (use duration) | `30s` |
+
+## Async Error Handling
+Since records are sent in batch fashion, Kinesis may report errors asynchronously.
+Errors can be recovered via channel returned by the `Sink.Events()` method.
+In addition to the `String()` method required by frizzle, currently only errors are
+returned by frinesis (no other event types) so all Events recovered will also conform
+to `error` interface.
+
+## Contributing
+Contributions welcome! Take a look at open issues.

batchproducer/LICENSE

@@ -0,0 +1,20 @@
+The MIT License (MIT)
+
+Copyright (c) 2013 SendGrid, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

batchproducer/README.md

@@ -0,0 +1,5 @@
+Kinesis `batchproducer` was forked from [sendgridlabs/go-kinesis](https://github.com/sendgridlabs/go-kinesis) 
+with the following significant modifications by Qntfy:
+* Use canonical AWS SDK Kinesis client instead of custom client written by sendgridlabs
+* Use [zap](https://github.com/uber-go/zap) for logging
+* Expose an `Events()` channel for asynchronous reporting of errors