Narwhal

Narwhal is a Ruby prefork worker designed to be broker agnostic and take advantage of features in *nix kernels

Architecture

Require ActiveJob
Require Ruby 2.x (copy-on-write)
Require *nix (select and signals)
Don't support daemonize or logfile, use deamontools, upstart, systemd, etc!

Adapters

Redis
Kafka
RabbitMQ
Beanstalkd

Master

state: wait_client is all clients are busy but messages are ready
- select(clients, sigqueue) to efficiently wait for next client to be ready (or signal handler)
state: wait_message is no messages are ready but clients are ready
- select(mq, sigqueue) to efficiently wait for next message to be ready (or signal handler)
state: running is clients are ready and messages are ready
- fetch message, pass to next client

Worker

state: ready is waiting for next message
- select(message, sigqueue)
state: processing is handling a job

Notes

Master init:
- spawn workers (prefork)
- all workers should start in ready state
Master tick:
- did we receive a signal? handle
- no clients available? select(clients, sigqueue)
- clients available but no messages? select(mq, sigqueue)
- clients available and messages? fetch next message and send to available client
Worker tick:
- select(message, sigqueue)
ActiveJob adapter for Narwhal
MQ adapter to Redis
MQ adapter for Kafka
INT/TERM - quick shutdown, kills all workers immediately
Should load app on startup
Should use ActiveJob to load jobs
REFACTOR: _after_fork and _before_fork for both master and worker internally
Add rails generator for configuring Narwhal

References

License

The MIT License (MIT)

Copyright (c) 2014 Brian Smith [email protected]