Logpush to Grafana Loki Endpoint

• Logpush to Grafana Loki Endpoint
  • Create the Worker
  • Logpush
    • Get Log Fields
    • Create Logpush Job
  • Debugging
  • Todo / Improvements

You can configure Cloudflare Logpush to send Logs to any HTTP destination, so let's send some logs to Grafana Loki.

NOTE: The auth is passed through to Loki based on a header in the Job definition in CloudFlare.

This Worker takes the incoming Logpush JSON Format and transforms it into something Loki understands:

• take incoming gzipped Logpush data, unpack it
• merge all fields into the Loki API format and send it off to the destination
• profit

Deploy in k8s

This fork is designed for running in k8s. deploy/ contains the yaml you'll need. Perhaps a helm chart will come later. Perhaps even a public docker image, for now you'll need to build it from the Dockerfile.

Create the Worker

• clone this repo
• update wrangler.toml to point to your Grafana Loki HTTP Endpoint
• feel free to make other adjustments in wrangler.toml to your liking
• deploy the worker:

npm run publish

Note down the returned URL to create the actual logpush job.

Logpush

Get Log Fields

First, get all the available log fields for your zone. To do this, get your Zone ID from the Cloudflare Dashboard and generate an API Key. Replace <ZONE_ID> and your-token-here in the following command:

Note: your Zone ID is not the hash in the url of cloudflare any more, it's in the overview page.

curl -s "https://api.cloudflare.com/client/v4/zones/<ZONE_ID>/logpush/datasets/http_requests/fields" -H "Authorization: Bearer your-token-here" -H 'Content-Type: application/json' | jq -r '.result|keys[]'|tr '\n' ',' | sed 's/,$/\n/'

You should see a comma separated list of available fields you can send to your destination

BotScore,BotScoreSrc,BotTags,CacheCacheStatus,CacheReserveUsed,CacheResponseBytes,CacheResponseStatus,CacheTieredFill,ClientASN,ClientCountry,ClientDeviceType,ClientIP,ClientIPClass,ClientMTLSAuthCertFingerprint,ClientMTLSAuthStatus,ClientRequestBytes,ClientRequestHost,ClientRequestMethod,ClientRequestPath,ClientRequestProtocol,ClientRequestReferer,ClientRequestScheme,ClientRequestSource,ClientRequestURI,ClientRequestUserAgent,ClientSSLCipher,ClientSSLProtocol,ClientSrcPort,ClientTCPRTTMs,ClientXRequestedWith,Cookies,EdgeCFConnectingO2O,EdgeColoCode,EdgeColoID,EdgeEndTimestamp,EdgePathingOp,EdgePathingSrc,EdgePathingStatus,EdgeRateLimitAction,EdgeRateLimitID,EdgeRequestHost,EdgeResponseBodyBytes,EdgeResponseBytes,EdgeResponseCompressionRatio,EdgeResponseContentType,EdgeResponseStatus,EdgeServerIP,EdgeStartTimestamp,EdgeTimeToFirstByteMs,FirewallMatchesActions,FirewallMatchesRuleIDs,FirewallMatchesSources,JA3Hash,OriginDNSResponseTimeMs,OriginIP,OriginRequestHeaderSendDurationMs,OriginResponseBytes,OriginResponseDurationMs,OriginResponseHTTPExpires,OriginResponseHTTPLastModified,OriginResponseHeaderReceiveDurationMs,OriginResponseStatus,OriginResponseTime,OriginSSLProtocol,OriginTCPHandshakeDurationMs,OriginTLSHandshakeDurationMs,ParentRayID,RayID,RequestHeaders,ResponseHeaders,SecurityLevel,SmartRouteColoID,UpperTierColoID,WAFAction,WAFFlags,WAFMatchedVar,WAFProfile,WAFRuleID,WAFRuleMessage,WorkerCPUTime,WorkerStatus,WorkerSubrequest,WorkerSubrequestCount,WorkerWallTimeUs,ZoneID,ZoneName

Create Logpush Job

Here's a curl request to create a Logpush Job with a lot of included fields already. You might want to update the fields= field below with the output from above. Please keep the &timestamps=unixnano:

The header_Authorization param should contain your username and password for Loki. You can generate this with https://www.debugbear.com/basic-auth-header-generator (I checked, it uses client-side JS and doesn't submit anywhere) if you don't have another choice.

curl --location --request POST 'https://api.cloudflare.com/client/v4/zones/<zone id>/logpush/jobs' \
--header 'X-Auth-Email: [email protected]' \
--header 'X-Auth-Key: abcSecretAPIKey' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "http",
    "logpull_options": "fields=BotScore,BotScoreSrc,CacheCacheStatus,CacheResponseBytes,CacheResponseStatus,CacheTieredFill,ClientASN,ClientCountry,ClientDeviceType,ClientIP,ClientIPClass,ClientRequestBytes,ClientRequestHost,ClientRequestMethod,ClientRequestPath,ClientRequestProtocol,ClientRequestReferer,ClientRequestURI,ClientRequestUserAgent,ClientSSLCipher,ClientSSLProtocol,ClientSrcPort,ClientXRequestedWith,EdgeColoCode,EdgeColoID,EdgeEndTimestamp,EdgePathingOp,EdgePathingSrc,EdgePathingStatus,EdgeRateLimitAction,EdgeRateLimitID,EdgeRequestHost,EdgeResponseBytes,EdgeResponseCompressionRatio,EdgeResponseContentType,EdgeResponseStatus,EdgeServerIP,EdgeStartTimestamp,FirewallMatchesActions,FirewallMatchesRuleIDs,FirewallMatchesSources,OriginIP,OriginResponseBytes,OriginResponseHTTPExpires,OriginResponseHTTPLastModified,OriginResponseStatus,OriginResponseTime,OriginSSLProtocol,ParentRayID,RayID,SecurityLevel,WAFAction,WAFFlags,WAFMatchedVar,WAFProfile,WAFRuleID,WAFRuleMessage,WorkerCPUTime,WorkerStatus,WorkerSubrequest,WorkerSubrequestCount,ZoneID&timestamps=unixnano",
    "destination_conf": "https://your.example.workers.dev?header_Authorization=Basic%20dXNlcm5hbWU6cGFzc3dvcmQK&job=lokiJobName",
    "max_upload_bytes": 5000000,
    "max_upload_records": 1000,
    "dataset": "http_requests",
    "frequency": "high",
    "enabled": true
}'

• please make sure to update the following fields in the curl request above:
• your own zone id instead of <zone id>
• your own API Key and/or E-Mail and Global API Key
• You might want to update the logpull_options (check API Docs)
• destination_conf enter your own workers domain in there, but keep the ?header_Authorization=Basic%20, just update the Authorization header with your own username and password
• Update the job query string to the stream name you want to see in Grafana later on.

If everything went according to plan, you should see your new Logpush Job in the Analytics => Log tab as well as in Grafana Loki. This is how it could look like.

Debugging

The initial creation of the logpush job will push one json line to Loki, you should see something along the lines:

{"content": "test", "filename": "test.txt"}

in your Loki logs.

Todo / Improvements

• Create some nice sample Dashboards to make sense out of the data
• Error Handling 🤷‍♂️