Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom JSON format for Caddy log structure #2699

Open
FanelliMarco opened this issue Jul 30, 2024 · 6 comments
Open

Custom JSON format for Caddy log structure #2699

FanelliMarco opened this issue Jul 30, 2024 · 6 comments
Labels
log/date/time format question

Comments

@FanelliMarco
Copy link

FanelliMarco commented Jul 30, 2024 •
edited
Loading

hi allinurl
I'm trying to use GoAccess to analyze my Caddy logs, but I'm having trouble creating a custom JSON format (using docker)

{"level":"info","ts":1624526415.449846,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"192.168.0.1","remote_port":"22","client_ip":"192.168.0.3","proto":"HTTP/1.1","method":"GET","host":"example.com","uri":"/","headers":{"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["example.com"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["10.0.0.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0  Chrome/127.0.0.0 Safari/537.36"],"Accept-Language":["en-US,en;q=0.9"]}},"bytes_read":0,"user_id":"","duration":0.001574238,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}

this is my goaccess.conf file for now

addr 0.0.0.0
port 7890
daemonize false
real-time-html true
log-format json={"level":"%e","ts":%T,"logger":"%v","msg":"%r","remote_ip":"$.request.remote_ip","remote_port":"$.request.remote_port","client_ip":"$.request.client_ip","proto":"$.request.proto","method":"$.request.method","host":"$.request.host","uri":"$.request.uri","bytes_read":%b,"user_id":"%e","duration":%D,"size":%b,"status":%s}, ignore-null
time-format %s
date-format %s
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html

i get this errors

FILE: /var/log/caddy/access.log
2024-07-31 00:28:18 ==1== Parsed 2 lines producing the following errors:
2024-07-31 00:28:18 ==1==
2024-07-31 00:28:18 ==1== Token 'h-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["example.com"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["10.0.0.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0  Chrome/127.0.0.0 Safari/537.36"],"Accept-Language":["en-US,en;q=0.9"]' doesn't match specifier '%s'
2024-07-31 00:28:18 ==1==
2024-07-31 00:28:18 ==1== Format Errors - Verify your log/date/time format

2024-07-31 00:50:40  [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-07-31 00:50:40 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-07-31 00:50:40 ==1== Config file: /etc/goaccess/goaccess.conf
2024-07-31 00:50:40 ==1== https://goaccess.io - <[email protected]>
2024-07-31 00:50:40 ==1== Released under the MIT License.
2024-07-31 00:50:40 ==1==
2024-07-31 00:50:40 ==1== FILE: /var/log/caddy/access.log
2024-07-31 00:50:40 ==1== Parsed 2 lines producing the following errors:
2024-07-31 00:50:40 ==1==
2024-07-31 00:50:40 ==1== A valid date is required.
2024-07-31 00:50:40 ==1== A valid date is required.
2024-07-31 00:50:40 ==1==
2024-07-31 00:50:40 ==1== Format Errors - Verify your log/date/time format

2024-07-31 00:59:12  [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-07-31 00:59:12 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-07-31 00:59:12 ==1== Config file: /etc/goaccess/goaccess.conf
2024-07-31 00:59:12 ==1== https://goaccess.io - <[email protected]>
2024-07-31 00:59:12 ==1== Released under the MIT License.
2024-07-31 00:59:12 ==1==
2024-07-31 00:59:12 ==1== FILE: /bin/sh
2024-07-31 00:59:12 ==1== Parsed 10 lines producing the following errors:
2024-07-31 00:59:12 ==1==
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1== Incompatible format due to early parsed line ending '\0'.
2024-07-31 00:59:12 ==1==
2024-07-31 00:59:12 ==1== Format Errors - Verify your log/date/time format
@allinurl
Copy link
Owner

This should do it:

# goaccess access.log --log-format=CADDY --date-spec=min

2024-07-30-211748_562x313_scrot

@FanelliMarco
Copy link
Author

I updated goaccess.conf as follows

addr 0.0.0.0
port 7890
daemonize false
real-time-html true
log-format CADDY
date-spec min
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html

it gives me this error

2024-07-31 06:28:49  [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-07-31 06:28:49 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-07-31 06:28:49 ==1== Config file: /etc/goaccess/goaccess.conf
2024-07-31 06:28:49 ==1== https://goaccess.io - <[email protected]>
2024-07-31 06:28:49 ==1== Released under the MIT License.
2024-07-31 06:28:49 ==1==
2024-07-31 06:28:49 ==1== FILE: /bin/sh
2024-07-31 06:28:49 ==1== Parsed 10 lines producing the following errors:
2024-07-31 06:28:49 ==1==
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1== IPv4/6 is required.
2024-07-31 06:28:49 ==1==
2024-07-31 06:28:49 ==1== Format Errors - Verify your log/date/time format

@allinurl
Copy link
Owner

It looks like the first 10 lines of your JSON log may not be valid. Could you please share the first 20 lines directly from your access.log?

@FanelliMarco
Copy link
Author

FanelliMarco commented Jul 31, 2024 •
edited
Loading

these are the same logs that I provided earlier basically. I don't know if I'm doing anything in particular wrong. In goaccess.conf file, i have specified the log format as CADDY, which is not compatible with the JSON log format produced by Caddy (i think).

{"level":"info","ts":1722377868.638059,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"1234","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXXXXXX","uri":"/","headers":{"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["XXXXXXX"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["XXXXXXX"],"Accept-Language":["en-US,en;q=0.9"]}},"bytes_read":0,"user_id":"","duration":0.001574238,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"info","ts":1722377884.6235218,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"1234","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXXXXXX","uri":"/","headers":{"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Fetch-Mode":["navigate"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"X-Forwarded-Proto":["https"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Priority":["u=0, i"],"Sec-Fetch-User":["?1"],"X-Forwarded-Host":["XXXXXXX"],"User-Agent":["XXXXXXX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Accept-Encoding":["gzip, deflate, br, zstd"]}},"bytes_read":0,"user_id":"","duration":0.000091731,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"info","ts":1722399905.8303173,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"1234","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXXXXXX","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"],"Priority":["u=0, i"],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"User-Agent":["XXXXXXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Host":["XXXXXXX"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Site":["none"]}},"bytes_read":0,"user_id":"","duration":0.00088854,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"info","ts":1722452074.5979362,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"anonymized-host","uri":"/favicon.ico","headers":{"X-Forwarded-Host":["XXXXXXX"],"X-Forwarded-Proto":["https"],"User-Agent":["XXXXXXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["image"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Authorization":["REDACTED"],"Priority":["u=1, i"],"Referer":["XXXXXXX"],"Sec-Gpc":["1"],"If-Modified-Since":["Mon, 14 Feb 2022 05:51:54 GMT"],"If-None-Match":["W/\"47e-17ef6c99890\""],"Sec-Ch-Ua":["\"Not)A;Brand\";v=\"99\", \"Brave\";v=\"127\", \"Chromium\";v=\"127\""],"Sec-Ch-Ua-Platform":["\"Windows\""]}},"bytes_read":0,"user_id":"root","duration":0.000941437,"size":0,"status":304,"resp_headers":{"X-Xss-Protection":["1; mode=block"],"Etag":["W/\"47e-17ef6c99890\""],"Date":["Wed, 31 Jul 2024 18:54:34 GMT"],"Content-Security-Policy":["default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self'; frame-ancestors 'self'; base-uri 'self'; form-action 'self';"],"Accept-Ranges":["bytes"],"Cache-Control":["public, max-age=0"],"Last-Modified":["Mon, 14 Feb 2022 05:51:54 GMT"],"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["SAMEORIGIN"],"Vary":["Accept-Encoding"]}}
{"level":"info","ts":1722377868.638059,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXX","uri":"/","headers":{"Priority":["u=0, i"],"Sec-Ch-Ua":[""Not)A;Brand";v="99", "Brave";v="127", "Chromium";v="127""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8"],"Sec-Fetch-Site":["none"],"X-Forwarded-Host":["XXX"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua-Platform":[""Windows""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["anonymized-user-agent"],"Accept-Language":["en-US,en;q=0.9"]}},"bytes_read":0,"user_id":"","duration":0.001574238,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm="restricted""]}}
{"level":"info","ts":1722377884.6235218,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXX","uri":"/","headers":{"Sec-Ch-Ua":[""Not)A;Brand";v="99", "Brave";v="127", "Chromium";v="127""],"Sec-Fetch-Mode":["navigate"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua-Platform":[""Windows""],"Sec-Gpc":["1"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"X-Forwarded-Proto":["https"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Site":["none"],"Priority":["u=0, i"],"Sec-Fetch-User":["?1"],"X-Forwarded-Host":["XXX"],"User-Agent":["anonymized-user-agent"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8"],"Accept-Encoding":["gzip, deflate, br, zstd"]}},"bytes_read":0,"user_id":"","duration":0.000091731,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm="restricted""]}}
{"level":"info","ts":1722399905.8303173,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"XXX.XXX.XXX.XXX","remote_port":"XXXX","client_ip":"XXX.XXX.XXX.XXX","proto":"HTTP/1.1","method":"GET","host":"XXX","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"X-Forwarded-For":["XXX.XXX.XXX.XXX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8"],"Priority":["u=0, i"],"Sec-Ch-Ua":[""Not)A;Brand";v="99", "Brave";v="127", "Chromium";v="127""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":[""Windows""],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Gpc":["1"],"User-Agent":["anonymized-user-agent"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Host":["XXX"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Site":["none"]}},"bytes_read":0,"user_id":"","duration":0.00088854,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm="restricted""]}}

maybe i need to put in the goaccess.conf file something like this

addr 0.0.0.0
port 7890
daemonize false
real-time-html true
log-format "%h %^[%d:%t %^] \"%r\" %s %b \"%u\" \"%H\" \"%R\""
date-spec %d:%t
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html

@allinurl
Copy link
Owner

allinurl commented Aug 3, 2024

That same command using the CADDY log format works for me.

Please try using --no-global-config and try:

# goaccess access.log --log-format=CADDY --date-spec=min

@FanelliMarco
Copy link
Author

FanelliMarco commented Aug 4, 2024 •
edited
Loading

i change goaccess.conf with this

addr 0.0.0.0
port 7890
daemonize false
real-time-html true
no-global-config true
log-format CADDY
date-spec min
debug-file /var/log/goaccess/debug.log
log-file /var/log/caddy/access.log
output /var/www/goaccess/index.html

and still give me the same error, at this point i really don't know

2024-08-04 18:18:06  [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}
2024-08-04 18:18:06 ==1== GoAccess - version 1.9.3 - Jul 22 2024 13:14:37
2024-08-04 18:18:06 ==1== Config file: /etc/goaccess/goaccess.conf
2024-08-04 18:18:06 ==1== https://goaccess.io - <[email protected]>
2024-08-04 18:18:06 ==1== Released under the MIT License.
2024-08-04 18:18:06 ==1==
2024-08-04 18:18:06 ==1== FILE: /bin/sh
2024-08-04 18:18:06 ==1== Parsed 10 lines producing the following errors:
2024-08-04 18:18:06 ==1==
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 Cleaning up resources...
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1== IPv4/6 is required.
2024-08-04 18:18:06 ==1==
2024-08-04 18:18:06 ==1== Format Errors - Verify your log/date/time format
2024-08-04 18:18:07  [SETTING UP STORAGE cat /var/log/goaccess/debug.log] {0} @ {0/s}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
log/date/time format question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@allinurl @FanelliMarco