Why are the md5 values ​​the same? when using httpx as sdk.

[r00tuser111]

I am using httpx as sdk, below is my test code.

package main

import (
	"fmt"
	"log"

	"github.com/projectdiscovery/goflags"
	"github.com/projectdiscovery/gologger"
	"github.com/projectdiscovery/gologger/levels"
	"github.com/projectdiscovery/httpx/runner"
)

func main() {
	gologger.DefaultLogger.SetMaxLevel(levels.LevelFatal) // increase the verbosity (optional)
	targets := []string{
		"www.hackerone.com",
		"docs.projectdiscovery.io",
	}
	var parseTarget = goflags.StringSlice(targets)
	options := runner.Options{
		Methods:         "GET",
		InputTargetHost: parseTarget,
		NoFallback:      true,
		Exclude:         goflags.StringSlice{"private-ips"},
		Hashes:          "md5",
		OnResult: func(r runner.Result) {
			// handle error
			if r.Err != nil {
				return
			}
			if r.StatusCode != 400 && r.StatusCode != 502 {
				fmt.Println(r.URL, r.Port, r.Host, r.Hashes)
			}
		},
	}
	options.CustomPorts.Set("80,8443,443,82,8881,7200,9981,444")
	options.Silent = true
	if err := options.ValidateOptions(); err != nil {
		log.Fatal(err)
	}

	httpxRunner, err := runner.New(&options)
	if err != nil {
		log.Fatal(err)
	}
	defer httpxRunner.Close()

	httpxRunner.RunEnumeration()

}

After running, the output is as follows：

https://docs.projectdiscovery.io:443 443 76.76.21.93 map[body_md5:d41d8cd98f00b204e9800998ecf8427e header_md5:54e363ee4a536ec6c93fd3fe98e68563]
https://www.hackerone.com:443 443 172.64.151.42 map[body_md5:d41d8cd98f00b204e9800998ecf8427e header_md5:6db24d5c6ff1386a699f23031c3e68be]
http://docs.projectdiscovery.io:80 80 76.76.21.123 map[body_md5:d41d8cd98f00b204e9800998ecf8427e header_md5:bf62c669adc3798fd7608d1d9e9e8bf9]

Notice that the body md5 of www.hackerone.com and docs.projectdiscovery.io is: d41d8cd98f00b204e9800998ecf8427e.

This is very unreasonable, because it is obvious that the md5 of the two websites should be inconsistent.

[r00tuser111]

This md5 is the hash value of the empty string.

Now the question becomes why body is empty.

[r00tuser111]

After some source code analysis, I found that there is a configuration item named: h.Options.MaxResponseBodySizeToRead in httpx/common/httpx/httpx.go.

if !generic.EqualsAny(httpresp.StatusCode, http.StatusSwitchingProtocols, http.StatusNotModified) {
var err error
respbody, err = io.ReadAll(io.LimitReader(httpresp.Body, h.Options.MaxResponseBodySizeToRead))
if err != nil && !shouldIgnoreBodyErrors {
return nil, err
}
}

Set MaxResponseBodySizeToRead: 2147483647,

The configuration item corresponding to cli should be

-rstr, -response-size-to-read int max response size to read in bytes (default 2147483647)

Now this problem is solved.

https://www.example.com:9981 9981 1.1.1.1 map[body_md5:d41d8cd98f00b204e9800998ecf8427e header_md5:72f1505a3fd0855fceba60a6818b350f]
https://www.example.com:82 82 1.1.1.1 map[body_md5:d41d8cd98f00b204e9800998ecf8427e header_md5:379f1be51c759f8fc5fe3e3e318f7c5a]