support alternative raw without 0.0.0.0 #2756

ldlac · 2024-10-16T05:42:06Z

Support this request #2719

how it should be used python updateHostsFile.py -a -g -i None

I made it that it cannot be combined with -c or -m, should it be?

welcome · 2024-10-16T05:42:09Z

Thank you for submitting this pull request! We’ll get back to you as soon as we can!

ldlac · 2024-10-16T05:42:53Z

updateHostsFile.py

@@ -259,6 +259,10 @@ def main():
    source_data_filename = settings["sourcedatafilename"]
    no_unified_hosts = settings["nounifiedhosts"]

+    settings["targetip"] = (
+        None if settings["targetip"] == "None" else settings["targetip"]


I preferred to go with an explicit None instead of ""

StevenBlack · 2024-10-17T22:57:56Z

Hey @ldlac thank you for this! Looks cool! Looks thisclose 🤏🏻 to mergable, right out of the box, which is awesome 👍🏻

Can I please ask for two additional things?

Can you please make -- ip None case insensitive? So -- ip none will work too. I might go as far as suggesting anything with non-numeric first character following --ip should probably work too, but I leave this to you to decide.
Can you please add documentation in the Command line options section of readme_template.md file?

Good work!

ldlac · 2024-10-18T00:38:45Z

Good recommendation for the --ip flag

I added an insightful explanation in the readme, let me know what you think

ldlac · 2024-10-18T12:30:10Z

By doing so, we'll break ipv6 no?

StevenBlack · 2024-10-18T12:34:41Z

@ldlac good point.

ldlac · 2024-10-19T22:51:58Z

I changed it to str(settings["targetip"]).lower() == "none" which will make it case insensitive

StevenBlack · 2024-10-19T23:22:41Z

@ldlac thank you very much! All tests pass, and all this behaves exactly as expected.

Except for one thing 😄

With the --ip none flag here are the first 50-lines of what is now not a hosts file (see below ⬇️⬇️)

Observe: we still get that header section with 0.0.0.0 localhost etc.

Expected: (well, I would expect...) no such header, since this is not a hosts file now. So maybe this PR should also bracket the code that writes the header section.

I also now wonder... should the --ip none flag be a super flag, override every other flag, and produce a file named domains.txt imstead of hosts, and additionally strip all comments and blank lines, and sort the domains too?

I'm willing to be convinced otherwise in any and all these things.

Welcome to Open Source 🙃

The first 50 lines of the (misnamed) hosts file produced by the --ip none flag:

# Title: StevenBlack/hosts
#
# This hosts file is a merged collection of hosts from reputable sources,
# with a dash of crowd sourcing via GitHub
#
# Date: 19 October 2024 23:10:22 (UTC)
# Number of unique domains: 117,213
#
# Fetch the latest version of this file: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
# Project home page: https://github.com/StevenBlack/hosts
# Project releases: https://github.com/StevenBlack/hosts/releases
#
# ===============================================================

127.0.0.1 localhost
127.0.0.1 localhost.localdomain
127.0.0.1 local
255.255.255.255 broadcasthost
::1 localhost
::1 ip6-localhost
::1 ip6-loopback
fe80::1%lo0 localhost
ff00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
0.0.0.0 0.0.0.0

# Custom host records are listed here.


# End of custom host records.
# Start StevenBlack

#=====================================
# Title: Hosts contributed by Steven Black
# http://stevenblack.com

ad-assets.futurecdn.net
ck.getcookiestxt.com
eu1.clevertap-prod.com
wizhumpgyros.com
coccyxwickimp.com
webmail-who-int.000webhostapp.com
010sec.com
01mspmd5yalky8.com
0byv9mgbn0.com
ns6.0pendns.org
dns.0pengl.com

StevenBlack · 2024-10-19T23:35:05Z

@ldlac an idea: we already have internal support for some of what I suggest above ⬆️ maybe the --ip none flag would set these things automatically?

--auto, -a            Run without prompting.

--keepdomaincomments, -k
                        Do not keep domain line comments.

--minimise, -m        Minimise the hosts file ignoring non-necessary lines
                        (empty lines and comments).

--noupdate, -n        Don't update from host data sources.

--skipstatichosts, -s
                        Skip static localhost entries in the final hosts file.

--skipstatichosts, -s
                        Skip static localhost entries in the final hosts file.

ldlac · 2024-10-20T01:54:15Z

updateHostsFile.py

@@ -287,19 +302,22 @@ def main():
    merge_file = create_initial_file(
        nounifiedhosts=no_unified_hosts,
    )
-    remove_old_hosts_file(settings["outputpath"], "hosts", settings["backup"])


I made few changes here since I introduced options["outputfilename"]

ldlac · 2024-10-20T01:55:55Z

updateHostsFile.py

@@ -943,14 +966,6 @@ def remove_dups_and_excl(merge_file, exclusion_regexes, output_file=None):
                if line and not line.startswith("#"):
                    settings["exclusions"].append(line)

-    if not os.path.exists(settings["outputpath"]):


I changed output_file to required, and I made a few changes here, the file is already ensure created by remove_old_hosts_file

ldlac · 2024-10-20T01:57:01Z

updateHostsFile.py

@@ -906,16 +928,18 @@ def minimise_file(input_file, target_ip, output_file):
    for line in input_file.readlines():
        line = line.decode("UTF-8")

-        if line.startswith(target_ip):
-            lines.append(line[: line.find("#")].strip() + "\n")
+        if target_ip is None or line.startswith(target_ip):


I don't think it was desired, there was some empty \n with the minimised flag, this will fix those

ldlac · 2024-10-20T02:01:17Z

Thanks for the feedback, I added 3 comments to support my changes to support the domains.txt format

Let me know if you think we should modify the header, I think it's fine even it if it mentions This hosts file is a merged collection ...

There's a few things that still bother me, the blacklist, the custom host file, those could still be pass as arg with the --ip none. If they contain the default content - only comments - these lines will be stripped

updateHostsFile.py

+    final_file = open(
+        path_join_robust(settings["outputpath"], options["outputfilename"]), "w+b"
+    )


support alternative raw

4a06f1d

ldlac commented Oct 16, 2024

View reviewed changes

update test

f452749

.

5da4bea

revert formatting

cbe9f36

ldlac added 2 commits October 19, 2024 18:50

.

a552043

format

41f6207

StevenBlack added discussion need broader input labels Oct 19, 2024

generate domains.txt

560a8a6

ldlac commented Oct 20, 2024

View reviewed changes

github-advanced-security bot found potential problems Oct 20, 2024

View reviewed changes

updateHostsFile.py

Comment on lines +309 to +311

final_file = open(

path_join_robust(settings["outputpath"], options["outputfilename"]), "w+b"

)

Check warning

Code scanning / CodeQL

File is not always closed Warning

File may not be closed if an exception is raised.

close temp_file on default case

a577961

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support alternative raw without 0.0.0.0 #2756

support alternative raw without 0.0.0.0 #2756

ldlac commented Oct 16, 2024

welcome bot commented Oct 16, 2024

ldlac Oct 16, 2024

StevenBlack commented Oct 17, 2024

ldlac commented Oct 18, 2024

ldlac commented Oct 18, 2024

StevenBlack commented Oct 18, 2024

ldlac commented Oct 19, 2024

StevenBlack commented Oct 19, 2024

StevenBlack commented Oct 19, 2024

ldlac Oct 20, 2024

ldlac Oct 20, 2024

ldlac Oct 20, 2024

ldlac commented Oct 20, 2024

support alternative raw without 0.0.0.0 #2756

Are you sure you want to change the base?

support alternative raw without 0.0.0.0 #2756

Conversation

ldlac commented Oct 16, 2024

welcome bot commented Oct 16, 2024

ldlac Oct 16, 2024

Choose a reason for hiding this comment

StevenBlack commented Oct 17, 2024

ldlac commented Oct 18, 2024

ldlac commented Oct 18, 2024

StevenBlack commented Oct 18, 2024

ldlac commented Oct 19, 2024

StevenBlack commented Oct 19, 2024

StevenBlack commented Oct 19, 2024

ldlac Oct 20, 2024

Choose a reason for hiding this comment

ldlac Oct 20, 2024

Choose a reason for hiding this comment

ldlac Oct 20, 2024

Choose a reason for hiding this comment

ldlac commented Oct 20, 2024