generate-domains-blocklist.py could be improved to simplify patterns

[1ennard]

The generate-domains-blocklist.py script is affected by this bug. It not idealy removes overlapping patterns when generating domain blocklists, regarding to the filtering rules.

When does this occur?

The issue occurs when executing the script to generate a blocklist from a domains-blocklist-local-additions.txt file.

Where does it happen?

Problably in the blocklists_from_config_file function of the generate-domains-blocklist.py script.

How do we replicate the issue?

1. Prepare a domains-blocklist-local-additions.txt file with the following content:

aaa.*
*.aaa
*aaa*
qaaaq.com
aaa.com
aaa.com.com
com.aaa
com.com.aaa

bbb.*
*.bbb
bbb.com
bbb.com.com
com.bbb
com.com.bbb

*ccc*
ccc.com
qcccq.com

2. Delete all urls from the domains-blocklist.conf
3. Execute the script using:

python3 generate-domains-blocklist.py -o blocklist.txt

4. The resulting blocklist.txt will contain:

########## Blocklist from file:domains-blocklist-local-additions.txt ##########

# Ignored duplicates: 2
# Ignored due to overlapping local patterns: 4

aaa.*
bbb.*
*aaa*
*ccc*
*.aaa
com.aaa
*.bbb
com.bbb
ccc.com
qaaaq.com
qcccq.com

Expected behavior (i.e. solution)

The Blocklist should only contain these entries, as they cover all cases:

*aaa*
*.bbb
bbb.*
*ccc*

Other Comments

Wiki page to Combining Blocklists

[jedisct1]

The tool was only written to remove overlapping names, and, to some extent, names that are matches by existing patterns.

Proving pattern equivalence or that a pattern is a superset of another is not implemented.

Since you appear to be a Python developer, is it a feature you could contribute?

[sergeevabc]

@1ennard, have you managed to enhance generate-domains-blocklist.py by yourself?
Please, share.