2ef33e936e contrib: update testnet torv3 hardcoded seeds (Jon Atack)
Pull request description:
As a follow-up to https://github.com/bitcoin/bitcoin/issues/13550 and #22060, replace the mostly unreachable testnet torv3 hardcoded seeds from v0.22 with new ones that are consistently reachable recently and that have service bit 1 set.
This needs to be done before v24.0 to make sure onion-only testnet nodes can still connect to the network.
Ways to test:
- Re-generate `src/chainparamsseeds.h` with `cd contrib/seeds && python3 generate-seeds.py . > ../../src/chainparamsseeds.h`, check if git tree stays the same
- Re-compile and create a new testnet node with `bitcoind -testnet -dnsseed=0 -onlynet=onion -proxy=127.0.0.1:9050` (or delete `~/.bitcoin/testnet3/peers.dat` and launch bitcoind with `-testnet -dnsseed=0`). Make sure there are no `addnode=` in your `bitcoin.conf`. The debug log should print "Adding fixed seeds". Check if the node is able to connect to the network and get blocks with for ex. `watch -t ./src/bitcoin-cli -testnet -rpcwait -netinfo 4`
- Check the addrman contains the seeds by running for ex. `bitcoin-cli -rpcwait -testnet getnodeaddresses 0 onion | jq -r '.[] | (.address + ":" + (.port|tostring) + " " + (.services|tostring))' | sort`
- Check if the addresses are connectable, for ex. with this python script by laanwj:
```python3
#!/usr/bin/env python3
import pprint
import subprocess
with open('contrib/seeds/nodes_test.txt') as f:
for line in (line for line in (line.rstrip().split('#', 1)[0] for line in f) if line):
pprint.pprint(line)
subprocess.call(["nc", "-v", "-x", "127.0.0.1:9050", "-z"] + line.split(':'))
```
Thanks to satsie (Stacie Waleyko) for help with the list.
ACKs for top commit:
satsie:
ACK 2ef33e936e
laanwj:
ACK 2ef33e936e
Tree-SHA512: 72d27ecba243089bd49c11e921855fba626a1e09ae9b17508254a3bbec4bec341ed6c3d5a4eabc2d37f20bafb8a47ecc7d125e0dda956512a9525ad83273ffd6
Add an argument `-a` to provide a asmap file to do the IP to ASN
lookups.
This speeds up the script greatly, and makes the output deterministic.
Also removes the dependency on `dns.lookup`.
I've annotated the output with ASxxxx comments to provide a way to
verify the functionality.
For now I've added instructions in README.md to download and use the
`demo.map` from the asmap repository. When we have some other mechanism
for distributing asmap files we could switch to that.
This continues #24824. I've removed all the fallbacks and extra
complexity, as everyone will be using the same instructions anyway.
Co-authored-by: Pieter Wuille <pieter.wuille@gmail.com>
Co-authored-by: James O'Beirne <james.obeirne@pm.me>
Co-authored-by: russeree <reese.russell@ymail.com>
I have some qualms with maintaining a suspicious hosts list as part as
the repository. But also, it's stale and irrelevant. I've checked the
entire list and none of them is connectable. Only one still appars in
`nodes_main.txt` but with low uptime and an old subversion string so it
wouldn't be picked in the first place.
Documentation:
- Use https URL for bitcoin.sipa.be (http sends a redirect, fooling
curl).
- Add explicit step to add manual seeds.
Code:
- Change PATTERN_ONION to v3 (effectively means that the no onion hosts
are delivered).
- Add versions to PATTERN_AGENT filter.
- Print specific message on resolve exception.
An IPv6 address from fc00::/8 could be either from the CJDNS network or
from a private-unroutable-reserved segment of IPv6. A seed node with
such an address must be from the CJDNS network, otherwise other peers
will not be able to connect to it.
961f148cb1 doc: update contrib/seeds/README dnspython installation info (Jon Atack)
dd7b5f46d8 script: fix deprecation warning in makeseeds.py (Jon Atack)
Pull request description:
Seen while reviewing #20237.
1. Fix a deprecation warning in `contrib/seeds/makeseeds.py`
```
makeseeds.py:139: DeprecationWarning: please use dns.resolver.resolve() instead
asn = int([x.to_text() for x in dns.resolver.query('.'.join(
```
- Per https://dnspython.readthedocs.io/en/latest/whatsnew.html, `dns.resolver.query()` was deprecated in `dnspython` version 2.0.0.
- See https://dnspython.readthedocs.io/en/latest/resolver-class.html for more info on the resolver class.
2. Update the `dnspython` dependency installation instructions in `contrib/seeds/README`
- The markdown rendering can be seen here: https://github.com/jonatack/bitcoin/tree/contrib-seeds-fixups/contrib/seeds
ACKs for top commit:
laanwj:
code review ACK 961f148cb1
Tree-SHA512: f9c4f318a1a0d35b8de147d24b72c534a1f58eece31e7cfa00b4149a63b6a618d8ca0312f52fd8056f3c645cf2ee68574ca02319fddffdad919a70cd33395d33
makeseeds.py:139: DeprecationWarning: please use dns.resolver.resolve() instead
asn = int([x.to_text() for x in dns.resolver.query('.'.join(
per https://dnspython.readthedocs.io/en/latest/whatsnew.html
dns.resolver.query() was deprecated in dnspython version 2.0.0
Stats:
```
IPv4 IPv6 Onion Pass
426728 59523 7900 Initial
426728 59523 7900 Skip entries with invalid address
426728 59523 7900 After removing duplicates
426727 59523 7900 Skip entries from suspicious hosts
123226 51785 7787 Enforce minimal number of blocks
121710 51322 7586 Require service bit 1
4706 1427 3749 Require minimum uptime
4124 1098 3681 Require a known and recent user agent
4033 1075 3681 Filter out hosts with multiple bitcoin ports
512 140 512 Look up ASNs and limit results per ASN and per net
```
Update hardcoded seeds from seeds_emzy.txt seeds_lukejr.txt
seeds_sipa.txt seeds_sjors.txt, according to release process.
Output from makeseeds.py:
```
IPv4 IPv6 Onion Pass
1364173 244127 2454 Initial
1364173 244127 2454 Skip entries with invalid address
1129552 213117 2345 After removing duplicates
1129548 213117 2345 Skip entries from suspicious hosts
338216 191944 2249 Enforce minimal number of blocks
336851 188993 2189 Require service bit 1
6998 1520 150 Require minimum uptime
5682 1290 89 Require a known and recent user agent
5622 1279 89 Filter out hosts with multiple bitcoin ports
512 146 89 Look up ASNs and limit results per ASN and per net
```
e1c582cbaa contrib: makeseeds: Read suspicious hosts from a file instead of hardcoding (Sanjay K)
Pull request description:
referring to: https://github.com/bitcoin/bitcoin/issues/17020
good first issue: reading SUSPICIOUS_HOSTS from a file.
I haven't changed the base hosts that were included in the original source, just made it readable from a file.
ACKs for top commit:
practicalswift:
ACK e1c582cbaa -- diff looks correct
Tree-SHA512: 18684abc1c02cf52d63f6f6ecd98df01a9574a7c470524c37e152296504e2e3ffbabd6f3208214b62031512aeb809a6d37446af82c9f480ff14ce4c42c98e7c2
- Change regular expression to cover recent versions, as well as
subversions with custom uacomment, and improve readability.
- Vary uptime requirements per network (onions are allowed to have less
uptime, to make sure we get enough of them)
- Add deduplication step (to allow simple concatentation of multiple seeds files).
- Log of number of nodes (per network) after every step.
Flagged by flake8 v3.6.0, as W605, plus a few others identified
incidentally, e.g. 59ffecf66cf4d08c4b431e457b083878d66a3fd6.
Note that r"\n" matches to "\n" under re.match/search.
44407100f broke inserting entries into addrman from static seeds
(as well as dnsseeds which did not support service bits). Static
seeds were already being filtered by UA for 0.13.1+ (ie
NODE_WITNESS), so simply changing the default service bits to
include NODE_WITNESS (and updating docs appropriately) is
sufficient.
For DNS Seeds, we will later fix by falling back to oneshot if a
seed does not support filtering.
Allow for non-8333 nodes to appear in the internal seeds. This will
allow bitcoind to bypas a filter on 8333. This also makes it possible to
use the same tool for e.g. testnet.
As hosts with multiple nodes per IP are likely abusive, add a filter to
remove these (the ASN check will take care of them for IPv4, but not
IPv6 or onion).
- Moved all seed related scripts to contrib/seeds for consistency
- Updated `makeseeds.py` to handle IPv6 and onions, fix regular
expression for recent Bitcoin Core versions
- Fixed a bug in `generate-seeds.py` with regard to IPv6 parsing
MacroFake
Jon Atack
Marnix
laanwj
Baas
Vasil Dimov
W. J. van der Laan
RandyMcMillan
Sanjay K
Brian Solon
Ben Woosley
Emil
Wladimir J. van der Laan
DrahtBot
practicalswift
John Bampton
Matt Corallo
isle2983
Matthew King
paveljanik
Pieter Wuille
super3
Gavin Andresen