Bots... Bots as Far as the Eye Can See...
My fight with bots seems eternal... Likely because it will be.
Counter Measures
I recently have started improving my robots.txt, and made a fail2ban jail for sketchy scrapers/bots. (If I get to it, fail2ban will be a whole other post.) I've also started adding some of the more annoying ones to my iptables.
Sand...
After adding a particularly annoying scraper called SeznamBot to my robots.txt, it was still pinging my robots.txt file far too often for my taste.
So I added its thankfully listed ipv4 blocks to my iptables.
After doing that it made me more annoyed... It started scraping my site on IPv6, and was not honoring my robots.txt any more...
Running Into a Wall
When I decided to start blocking their IPv6 ranges, I ran into a wall. In hindsight, it was probably a self imposed wall. I'm going to blame this mental block on my Cisco IOS class from over the summer.
Anyway, "How do I block an IPv6 range with ip6tables?"... Well, this question really turned up Nothing; nothing at all despite my best google-fu... Which is why I'm doing this write up.
Simpler Than You Think
After a day of searching I stumbled upon a IPv6 /CIDR table (Added at end of post), and then it clicked in my head. You block ranges the same as IPv4, you just have to deal with a lot more classless ranges. A lot more.
It is at this point I'm going to rant about Cisco's IOS being stupid, and causing this brain fart for me... For some incomprehensible reason to me, Cisco commands don't let you use /CIDR when dealing with IPv6. <scowls at cisco>
Solution
And here are the three rules I wanted.
ip6tables -A INPUT -j DROP -s 2a02:598:a::78:00/112 -m comment --comment "SeznamBot"
ip6tables -A INPUT -j DROP -s 2a02:598:a::79:00/112 -m comment --comment "SeznamBot"
ip6tables -A INPUT -j DROP -s 2a02:598:2::00/112 -m comment --comment "SeznamBot"
So I now have one bot blocked, ~∞ left to go!
Bonus: You Can Comment Rules
As you might have just noticed in the above rules, you can add comments directly to your rules with a -m comment --comment "Blah".
These comments will then show when you print your tables.
target prot opt source destination
DROP all 2a02:598:a::78:0/112 anywhere /* SeznamBot */
DROP all 2a02:598:a::79:0/112 anywhere /* SeznamBot */
DROP all 2a02:598:2::/112 anywhere /* SeznamBot */
A Fancy Table: Brought to You by Markdown
Table's Source MediaWiki - Range blocks/IPv6
| CIDR | Start of range | End of range | Total addresses |
|---|---|---|---|
| ::/0 | :: | ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 340,282,366,920,938,463,463,374,607,431,768,211,456 |
| ::/1 | :: | 7fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 170,141,183,460,469,231,731,687,303,715,884,105,728 |
| ::/2 | :: | 3fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 85,070,591,730,234,615,865,843,651,857,942,052,864 |
| 2000::/3 | 2000:: | 3fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 42,535,295,865,117,307,932,921,825,928,921,026,432 |
| 2000::/4 | 2000:: | 2fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 21,267,647,932,558,653,966,460,912,964,485,513,216 |
| 2000::/5 | 2000:: | 27ff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 10,633,823,966,279,326,983,230,456,482,242,756,608 |
| 2000::/6 | 2000:: | 23ff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 5,316,911,983,139,663,491,615,228,241,121,378,304 |
| 2000::/7 | 2000:: | 21ff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 2,658,455,991,569,831,745,807,614,120,560,689,152 |
| 2000::/8 | 2000:: | 20ff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 1,329,227,995,784,915,872,903,807,060,280,344,576 |
| 2000::/9 | 2000:: | 207f:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 664,613,997,892,457,936,451,903,530,140,172,288 |
| 2000::/10 | 2000:: | 203f:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 332,306,998,946,228,968,225,951,765,070,086,144 |
| 2000::/11 | 2000:: | 201f:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 166,153,499,473,114,484,112,975,882,535,043,072 |
| 2000::/12 | 2000:: | 200f:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 83,076,749,736,557,242,056,487,941,267,521,536 |
| 2000::/13 | 2000:: | 2007:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 41,538,374,868,278,621,028,243,970,633,760,768 |
| 2000::/14 | 2000:: | 2003:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 20,769,187,434,139,310,514,121,985,316,880,384 |
| 2000::/15 | 2000:: | 2001:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 10,384,593,717,069,655,257,060,992,658,440,192 |
| 2001::/16 | 2001:: | 2001:ffff:ffff:ffff:ffff:ffff:ffff:ffff | 5,192,296,858,534,827,628,530,496,329,220,096 |
| 2001::/17 | 2001:: | 2001:7fff:ffff:ffff:ffff:ffff:ffff:ffff | 2,596,148,429,267,413,814,265,248,164,610,048 |
| 2001::/18 | 2001:: | 2001:3fff:ffff:ffff:ffff:ffff:ffff:ffff | 1,298,074,214,633,706,907,132,624,082,305,024 |
| 2001::/19 | 2001:: | 2001:1fff:ffff:ffff:ffff:ffff:ffff:ffff | 649,037,107,316,853,453,566,312,041,152,512 |
| 2001::/20 | 2001:: | 2001:0fff:ffff:ffff:ffff:ffff:ffff:ffff | 324,518,553,658,426,726,783,156,020,576,256 |
| 2001::/21 | 2001:: | 2001:07ff:ffff:ffff:ffff:ffff:ffff:ffff | 162,259,276,829,213,363,391,578,010,288,128 |
| 2001::/22 | 2001:: | 2001:03ff:ffff:ffff:ffff:ffff:ffff:ffff | 81,129,638,414,606,681,695,789,005,144,064 |
| 2001::/23 | 2001:: | 2001:01ff:ffff:ffff:ffff:ffff:ffff:ffff | 40,564,819,207,303,340,847,894,502,572,032 |
| 2001::/24 | 2001:: | 2001:00ff:ffff:ffff:ffff:ffff:ffff:ffff | 20,282,409,603,651,670,423,947,251,286,016 |
| 2001::/25 | 2001:: | 2001:007f:ffff:ffff:ffff:ffff:ffff:ffff | 10,141,204,801,825,835,211,973,625,643,008 |
| 2001::/26 | 2001:: | 2001:003f:ffff:ffff:ffff:ffff:ffff:ffff | 5,070,602,400,912,917,605,986,812,821,504 |
| 2001::/27 | 2001:: | 2001:001f:ffff:ffff:ffff:ffff:ffff:ffff | 2,535,301,200,456,458,802,993,406,410,752 |
| 2001::/28 | 2001:: | 2001:000f:ffff:ffff:ffff:ffff:ffff:ffff | 1,267,650,600,228,229,401,496,703,205,376 |
| 2001::/29 | 2001:: | 2001:0007:ffff:ffff:ffff:ffff:ffff:ffff | 633,825,300,114,114,700,748,351,602,688 |
| 2001::/30 | 2001:: | 2001:0003:ffff:ffff:ffff:ffff:ffff:ffff | 316,912,650,057,057,350,374,175,801,344 |
| 2001::/31 | 2001:: | 2001:0001:ffff:ffff:ffff:ffff:ffff:ffff | 158,456,325,028,528,675,187,087,900,672 |
| 2001:db8::/32 | 2001:db8:: | 2001:db8:ffff:ffff:ffff:ffff:ffff:ffff | 79,228,162,514,264,337,593,543,950,336 |
| 2001:db8::/33 | 2001:db8:: | 2001:db8:7fff:ffff:ffff:ffff:ffff:ffff | 39,614,081,257,132,168,796,771,975,168 |
| 2001:db8::/34 | 2001:db8:: | 2001:db8:3fff:ffff:ffff:ffff:ffff:ffff | 19,807,040,628,566,084,398,385,987,584 |
| 2001:db8::/35 | 2001:db8:: | 2001:db8:1fff:ffff:ffff:ffff:ffff:ffff | 9,903,520,314,283,042,199,192,993,792 |
| 2001:db8::/36 | 2001:db8:: | 2001:db8:0fff:ffff:ffff:ffff:ffff:ffff | 4,951,760,157,141,521,099,596,496,896 |
| 2001:db8::/37 | 2001:db8:: | 2001:db8:07ff:ffff:ffff:ffff:ffff:ffff | 2,475,880,078,570,760,549,798,248,448 |
| 2001:db8::/38 | 2001:db8:: | 2001:db8:03ff:ffff:ffff:ffff:ffff:ffff | 1,237,940,039,285,380,274,899,124,224 |
| 2001:db8::/39 | 2001:db8:: | 2001:db8:01ff:ffff:ffff:ffff:ffff:ffff | 618,970,019,642,690,137,449,562,112 |
| 2001:db8::/40 | 2001:db8:: | 2001:db8:00ff:ffff:ffff:ffff:ffff:ffff | 309,485,009,821,345,068,724,781,056 |
| 2001:db8::/41 | 2001:db8:: | 2001:db8:007f:ffff:ffff:ffff:ffff:ffff | 154,742,504,910,672,534,362,390,528 |
| 2001:db8::/42 | 2001:db8:: | 2001:db8:003f:ffff:ffff:ffff:ffff:ffff | 77,371,252,455,336,267,181,195,264 |
| 2001:db8::/43 | 2001:db8:: | 2001:db8:001f:ffff:ffff:ffff:ffff:ffff | 38,685,626,227,668,133,590,597,632 |
| 2001:db8::/44 | 2001:db8:: | 2001:db8:000f:ffff:ffff:ffff:ffff:ffff | 19,342,813,113,834,066,795,298,816 |
| 2001:db8::/45 | 2001:db8:: | 2001:db8:0007:ffff:ffff:ffff:ffff:ffff | 9,671,406,556,917,033,397,649,408 |
| 2001:db8::/46 | 2001:db8:: | 2001:db8:0003:ffff:ffff:ffff:ffff:ffff | 4,835,703,278,458,516,698,824,704 |
| 2001:db8::/47 | 2001:db8:: | 2001:db8:0001:ffff:ffff:ffff:ffff:ffff | 2,417,851,639,229,258,349,412,352 |
| 2001:db8::/48 | 2001:db8:: | 2001:db8:0000:ffff:ffff:ffff:ffff:ffff | 1,208,925,819,614,629,174,706,176 |
| 2001:db8::/49 | 2001:db8:: | 2001:db8:0000:7fff:ffff:ffff:ffff:ffff | 604,462,909,807,314,587,353,088 |
| 2001:db8::/50 | 2001:db8:: | 2001:db8:0000:3fff:ffff:ffff:ffff:ffff | 302,231,454,903,657,293,676,544 |
| 2001:db8::/51 | 2001:db8:: | 2001:db8:0000:1fff:ffff:ffff:ffff:ffff | 151,115,727,451,828,646,838,272 |
| 2001:db8::/52 | 2001:db8:: | 2001:db8:0000:0fff:ffff:ffff:ffff:ffff | 75,557,863,725,914,323,419,136 |
| 2001:db8::/53 | 2001:db8:: | 2001:db8:0000:07ff:ffff:ffff:ffff:ffff | 37,778,931,862,957,161,709,568 |
| 2001:db8::/54 | 2001:db8:: | 2001:db8:0000:03ff:ffff:ffff:ffff:ffff | 18,889,465,931,478,580,854,784 |
| 2001:db8::/55 | 2001:db8:: | 2001:db8:0000:01ff:ffff:ffff:ffff:ffff | 9,444,732,965,739,290,427,392 |
| 2001:db8::/56 | 2001:db8:: | 2001:db8:0000:00ff:ffff:ffff:ffff:ffff | 4,722,366,482,869,645,213,696 |
| 2001:db8::/57 | 2001:db8:: | 2001:db8:0000:007f:ffff:ffff:ffff:ffff | 2,361,183,241,434,822,606,848 |
| 2001:db8::/58 | 2001:db8:: | 2001:db8:0000:003f:ffff:ffff:ffff:ffff | 1,180,591,620,717,411,303,424 |
| 2001:db8::/59 | 2001:db8:: | 2001:db8:0000:001f:ffff:ffff:ffff:ffff | 590,295,810,358,705,651,712 |
| 2001:db8::/60 | 2001:db8:: | 2001:db8:0000:000f:ffff:ffff:ffff:ffff | 295,147,905,179,352,825,856 |
| 2001:db8::/61 | 2001:db8:: | 2001:db8:0000:0007:ffff:ffff:ffff:ffff | 147,573,952,589,676,412,928 |
| 2001:db8::/62 | 2001:db8:: | 2001:db8:0000:0003:ffff:ffff:ffff:ffff | 73,786,976,294,838,206,464 |
| 2001:db8::/63 | 2001:db8:: | 2001:db8:0000:0001:ffff:ffff:ffff:ffff | 36,893,488,147,419,103,232 |
| 2001:db8::/64 | 2001:db8:: | 2001:db8:0000:0000:ffff:ffff:ffff:ffff | 18,446,744,073,709,551,616 |
| 2001:db8::/65 | 2001:db8:: | 2001:db8:0000:0000:7fff:ffff:ffff:ffff | 9,223,372,036,854,775,808 |
| 2001:db8::/66 | 2001:db8:: | 2001:db8:0000:0000:3fff:ffff:ffff:ffff | 4,611,686,018,427,387,904 |
| 2001:db8::/67 | 2001:db8:: | 2001:db8:0000:0000:1fff:ffff:ffff:ffff | 2,305,843,009,213,693,952 |
| 2001:db8::/68 | 2001:db8:: | 2001:db8:0000:0000:0fff:ffff:ffff:ffff | 1,152,921,504,606,846,976 |
| 2001:db8::/69 | 2001:db8:: | 2001:db8:0000:0000:07ff:ffff:ffff:ffff | 576,460,752,303,423,488 |
| 2001:db8::/70 | 2001:db8:: | 2001:db8:0000:0000:03ff:ffff:ffff:ffff | 288,230,376,151,711,744 |
| 2001:db8::/71 | 2001:db8:: | 2001:db8:0000:0000:01ff:ffff:ffff:ffff | 144,115,188,075,855,872 |
| 2001:db8::/72 | 2001:db8:: | 2001:db8:0000:0000:00ff:ffff:ffff:ffff | 72,057,594,037,927,936 |
| 2001:db8::/73 | 2001:db8:: | 2001:db8:0000:0000:007f:ffff:ffff:ffff | 36,028,797,018,963,968 |
| 2001:db8::/74 | 2001:db8:: | 2001:db8:0000:0000:003f:ffff:ffff:ffff | 18,014,398,509,481,984 |
| 2001:db8::/75 | 2001:db8:: | 2001:db8:0000:0000:001f:ffff:ffff:ffff | 9,007,199,254,740,992 |
| 2001:db8::/76 | 2001:db8:: | 2001:db8:0000:0000:000f:ffff:ffff:ffff | 4,503,599,627,370,496 |
| 2001:db8::/77 | 2001:db8:: | 2001:db8:0000:0000:0007:ffff:ffff:ffff | 2,251,799,813,685,248 |
| 2001:db8::/78 | 2001:db8:: | 2001:db8:0000:0000:0003:ffff:ffff:ffff | 1,125,899,906,842,624 |
| 2001:db8::/79 | 2001:db8:: | 2001:db8:0000:0000:0001:ffff:ffff:ffff | 562,949,953,421,312 |
| 2001:db8::/80 | 2001:db8:: | 2001:db8:0000:0000:0000:ffff:ffff:ffff | 281,474,976,710,656 |
| 2001:db8::/81 | 2001:db8:: | 2001:db8:0000:0000:0000:7fff:ffff:ffff | 140,737,488,355,328 |
| 2001:db8::/82 | 2001:db8:: | 2001:db8:0000:0000:0000:3fff:ffff:ffff | 70,368,744,177,664 |
| 2001:db8::/83 | 2001:db8:: | 2001:db8:0000:0000:0000:1fff:ffff:ffff | 35,184,372,088,832 |
| 2001:db8::/84 | 2001:db8:: | 2001:db8:0000:0000:0000:0fff:ffff:ffff | 17,592,186,044,416 |
| 2001:db8::/85 | 2001:db8:: | 2001:db8:0000:0000:0000:07ff:ffff:ffff | 8,796,093,022,208 |
| 2001:db8::/86 | 2001:db8:: | 2001:db8:0000:0000:0000:03ff:ffff:ffff | 4,398,046,511,104 |
| 2001:db8::/87 | 2001:db8:: | 2001:db8:0000:0000:0000:01ff:ffff:ffff | 2,199,023,255,552 |
| 2001:db8::/88 | 2001:db8:: | 2001:db8:0000:0000:0000:00ff:ffff:ffff | 1,099,511,627,776 |
| 2001:db8::/89 | 2001:db8:: | 2001:db8:0000:0000:0000:007f:ffff:ffff | 549,755,813,888 |
| 2001:db8::/90 | 2001:db8:: | 2001:db8:0000:0000:0000:003f:ffff:ffff | 274,877,906,944 |
| 2001:db8::/91 | 2001:db8:: | 2001:db8:0000:0000:0000:001f:ffff:ffff | 137,438,953,472 |
| 2001:db8::/92 | 2001:db8:: | 2001:db8:0000:0000:0000:000f:ffff:ffff | 68,719,476,736 |
| 2001:db8::/93 | 2001:db8:: | 2001:db8:0000:0000:0000:0007:ffff:ffff | 34,359,738,368 |
| 2001:db8::/94 | 2001:db8:: | 2001:db8:0000:0000:0000:0003:ffff:ffff | 17,179,869,184 |
| 2001:db8::/95 | 2001:db8:: | 2001:db8:0000:0000:0000:0001:ffff:ffff | 8,589,934,592 |
| 2001:db8::/96 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:ffff:ffff | 4,294,967,296 |
| 2001:db8::/97 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:7fff:ffff | 2,147,483,648 |
| 2001:db8::/98 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:3fff:ffff | 1,073,741,824 |
| 2001:db8::/99 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:1fff:ffff | 536,870,912 |
| 2001:db8::/100 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0fff:ffff | 268,435,456 |
| 2001:db8::/101 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:07ff:ffff | 134,217,728 |
| 2001:db8::/102 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:03ff:ffff | 67,108,864 |
| 2001:db8::/103 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:01ff:ffff | 33,554,432 |
| 2001:db8::/104 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:00ff:ffff | 16,777,216 |
| 2001:db8::/105 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:007f:ffff | 8,388,608 |
| 2001:db8::/106 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:003f:ffff | 4,194,304 |
| 2001:db8::/107 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:001f:ffff | 2,097,152 |
| 2001:db8::/108 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:000f:ffff | 1,048,576 |
| 2001:db8::/109 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0007:ffff | 524,288 |
| 2001:db8::/110 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0003:ffff | 262,144 |
| 2001:db8::/111 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0001:ffff | 131,072 |
| 2001:db8::/112 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:ffff | 65,536 |
| 2001:db8::/113 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:7fff | 32,768 |
| 2001:db8::/114 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:3fff | 16,384 |
| 2001:db8::/115 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:1fff | 8,192 |
| 2001:db8::/116 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:0fff | 4,096 |
| 2001:db8::/117 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:07ff | 2,048 |
| 2001:db8::/118 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:03ff | 1,024 |
| 2001:db8::/119 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:01ff | 512 |
| 2001:db8::/120 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:00ff | 256 |
| 2001:db8::/121 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:007f | 128 |
| 2001:db8::/122 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:003f | 64 |
| 2001:db8::/123 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:001f | 32 |
| 2001:db8::/124 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:000f | 16 |
| 2001:db8::/125 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:0007 | 8 |
| 2001:db8::/126 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:0003 | 4 |
| 2001:db8::/127 | 2001:db8:: | 2001:db8:0000:0000:0000:0000:0000:0001 | 2 |
| 2001:db8::/128 | 2001:db8:: | 2001:db8:: | 1 |
Links
robots.txt
sketchy scrapers/bots
MediaWiki - Range blocks/IPv6