Bots... Bots as Far as the Eye Can See...

My fight with bots seems eternal... Likely because it will be.

Counter Measures

I recently have started improving my robots.txt, and made a fail2ban jail for sketchy scrapers/bots. (If I get to it, fail2ban will be a whole other post.) I've also started adding some of the more annoying ones to my iptables.

Sand...

After adding a particularly annoying scraper called SeznamBot to my robots.txt, it was still pinging my robots.txt file far too often for my taste. So I added its thankfully listed ipv4 blocks to my iptables. After doing that it made me more annoyed... It started scraping my site on IPv6, and was not honoring my robots.txt any more...

Running Into a Wall

When I decided to start blocking their IPv6 ranges, I ran into a wall. In hindsight, it was probably a self imposed wall. I'm going to blame this mental block on my Cisco IOS class from over the summer.

Anyway, "How do I block an IPv6 range with ip6tables?"... Well, this question really turned up Nothing; nothing at all despite my best google-fu... Which is why I'm doing this write up.

Simpler Than You Think

After a day of searching I stumbled upon a IPv6 /CIDR table (Added at end of post), and then it clicked in my head. You block ranges the same as IPv4, you just have to deal with a lot more classless ranges. A lot more.

It is at this point I'm going to rant about Cisco's IOS being stupid, and causing this brain fart for me... For some incomprehensible reason to me, Cisco commands don't let you use /CIDR when dealing with IPv6. <scowls at cisco>

Solution

And here are the three rules I wanted.

ip6tables -A INPUT -j DROP -s 2a02:598:a::78:00/112 -m comment --comment "SeznamBot"
ip6tables -A INPUT -j DROP -s 2a02:598:a::79:00/112 -m comment --comment "SeznamBot"
ip6tables -A INPUT -j DROP -s 2a02:598:2::00/112 -m comment --comment "SeznamBot"

So I now have one bot blocked, ~∞ left to go!

Bonus: You Can Comment Rules

As you might have just noticed in the above rules, you can add comments directly to your rules with a -m comment --comment "Blah". These comments will then show when you print your tables.

target     prot opt source               destination
DROP       all      2a02:598:a::78:0/112  anywhere             /* SeznamBot */
DROP       all      2a02:598:a::79:0/112  anywhere             /* SeznamBot */
DROP       all      2a02:598:2::/112      anywhere             /* SeznamBot */

A Fancy Table: Brought to You by Markdown

Table's Source MediaWiki - Range blocks/IPv6

CIDR Start of range End of range Total addresses
::/0 :: ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 340,282,366,920,938,463,463,374,607,431,768,211,456
::/1 :: 7fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 170,141,183,460,469,231,731,687,303,715,884,105,728
::/2 :: 3fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 85,070,591,730,234,615,865,843,651,857,942,052,864
2000::/3 2000:: 3fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 42,535,295,865,117,307,932,921,825,928,921,026,432
2000::/4 2000:: 2fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 21,267,647,932,558,653,966,460,912,964,485,513,216
2000::/5 2000:: 27ff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 10,633,823,966,279,326,983,230,456,482,242,756,608
2000::/6 2000:: 23ff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 5,316,911,983,139,663,491,615,228,241,121,378,304
2000::/7 2000:: 21ff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 2,658,455,991,569,831,745,807,614,120,560,689,152
2000::/8 2000:: 20ff:ffff:ffff:ffff:ffff:ffff:ffff:ffff 1,329,227,995,784,915,872,903,807,060,280,344,576
2000::/9 2000:: 207f:ffff:ffff:ffff:ffff:ffff:ffff:ffff 664,613,997,892,457,936,451,903,530,140,172,288
2000::/10 2000:: 203f:ffff:ffff:ffff:ffff:ffff:ffff:ffff 332,306,998,946,228,968,225,951,765,070,086,144
2000::/11 2000:: 201f:ffff:ffff:ffff:ffff:ffff:ffff:ffff 166,153,499,473,114,484,112,975,882,535,043,072
2000::/12 2000:: 200f:ffff:ffff:ffff:ffff:ffff:ffff:ffff 83,076,749,736,557,242,056,487,941,267,521,536
2000::/13 2000:: 2007:ffff:ffff:ffff:ffff:ffff:ffff:ffff 41,538,374,868,278,621,028,243,970,633,760,768
2000::/14 2000:: 2003:ffff:ffff:ffff:ffff:ffff:ffff:ffff 20,769,187,434,139,310,514,121,985,316,880,384
2000::/15 2000:: 2001:ffff:ffff:ffff:ffff:ffff:ffff:ffff 10,384,593,717,069,655,257,060,992,658,440,192
2001::/16 2001:: 2001:ffff:ffff:ffff:ffff:ffff:ffff:ffff 5,192,296,858,534,827,628,530,496,329,220,096
2001::/17 2001:: 2001:7fff:ffff:ffff:ffff:ffff:ffff:ffff 2,596,148,429,267,413,814,265,248,164,610,048
2001::/18 2001:: 2001:3fff:ffff:ffff:ffff:ffff:ffff:ffff 1,298,074,214,633,706,907,132,624,082,305,024
2001::/19 2001:: 2001:1fff:ffff:ffff:ffff:ffff:ffff:ffff 649,037,107,316,853,453,566,312,041,152,512
2001::/20 2001:: 2001:0fff:ffff:ffff:ffff:ffff:ffff:ffff 324,518,553,658,426,726,783,156,020,576,256
2001::/21 2001:: 2001:07ff:ffff:ffff:ffff:ffff:ffff:ffff 162,259,276,829,213,363,391,578,010,288,128
2001::/22 2001:: 2001:03ff:ffff:ffff:ffff:ffff:ffff:ffff 81,129,638,414,606,681,695,789,005,144,064
2001::/23 2001:: 2001:01ff:ffff:ffff:ffff:ffff:ffff:ffff 40,564,819,207,303,340,847,894,502,572,032
2001::/24 2001:: 2001:00ff:ffff:ffff:ffff:ffff:ffff:ffff 20,282,409,603,651,670,423,947,251,286,016
2001::/25 2001:: 2001:007f:ffff:ffff:ffff:ffff:ffff:ffff 10,141,204,801,825,835,211,973,625,643,008
2001::/26 2001:: 2001:003f:ffff:ffff:ffff:ffff:ffff:ffff 5,070,602,400,912,917,605,986,812,821,504
2001::/27 2001:: 2001:001f:ffff:ffff:ffff:ffff:ffff:ffff 2,535,301,200,456,458,802,993,406,410,752
2001::/28 2001:: 2001:000f:ffff:ffff:ffff:ffff:ffff:ffff 1,267,650,600,228,229,401,496,703,205,376
2001::/29 2001:: 2001:0007:ffff:ffff:ffff:ffff:ffff:ffff 633,825,300,114,114,700,748,351,602,688
2001::/30 2001:: 2001:0003:ffff:ffff:ffff:ffff:ffff:ffff 316,912,650,057,057,350,374,175,801,344
2001::/31 2001:: 2001:0001:ffff:ffff:ffff:ffff:ffff:ffff 158,456,325,028,528,675,187,087,900,672
2001:db8::/32 2001:db8:: 2001:db8:ffff:ffff:ffff:ffff:ffff:ffff 79,228,162,514,264,337,593,543,950,336
2001:db8::/33 2001:db8:: 2001:db8:7fff:ffff:ffff:ffff:ffff:ffff 39,614,081,257,132,168,796,771,975,168
2001:db8::/34 2001:db8:: 2001:db8:3fff:ffff:ffff:ffff:ffff:ffff 19,807,040,628,566,084,398,385,987,584
2001:db8::/35 2001:db8:: 2001:db8:1fff:ffff:ffff:ffff:ffff:ffff 9,903,520,314,283,042,199,192,993,792
2001:db8::/36 2001:db8:: 2001:db8:0fff:ffff:ffff:ffff:ffff:ffff 4,951,760,157,141,521,099,596,496,896
2001:db8::/37 2001:db8:: 2001:db8:07ff:ffff:ffff:ffff:ffff:ffff 2,475,880,078,570,760,549,798,248,448
2001:db8::/38 2001:db8:: 2001:db8:03ff:ffff:ffff:ffff:ffff:ffff 1,237,940,039,285,380,274,899,124,224
2001:db8::/39 2001:db8:: 2001:db8:01ff:ffff:ffff:ffff:ffff:ffff 618,970,019,642,690,137,449,562,112
2001:db8::/40 2001:db8:: 2001:db8:00ff:ffff:ffff:ffff:ffff:ffff 309,485,009,821,345,068,724,781,056
2001:db8::/41 2001:db8:: 2001:db8:007f:ffff:ffff:ffff:ffff:ffff 154,742,504,910,672,534,362,390,528
2001:db8::/42 2001:db8:: 2001:db8:003f:ffff:ffff:ffff:ffff:ffff 77,371,252,455,336,267,181,195,264
2001:db8::/43 2001:db8:: 2001:db8:001f:ffff:ffff:ffff:ffff:ffff 38,685,626,227,668,133,590,597,632
2001:db8::/44 2001:db8:: 2001:db8:000f:ffff:ffff:ffff:ffff:ffff 19,342,813,113,834,066,795,298,816
2001:db8::/45 2001:db8:: 2001:db8:0007:ffff:ffff:ffff:ffff:ffff 9,671,406,556,917,033,397,649,408
2001:db8::/46 2001:db8:: 2001:db8:0003:ffff:ffff:ffff:ffff:ffff 4,835,703,278,458,516,698,824,704
2001:db8::/47 2001:db8:: 2001:db8:0001:ffff:ffff:ffff:ffff:ffff 2,417,851,639,229,258,349,412,352
2001:db8::/48 2001:db8:: 2001:db8:0000:ffff:ffff:ffff:ffff:ffff 1,208,925,819,614,629,174,706,176
2001:db8::/49 2001:db8:: 2001:db8:0000:7fff:ffff:ffff:ffff:ffff 604,462,909,807,314,587,353,088
2001:db8::/50 2001:db8:: 2001:db8:0000:3fff:ffff:ffff:ffff:ffff 302,231,454,903,657,293,676,544
2001:db8::/51 2001:db8:: 2001:db8:0000:1fff:ffff:ffff:ffff:ffff 151,115,727,451,828,646,838,272
2001:db8::/52 2001:db8:: 2001:db8:0000:0fff:ffff:ffff:ffff:ffff 75,557,863,725,914,323,419,136
2001:db8::/53 2001:db8:: 2001:db8:0000:07ff:ffff:ffff:ffff:ffff 37,778,931,862,957,161,709,568
2001:db8::/54 2001:db8:: 2001:db8:0000:03ff:ffff:ffff:ffff:ffff 18,889,465,931,478,580,854,784
2001:db8::/55 2001:db8:: 2001:db8:0000:01ff:ffff:ffff:ffff:ffff 9,444,732,965,739,290,427,392
2001:db8::/56 2001:db8:: 2001:db8:0000:00ff:ffff:ffff:ffff:ffff 4,722,366,482,869,645,213,696
2001:db8::/57 2001:db8:: 2001:db8:0000:007f:ffff:ffff:ffff:ffff 2,361,183,241,434,822,606,848
2001:db8::/58 2001:db8:: 2001:db8:0000:003f:ffff:ffff:ffff:ffff 1,180,591,620,717,411,303,424
2001:db8::/59 2001:db8:: 2001:db8:0000:001f:ffff:ffff:ffff:ffff 590,295,810,358,705,651,712
2001:db8::/60 2001:db8:: 2001:db8:0000:000f:ffff:ffff:ffff:ffff 295,147,905,179,352,825,856
2001:db8::/61 2001:db8:: 2001:db8:0000:0007:ffff:ffff:ffff:ffff 147,573,952,589,676,412,928
2001:db8::/62 2001:db8:: 2001:db8:0000:0003:ffff:ffff:ffff:ffff 73,786,976,294,838,206,464
2001:db8::/63 2001:db8:: 2001:db8:0000:0001:ffff:ffff:ffff:ffff 36,893,488,147,419,103,232
2001:db8::/64 2001:db8:: 2001:db8:0000:0000:ffff:ffff:ffff:ffff 18,446,744,073,709,551,616
2001:db8::/65 2001:db8:: 2001:db8:0000:0000:7fff:ffff:ffff:ffff 9,223,372,036,854,775,808
2001:db8::/66 2001:db8:: 2001:db8:0000:0000:3fff:ffff:ffff:ffff 4,611,686,018,427,387,904
2001:db8::/67 2001:db8:: 2001:db8:0000:0000:1fff:ffff:ffff:ffff 2,305,843,009,213,693,952
2001:db8::/68 2001:db8:: 2001:db8:0000:0000:0fff:ffff:ffff:ffff 1,152,921,504,606,846,976
2001:db8::/69 2001:db8:: 2001:db8:0000:0000:07ff:ffff:ffff:ffff 576,460,752,303,423,488
2001:db8::/70 2001:db8:: 2001:db8:0000:0000:03ff:ffff:ffff:ffff 288,230,376,151,711,744
2001:db8::/71 2001:db8:: 2001:db8:0000:0000:01ff:ffff:ffff:ffff 144,115,188,075,855,872
2001:db8::/72 2001:db8:: 2001:db8:0000:0000:00ff:ffff:ffff:ffff 72,057,594,037,927,936
2001:db8::/73 2001:db8:: 2001:db8:0000:0000:007f:ffff:ffff:ffff 36,028,797,018,963,968
2001:db8::/74 2001:db8:: 2001:db8:0000:0000:003f:ffff:ffff:ffff 18,014,398,509,481,984
2001:db8::/75 2001:db8:: 2001:db8:0000:0000:001f:ffff:ffff:ffff 9,007,199,254,740,992
2001:db8::/76 2001:db8:: 2001:db8:0000:0000:000f:ffff:ffff:ffff 4,503,599,627,370,496
2001:db8::/77 2001:db8:: 2001:db8:0000:0000:0007:ffff:ffff:ffff 2,251,799,813,685,248
2001:db8::/78 2001:db8:: 2001:db8:0000:0000:0003:ffff:ffff:ffff 1,125,899,906,842,624
2001:db8::/79 2001:db8:: 2001:db8:0000:0000:0001:ffff:ffff:ffff 562,949,953,421,312
2001:db8::/80 2001:db8:: 2001:db8:0000:0000:0000:ffff:ffff:ffff 281,474,976,710,656
2001:db8::/81 2001:db8:: 2001:db8:0000:0000:0000:7fff:ffff:ffff 140,737,488,355,328
2001:db8::/82 2001:db8:: 2001:db8:0000:0000:0000:3fff:ffff:ffff 70,368,744,177,664
2001:db8::/83 2001:db8:: 2001:db8:0000:0000:0000:1fff:ffff:ffff 35,184,372,088,832
2001:db8::/84 2001:db8:: 2001:db8:0000:0000:0000:0fff:ffff:ffff 17,592,186,044,416
2001:db8::/85 2001:db8:: 2001:db8:0000:0000:0000:07ff:ffff:ffff 8,796,093,022,208
2001:db8::/86 2001:db8:: 2001:db8:0000:0000:0000:03ff:ffff:ffff 4,398,046,511,104
2001:db8::/87 2001:db8:: 2001:db8:0000:0000:0000:01ff:ffff:ffff 2,199,023,255,552
2001:db8::/88 2001:db8:: 2001:db8:0000:0000:0000:00ff:ffff:ffff 1,099,511,627,776
2001:db8::/89 2001:db8:: 2001:db8:0000:0000:0000:007f:ffff:ffff 549,755,813,888
2001:db8::/90 2001:db8:: 2001:db8:0000:0000:0000:003f:ffff:ffff 274,877,906,944
2001:db8::/91 2001:db8:: 2001:db8:0000:0000:0000:001f:ffff:ffff 137,438,953,472
2001:db8::/92 2001:db8:: 2001:db8:0000:0000:0000:000f:ffff:ffff 68,719,476,736
2001:db8::/93 2001:db8:: 2001:db8:0000:0000:0000:0007:ffff:ffff 34,359,738,368
2001:db8::/94 2001:db8:: 2001:db8:0000:0000:0000:0003:ffff:ffff 17,179,869,184
2001:db8::/95 2001:db8:: 2001:db8:0000:0000:0000:0001:ffff:ffff 8,589,934,592
2001:db8::/96 2001:db8:: 2001:db8:0000:0000:0000:0000:ffff:ffff 4,294,967,296
2001:db8::/97 2001:db8:: 2001:db8:0000:0000:0000:0000:7fff:ffff 2,147,483,648
2001:db8::/98 2001:db8:: 2001:db8:0000:0000:0000:0000:3fff:ffff 1,073,741,824
2001:db8::/99 2001:db8:: 2001:db8:0000:0000:0000:0000:1fff:ffff 536,870,912
2001:db8::/100 2001:db8:: 2001:db8:0000:0000:0000:0000:0fff:ffff 268,435,456
2001:db8::/101 2001:db8:: 2001:db8:0000:0000:0000:0000:07ff:ffff 134,217,728
2001:db8::/102 2001:db8:: 2001:db8:0000:0000:0000:0000:03ff:ffff 67,108,864
2001:db8::/103 2001:db8:: 2001:db8:0000:0000:0000:0000:01ff:ffff 33,554,432
2001:db8::/104 2001:db8:: 2001:db8:0000:0000:0000:0000:00ff:ffff 16,777,216
2001:db8::/105 2001:db8:: 2001:db8:0000:0000:0000:0000:007f:ffff 8,388,608
2001:db8::/106 2001:db8:: 2001:db8:0000:0000:0000:0000:003f:ffff 4,194,304
2001:db8::/107 2001:db8:: 2001:db8:0000:0000:0000:0000:001f:ffff 2,097,152
2001:db8::/108 2001:db8:: 2001:db8:0000:0000:0000:0000:000f:ffff 1,048,576
2001:db8::/109 2001:db8:: 2001:db8:0000:0000:0000:0000:0007:ffff 524,288
2001:db8::/110 2001:db8:: 2001:db8:0000:0000:0000:0000:0003:ffff 262,144
2001:db8::/111 2001:db8:: 2001:db8:0000:0000:0000:0000:0001:ffff 131,072
2001:db8::/112 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:ffff 65,536
2001:db8::/113 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:7fff 32,768
2001:db8::/114 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:3fff 16,384
2001:db8::/115 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:1fff 8,192
2001:db8::/116 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:0fff 4,096
2001:db8::/117 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:07ff 2,048
2001:db8::/118 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:03ff 1,024
2001:db8::/119 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:01ff 512
2001:db8::/120 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:00ff 256
2001:db8::/121 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:007f 128
2001:db8::/122 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:003f 64
2001:db8::/123 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:001f 32
2001:db8::/124 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:000f 16
2001:db8::/125 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:0007 8
2001:db8::/126 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:0003 4
2001:db8::/127 2001:db8:: 2001:db8:0000:0000:0000:0000:0000:0001 2
2001:db8::/128 2001:db8:: 2001:db8:: 1

robots.txt
sketchy scrapers/bots
MediaWiki - Range blocks/IPv6

- demure