
Software developer, intermittent indie game dev, formerly u/captainbland on reddit. Also kind of interested in medical imaging etc.


I guess it depends too much on the nature of the crawler. Does it actually extract links from robots.txt or is it merely ignoring them? If the crawlers are distributed, do page hits come from the same IP that the robots.txt was hit from?
It gets harder and harder to get away from CDNs and captchas, which are not exactly good things from an open source POV for the most part.
I don’t mind, 90% of the value I’m getting out of this is making the sales of the game the MAGA people feel betrayed by go up by one