asciimoo/hister: Your own search engine

Your own search engine

Cloudflare Just Gave AI Training Bots the Middle Finger

Cloudflare just gave website owners a new weapon against AI crawlers: keep the search traffic, block the AI training. After years of watching bots consume the web’s content, publishers finally have an easier way to tell AI companies where to go.

New font turns ordinary webpages into nonsense for AI scrapers - Ars Technica

“ShieldFont” aims to poison AI training data without making pages unreadable for people.

Crawlee for Python: Build a Web Crawling Pipeline with Robots Handling, Link Graphs, and RAG Chunk Export - MarkTechPost

Build a Crawlee for Python pipeline that crawls static and JavaScript-rendered pages, extracts structured data, and exports AI-ready datasets

Playwright

Pydantic

Google explains how crawling works in 2026

Google went through crawling, fetching, and the bytes it processes.

Crawl entire websites with a single API call using Browser Rendering · Changelog

Browser Rendering's new /crawl endpoint lets you submit a starting URL and automatically discover, render, and return content from an entire website as HTML, Markdown, or structured JSON.

How to Remove a Web Page from Google

The reasons for deindexing a page haven't much changed over the years, although the tools and tactics have evolved.

How to Fingerprint Websites With WhatWeb - A Practical, Hands-On Guide

Another one of our simple, hands-on pen-testing tutorials. This one teaches you to finger a website for status code, server info, CMS etc.

Imagine you’re curious about what powers a website: the CMS, web server, frameworks, analytics tools, or plugins behind it. WhatWeb can tell you all that right from the Linux command line. It’s like getting a quick peek under the hood of any site.

Crawling a billion web pages in just over 24 hours, in 2025

Crawlers And Agents And Bots, Oh My: Time To Clarify Robots.txt

Perplexity is an up-and-coming AI company that has broad ambition to compete with Google in the search market by providing answers to user queries with AI as its core technology. They’ve been…

Can Robots.txt Files Really Stop AI Crawlers?

The text file that runs the internet

As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.

Training Data for the Price of a Sandwich: Common Crawl’s Impact on Generative AI

Mozilla research finds that Common Crawl's outsized role in the generative AI boom has improved transparency and competition, but is also contributing to biased and opaque generative AI models.

How to crawl a quarter billion webpages in 40 hours

How to Use Web Crawlers for SEO

Web crawler tools are essential for search engine optimization, but the info they provide can be overwhelming. Here's how to prioritize and cut through the clutter.

Making a DNS query in Ruby from scratch

Home - YaCy

YaCy P2P - Decentralized Search Engine

SEO How-to, Part 9: Diagnosing Crawler Issues

Search engines must crawl and index your site before it can rank in organic search. Thus optimizing your content is pointless if search engines cannot access it. This is the ninth installment in my “SEO How-to” series.

Usage · ArchiveBox/ArchiveBox Wiki · GitHub

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more... - ArchiveBox/ArchiveBox