Does SearX let you filter out search results from certain sites? I really need to start filtering domains of content-mills entirely. Yes I know they're cropping up constantly, but that doesn't mean I have to keep seeing the same ones. Next step is to filter every domain that uses NameCheap as a registrar.

@freakazoid Not natively, but here's what I do:

Search request goes into Searx via REST API.

Search request comes back as JSON events.

For each URL, use a Website Agent to run it past a whois lookup API. Grab the registrar from the results (in your case), append to the end as a new key.

Run the event through a Pattern Detector. If the registrar key matches Namecheap, drop the event.

@drwho Given how popular it is to abuse whois APIs, I'm guessing I won't be able to find one I can do that with without paying.

@freakazoid Here is what I have in mind:

FaaS is basically "thing with a known entry point inside a container." You can put anything else you want inside that container, too. For example, a CLI WHOIS client that can output at least nominally machine parsable output.

Send the FaaS a URL. Fuzz out the domain.

The FaaS runs the CLI WHOIS client with the domain as an argument. It captures the output.

The FaaS packages the output somehow nice and returns it.


@drwho I'm familiar with how FaaS works. The problem is that the whois CLI is still calling a network API, and the APIs are rate-limited to prevent abuse.

@freakazoid Because I've never tried, how hard can you hit a whois service with the plain old CLI util before you get blocked?

Sign in to participate in the conversation
R E T R O  S O C I A L

A social network for the 19A0s.