Federated Republic of Sean is a user on retro.social. You can follow them or interact with them if you have an account anywhere in the fediverse.

Does SearX let you filter out search results from certain sites? I really need to start filtering domains of content-mills entirely. Yes I know they're cropping up constantly, but that doesn't mean I have to keep seeing the same ones. Next step is to filter every domain that uses NameCheap as a registrar.

@freakazoid Not natively, but here's what I do:

Search request goes into Searx via REST API.

Search request comes back as JSON events.

For each URL, use a Website Agent to run it past a whois lookup API. Grab the registrar from the results (in your case), append to the end as a new key.

Run the event through a Pattern Detector. If the registrar key matches Namecheap, drop the event.

@drwho Given how popular it is to abuse whois APIs, I'm guessing I won't be able to find one I can do that with without paying.

@freakazoid Here is what I have in mind:

FaaS is basically "thing with a known entry point inside a container." You can put anything else you want inside that container, too. For example, a CLI WHOIS client that can output at least nominally machine parsable output.

Send the FaaS a URL. Fuzz out the domain.

The FaaS runs the CLI WHOIS client with the domain as an argument. It captures the output.

The FaaS packages the output somehow nice and returns it.

Federated Republic of Sean @freakazoid

@drwho I'm familiar with how FaaS works. The problem is that the whois CLI is still calling a network API, and the APIs are rate-limited to prevent abuse.

· Web · 0 · 0

@freakazoid Because I've never tried, how hard can you hit a whois service with the plain old CLI util before you get blocked?