Inverting the Web
We use search engines because the Web does not support accessing documents by anything other than URL. This puts a huge amount of control in the hands of the search engine company and those who control the DNS hierarchy.
Given that search engine companies can barely keep up with the constant barrage of attacks, commonly known as "SEO". intended to lower the quality of their results, a distributed inverted index seems like it would be impossible to build.
Inverting the Web
Which is why I think we need to look to another web/network, which is people's social connections.
We just need a way for everyone to maintain and share a personal index that contains both information from automated scraping (i.e. keywords, SIPs, etc) and any notes or other metadata the person cares to attach to the page, domain, etc.
Of course, an index large enough to serve most queries you might be interested in would be pretty big, similar to the size of the document set.
Inverting the Web
The inverted index of the documents themselves is the biggest part. And since it's easy to verify, perhaps that part can be handled in a more centeralized, possibly federated, fashion?
@freakazoid What methods *other* than URL are you suggesting? Because it is imply a Universal Resource Locator (or Identifier, as URI).
Not all online content is social / personal. I'm not understanding your suggestion well enough to criticise it, but it seems to have some ... capacious holes.
My read is that search engines are a necessity born of no intrinsic indexing-and-forwarding capability which would render them unnecessary. THAT still has further issues (mostly around trust)...
@freakazoid ... and reputation.
But a mechanism in which:
1. Websites could self-index.
2. Indexes could be shared, aggregated, and forwarded.
4. Search could be distributed.
5. Auditing against false/misleading indexing was supported.
6. Original authorship / first-publication was known
... might disrupt things a tad.
NB: the reputation bits might build off social / netgraph models.
But yes, I've been thinking on this.
@dredmorbius This is not a fully fleshed out idea yet, but the "L" was the important bit. People generally don't care about the location of the content. They care about the content of the content, and other stuff about the content like the author, etc.
Just think about how people generally navigate the web these days. They don't type a URL into their addressbar or click a bookmark. They type a search query into their address bar, which will generally bring up Google results.
@dredmorbius So the question I want to answer is how do we enable that kind of navigation, or something similarly easy to understand, without giving a whole bunch of power to a single entity? How do we leverage people's existing trust networks, or existing reputable (generally topic-specific) databases to provide results with at least as good of quality as Google's?
The idea we had with xanadu is that, because links are part of an overlay instead of embedded, people would send each other packs of links between different documents, and you might subscribe to a themed feed of links the way you subscribe to an RSS feed or follow an account. It was supposed to be p2p but could be federated -- but doesn't work if overlay links don't work (i.e., if content is mutable or addresses are)
@enkiv2 Do you know / have you worked with Andrew Pam / Xanadu Australia?
I discovered his work w/ Xanadu in the closing months of G+.
@freakazoid Sorry, what "L"?
I'm not seeing reference to this and am confused.
@dredmorbius Sorry, I mean the "L" in "URL". It's a uniform resource *locator*.
Google is trying to build the thing I'm talking about, only it will be designed to give them even more power than they already have by hiding URLs entirely, making it so that there's no chance at all to navigate the web successfully without them.
@freakazoid OK, yes.
And, old hat to you, but the idea was to "locate on the Internet, by server and path": https://espace.cern.ch/webservices-help/GeneralUserInformation/GeneralinformationaboutWWW/Pages/Howthewebworks.aspx
... in a system literally designed by nuclear particle physicists.
Alternatively, L = I, "identifier".
Location == Identity.
Part of that remains valid. Part of it ... may not.
I've been kicking around the idea of a (local) document-oriented "filesystem" in which specifiers are effectively metadata descriptors or content-based keys. https://old.reddit.com/r/dredmorbius/comments/6bgowu/what_if_the_web_was_filesystemaccessible/
@dredmorbius Yeah, I've thought about similar approaches. Directories aren't required to be listable. Unordered bags of KV pairs don't map super well to hierarchical paths, but it's not like that matters very much. For most people a filesystem interface wouldn't matter anyway; they want a browser-ish application.
@freakazoid It ... depends.
There are times you want a _very specific_ resource.
It's not just _content_ that matters, but ownership, provenance, who can / did change / modify it, etc., etc.
There are times when "what colour is the sky?" can be answered by any of thousands of references.
The fact that _approximate, content-described results_ are _sometimes_ or even _often_ appropriate doesn't mean _always_.
@dredmorbius Indeed, but I'm not talking about getting rid of URLs, and for such things search engines end up just acting as a URL directory, since you will look until you see the URL you want.
@freakazoid A directory-path-based specification is saying "find this precise linked-list chain of directory specifications, with the implied properties of ownership, access permissions, modification history, provenance, etc., etc."
People looking for docs may allow slack. Software looking for libraries, somewhat less so.
And even humans looking for specific documentary authority may want a specific result.
@freakazoid The key for me is that _search is identity_, or at least _an identifier_, if _a search query_ returns _precisely one match_.
(Other options being "null" or "list".)
@dredmorbius I'm not sure I understand. It's possible for searches to return singleton results by accident. It seems like what you want is to distinguish between searchable metadata fields that uniquely identify resources and those that don't.
@freakazoid Right, that IS a problem, and a BIG one.
Possibly THE problem.
Q: Can documents be reasonably self-describing or self-identifying?
@dredmorbius I assume you mean *securely* self-describing?
Most distributed storage systems that try to defend against malicious nodes use exactly two types of keys, each self-certifying: content hash for immutable values and public key hash for mutable ones.
Beyond that you're into the realm of the subjective. My thinking here was to have signed triples ala RDF and use some kind of reputation system, i.e. web of trust, to decide which to trust.
@dredmorbius You could also provide a way to express *negative* triples as a way to try to correct errors or deliberate spam injected by others.
@freakazoid I'm not familiar with signed triples / negative triples. Sec....
@freakazoid I'd settle for "functionally" or "sufficiently".
I realise that any document under active attack (to change/misrepresent) would require more stringent methods.
But something that "usually works" would be a huge step forward.
@freakazoid Re: navigation.
1. Google are trying hard to kill off the URL.
2. There may be user-pattern based reasons to do just that.
3. URLs and DNS map ... poorly ... to meatspace notions of locality and identity. In large part due to the actions of websites, search engines, browser devs, SEO, and domain registrars.
4. A namespace with at _least_ a half-million entities and little sensible structure ... is far beyond human scale.
5. It's mostly reputation.
@dredmorbius I agree that killing off the URL is a worthy goal, which makes it a perfect weapon for Google to deal its final killing blow to the open Web.
As for scale, IIRC you can serve 90+% of web search requests with coverage of only about 5% of the space. Something like 99% Google results are served entirely from RAM. They don't even expect to serve useful results from their largest index; it exists primarily to give the impression of completeness.
Also YaCy as sean mentioned.
There's also something that is/was used for Firefox keyword search, I think OpenSearch, a standard used by multiple sites, pioneered by Amazon.
Being dropped by Firefox BTW.
That provides a query API only, not a distributed index, though.
@kick @enkiv2 @dredmorbius Not true; there are several decentralized routing systems out there. UIP, 6/4, Yggdrasil, Cjdns, I2P, and Tor hidden services to name just a few. Once you're no longer using names that are human-memorizable you can move to addresses that are public key hashes and thus self-certifying.
A system designed for content retrieval doesn't really need a way to refer to location at all. IPFS, for example, only needs content-based keys and signature-based keys.
@kick I'm with you in advocating for human-readable systems. IPv4 is only very barely human-readable, almost entirely by techies. IPv6 simply isn't, nor are most other options.
Arguably DNS is reaching a non-human-readable status through TLD propogation.
Borrowing from some ideas I've been kicking around of search-as-identity (with ... possible additional elements to avoid spoof attacks), and the fact that HTTP's URL is *NOT* bound to DNS, there may be ways around this.
@kick I'll disagree with you that WoT doesn't scale, again, at least in part.
We rely on a mostly-localised WoT all the time in meatspace. Infotech networks' spatial-insensitivity makes this ... hard to replicate, but I'm not prepared to say it's _entirely_ impossible.
Addressing based on underlying identifiers, tied to more than just content (I'm pretty sure that _isn't_ ultimately sufficient), we might end up with _something_ useful.
@kick @enkiv2 @dredmorbius @freakazoid This body remembers when the definition of "geek" was someone who used a computer to exchange text chat messages to people. At least, that's what it meant at UCSC. Going back further, was it Augustine who was mightily impressed that Anselm could read without moving his lips?
@kick To be clear, I'm trying to distinguish WoT-as-concept as opposed to WoT-as-implementation.
In the sense of people relying on a trust-based network in ordinary social and commerce interactions in real life, not in a PGP or other PKI sense, that's effectively simply _how we operate_.
Technically-mediated interactions introduce complications -- limited information, selective disclosure, distance, access-at-a-distance.
But the principles of meatsapce trust can apply.
@kick That is: direct vs. indirect knowledge. Referrals. TOFU. Repeated encounters. Tokenised or transactional-proof validations.
Those are the _principles_.
The specific _mechanics_ of trust on a technical network are harder, but ... probably tractable. The hurdle for now seems to be arriving at data and hardware standards. We've gone through several iterations which Scale Very Poorly or Are Hard To Use.
We can do better at both.
@kick A roundabout response, though I think it gets somewhere close to an answer.
"Trust" itself is not _perfect knowledge_, but _an extension of belief beyond the limits of direct experience._ The etymology's interesting: https://www.etymonline.com/word/trust
Trust is probabalistic.
Outside of direct experience, you're always trusting in _something_. And ultimately there's no direct experience -- even our sight, optic nerve, visual perception, sensation, memory, etc., are fallable.
@kick Building off the notion that "reality is what, when you stop believing in it, refuses to go away", we validate trust in received assertions of reality through multiple measures.
Some by the same channel, some by independent ones.
Getting slighly more concrete:
Simulator sickness is a problem commercial and military pilots experience with flight simulators. The problem is the simulator lies, and visual and vestibular inputs disagree. Sims are good, not perfect.
@kick I don't know if you've ever dealt with a habitual liar, or someone whose mental processes are so disrupted that they can't recall, or recall incorrectly, or misrepresent past events (or present ones). It's tremendously disorienting.
Our own memories are glitchy enough that you start doubtiing yourself. Having a record (journal, diary, receipts, independent witnesses) helps hugely.
Getting to theories of truth, consistency and correspondence seem to work best.
@kick Is a given narrative or representation *internally* consistent, or at least mostly so? And does it correspond to observable external realities (or again, mostly so)?
Mechanisms of trust generally try to achieve consistency or correspondence, sometimes both. In information systems, we tend to use one-way hashes, because those support the computational needs, but the hashes themselves are used to create a consistency or correspondence.
@kick So, in the "we have your dad hostage" situation, the scammer's failure was one of correspondence: dad was already dead.
But how you'd check this, *if you had the presence of mind to do so*, would be to attempt independent verification through other channels.
Call his number directly, or your mother's (assuming both are still alive and together), or current partner's. Ask to speak to him. Call the police, etc.
Falsehoods are common to any comms regime.
But yeah, a decent answer.
I do kind of worry about how fallible most WoT implementations are^1, but there definitely might be a way to do it, I’ll cede.
^1 Given that I as a random finance dork managed to reimplement the recent FastSpeech papers in ten days and get results decent enough to fool my SO when using it over a phone call (modern carriers started compressing call audio poorly when they internally moved to VOIP and the quality is pretty poor as a result), my confidence in what has previously been seen in a relatively decent way to verify (audio) has lessened slightly.
@kick I have been warning close friends and family members (some elderly and prone to dismiss technological threats and concerns as "nonsense" or "nothing I would want to use" or "beyond my understanding" or "but why would someone do that", v. frustrating) about DeepFakes and FastSpeech technologies.
I know that at least one has had faked-voice scam phone calls, though they realised this eventually. I'm predicting #deathOfTelephony based in part on this, BTW.
@kragen As with most words, there's a range of meanings. I'll admit to having pulled "extension of belief beyond the limits of experience" out of my hat, so it's not entirely standard. And that's "trust as a state of knowledge".
There's also the notion of "to put one's trust in (someone|something)", which can mean a binary rather than probablistic committment. We also have provisional or total trust.
Trust me, it's complicated.
@dredmorbius @kick @enkiv2 @freakazoid
Of course, one look at the state of computer security shows that for most cases (even very important ones) the social countermeasures are weaker than the technical ones. It's a lot easier to social engineer or rubber hose than to crack even a pretty weak password.
@dredmorbius @kick @enkiv2 @freakazoid
Search-as-identity is one solution, but I prefer petnames -- a decentralized identity system for decentralized networks. If somebody wants to find something globally it's fine to rely upon something strict but unmemorable, but finding stuff that's already resident on your box or that your direct connections are sharing ought to be a personal or community affair.
@enkiv2 SAI and petnames are two points in a space (not sure if 1D or n-dimensional).
Search utilises characteristics which may be internally-specified (content, transforms) or extenal (metadata, assigned identifiers).
Petnames are locally-assigned non-global identifiers. They may be _shared_ among some group, but they're localised, folksonomic, nonauthoritative.
(Though local names can become global with time/use/convention.)
@kick HTTP isn't fully DNS-independent. For virtualhosts on the same IP, the webserver distinguishes between content based on the host portion of the HTTP request.
If you request by IP, you'll get only the default / primary host on that IP address.
That's not _necessarily_ operating through DNS, but HTTP remains hostname-aware.
A social network for the 19A0s.