I've been quieter than normal recently.
Time to fix that.
This is a thread about the things that I am working on/thinking about, and how they relate back to the current state of the world.
It's going to be one of /those/ threads, probably, so please forgive me if it takes me a little while to respond to you if you jump in the replies. I'll get there, I promise.
But first, some music.
I'm thinking Blues, but I'm always thinking Blues. Let's see what I can find on the shelves.
Alright, I'm spinning a shockingly clean copy of The Original American Folk Blues festival (if you've never heard it, do yourself a favor), drinking a cup of coffee (Decaf! The Horror!) and thinking about the #internet, the #web, #gopher, #usenet, The American Media Oligopoly, #copyright, #DIYMedia and the #PublicDomain.
It's been stormy. My head is cloudy, my bones ache, and I'm honestly terrified to my core of the future.
So this should be fun!
Sometime between 10 years ago and 5 years ago, @Ethancdavenport suggested to me that it'd be really cool if we could get a bunch of little intranets connected to one another all over the country/planet, and rebuild the internet.
This was honestly probably in the wake of the first brush with SOPA/PIPA?
God that seems like it was ancient history.
And it was probably before that, and just came up again in the wake of SOPA/PIPA.
Either way, the idea struck a chord with me.
The exact genesis of the idea isn't super important.
The point is that I started thinking about networking without/outside the internet.
As time has worn on, this idea has moved closer and closer to the forefront of my mind.
I've done various experiments with BBS style systems, and shared servers.
I spent too much time reading about the history of Usenet, and watching documentaries about BBSs, and just *thinking* about what a national or a global non-internet would look like.
I've written a lot, in other places about how the modern web is a trashfire. I'm not going to re-hash all those ideas here, but I'll sum up:
- Massive downloads for no reason
- Arbitrary code execution
- Tracking you constantly (advertising)
- Increasingly silo'd
- Tracking you constantly (NSA Panopticon)
- Bad Laws (we'll discuss this at length.)
- EME/DRM in browsers
We could also talk about how the modern Internet is a trash fire, but that'd basically just be me saying "NAT makes it difficult to self host, and we need more ISPs to offer IPv6 addresses" and ... well that's not what I want to talk about right now.
Before I go any further, I want to talk about USENET.
Do you remember usenet?
I'm going to quote wikipedia about usenet so that I can save myself some time.
"Usenet is a worldwide distributed discussion system available on computers. It was developed from the general-purpose UUCP dial-up network architecture. Tom Truscott and Jim Ellis conceived the idea in 1979, and it was established in 1980.
Users read and post messages to one or more categories, known as newsgroups."
More from the wiki article on usenet:
"Usenet resembles a bulletin board system (BBS) in many respects and is the precursor to Internet forums that are widely used today. Discussions are threaded, as with web forums and BBSs, though posts are stored on the server sequentially.
And then, the good bit. Which won't fit in this toot and will need to go in the next one.
"One notable difference between a BBS or web forum and Usenet is the absence of a central server and dedicated administrator. Usenet is distributed among a large, constantly changing conglomeration of servers that store and forward messages to one another in so-called news feeds. Individual users may read messages from and post messages to a local server operated by a commercial usenet provider, their Internet service provider, university, employer, or their own server."
So that's ~1200 characters of Usenet summary from wikipedia.
The whole article isn't bad, honestly, but that has most of the bits you need to understand where I'm going.
The important points to take away:
- Usenet predated the internet by a good while
- You still connect to a usenet server, but you (traditionally) connected directly to it over dialup
- Usenet fully supports servers popping up and disappearing all the time.
- At this point, usenet is mostly spam, piracy, and viruses.
Like BBSs, Usenet is basically only accessed over the internet today.
Traditionally, though, there wasn't an internet.
usenet posts spread by having college servers straight up call one another, and transfer data. (Or, in some cases, by mailing truck loads of tape to another continent in order to bootstrap a new server, because it took less time to arrive than it would have over dialup.)
So your local server fetches the posts.
Then, people with shell accounts on those multi user servers could log in to their account, and fetch the news to their local newsreader.
They'd treat the messages kind of like emails. When they were done, they press a button, and sync their new posts back to their local usenet server.
The next time their server calls another server, those posts propagate the whole network.
@golf_oil @ajroach42 it’s worth keeping in mind that email continues to be a federated, distributed protocol as well, and it, too, was originally designed to work in a store-and-forward way which could take DAYS to propagate. And if you’ve ever tried to diagnose a delivery delay on your own server you might end up being really annoyed by that. :)
Also NNTP tried to add in a means of revoking or deleting a message and that is as always one of the two hardest problems (naming things, cache invalidation, and off-by-one errors)
In some ways, it was honestly a lot like the current fediverse. In other ways, it was not at all like the current fediverse.
If you want to get a feel for what it was like, check this out: http://olduse.net/
It's an archive of Usenet, updating in real time, as if people were making these posts.
And that's *really* fucking cool.
Here's some more information about the Old Usenet website: https://joeyh.name/blog/entry/announcing_olduse.net/
There was, at one time, a blog post about trucking the tapes across Canada too, but I can't find a copy of that which is still online.
Shame, but that's the way these things go.
So Usenet was designed to allow people all over the place to talk to one another over intermittent, slow, indirect connections.
These schools and other places used a lot of the same ideas for cross system Email, if I'm not mistaken. I dunno, it's been a long time since I read about email.
The upshot, though, is that we have something of a template for how to propagate certain kinds of information over shitty intermittent connections.
Of course, usenet isn't a solution today, because the problems we're trying to solve today aren't the problems that usenet solved.
But it's important to think about, and to remember, that The Web isn't the only way.
Which brings us back to the internet, and the ways it's falling apart.
Personally, my biggest concern about the modern internet is that the whole thing is resoundingly fragile, while acting like it is indestructible. (https://boingboing.net/2016/11/11/the-internets-core-infrastru.html)
It was designed to be fault tolerant. It was designed to be censorship resistant.
In some ways, it is!
In other ways, it's a massive, ubiquitous tool of surveillance.
The Web was designed to deliver documents with light markup, and it's now used as an application layer, complete with arbitrary code execution.
The internet, and the web, are so big that they are resistant to change. They have inertial mass.
@ajroach42 The fact that the web can evolve at all with as big and complicated as it is is fairly miraculous. To a large extent I credit the test infrastructure Google built (and then talked about) for Chrome to make it so that they could build a compatible browser without an insurmountable amount of effort.
I gotta do another quick sidebar here.
Ultimately, what I want to talk about is building a regional (then national, the international) network that has more in common with fidonet and BBSs and Usenet than the internet, and how we could do that, and what it would look like.
But, before I do that, I gotta talk about one of the biggest reasons that the modern internet is less good than I want it to be, and that's The Copyright Industry.
I have a lot to say about copyright, and I also just recieved some Pizza. So I'm going to eat, and then I'll talk about copyright, why big content is evil, and what I think we should be doing about it.
Then I'll circle back around to how to build a new fidonet.
Oh, while I'm eating: http://www.lowtechmagazine.com/2015/10/how-to-build-a-low-tech-internet.html
Go read that.
It's got some not bad ideas.
Okay, Pizza has been consumed.
So a lot of the threats to the internet come in the form of real shitty laws that the content industry or the telcom industry (and, in case you weren't aware, these are often the Same Companies) really want to see passed.
Content companies, that is to say, Disney and the rest, are evil at their heart.
I've covered the ways why before:
and I've talked a little bit about what to do about it.
I wanna do more of the second.
Simply put, we have to stop giving money to our media oligopoly.
That means we have to stop paying for media made within that media oligopoly.
That means we have to start making our own media, consuming more public domain media, using the local library's DVD collection and, when all else fails, piracy.
I can't help with the second two, but I can sure as hell help with the first two.
This is one of the small number of projects I'm actively working on.
I currently have a library of ~1000 feature films, a few dozen film serials, and hundreds of TV shows and Radio Programs that are in the public domain.
I'm working to catalog and organize these and get them in to a format that is consumable from other people.
I'm focusing on tracking down the best available copies of these public domain works, and on presenting them in a way that is conducive to discovery.
Of course, you'll also be able to submit your own CC-BY or CC-BY-SA content, which our team of crack reviewers (that is to say, me and a coupla punks I know) will consume and categorize along side the PD stuff.
In an ideal world, we could become something less like Project Gutenberg for film, and more like a repository of the best DRM free Open Licence video and audio in the world.
But I'd settle for PG for film, if that's where we land.
But the big tech companies are trying to make running sites that accept user submissions onerously expensive. Essentially impossible.
If the EU copyright thing passes, I wouldn't really be able to afford to operate in the EU. (and that's without the burden of international copyright to think about.)
I don't kid myself that I stand a chance of actually making an impact on the bottom line of hollywood, but starving them financially is only part of the goal. Getting them out of my head is the other.
Okay, back to building a modern distributed BBS/Usenet system.
I've talked about doing this within Dat or something like SSB, presenting something that pretends to be the modern web, complete with all it's problems, but only gives you access to whatever files are stored locally.
I've talked about doing it with something like UUCP or NNTP on the server side. I've even considered just using newsreaders on the client side.
There are problems with all of these approaches on the client side and the server side.
Let's ignore the "how" for a minute and talk about what I want to accomplish.
I want a network of servers (I usually call them nodes in my head) that a user can interact with from their laptop or mobile device over a wireless network.
I want these nodes to present local chat, a local and a global message board, something akin to email, and something akin to web pages.
If we have video content or audio content, that's also fine with me.
I want users to be able to sync any of this content to their local device, and for them to be able to set aside a portion of storage space on their device to serve as carrier pigeon space, if they choose to do so.
I want the nodes to be able to sync data directly, if the opportunity is available. I want them to be able to sync data via sneakernet or dialup or RF or long range wifi, and for them to be able to be certain of the data's origin, and that it has not been tampered with.
I want users to be able to sync data directly from their devices to the nodes, serving as a kind of sneakernet themselves. That's the reason I've toyed with Dat and SSB.
But Storage is a problem when you're whole network is one machine, you know?
And I don't know what the solution to that issue is (other than expiring content, or adding more disks.)
SSB particularly seems to mandate massive amounts of storage space, and doesn't provide any provisions for deleting things.
I'm not okay with either of those things.
There are more hurdles than just storage though.
Computers expect networks to be connected to the internet. Phones expect networks to be connected to the internet.
When they aren't, computers and phones behave oddly. imessage, for example, won't work at all if you don't have internet on your wifi connection, even though you have a cellular connection.
You can't tell some apps to use the LAN and some to use the cellular connection.
I lost a few hours of my life a few years ago researching ways to do this over bluetooth.
I didn't really get anywhere, but it still seems like a viable solution.
I don't think I even want the network to be accessed with a traditional web browser by default. (falling back to a web browser is fine, I guess. Having it as a legacy option. But this isn't The Web, and pretending that it is will lead to incorrect expectations about what it can do.)
Of course, all of this must be built to understand and use strong encryption from the ground up. And that leads to even more trouble.
Anyway, that's what's on my mind right now.
Dismantling Hollywood and building a modern Usenet successor.
I'd love to know your thoughts, or hear your take on what I have wrong, or what I might not have thought of.
I'm going to go play cards.
@ajroach42 Man, I'm not gonna go through and individually like all these, but this is some good shit. I'm really interested in following/helping to build what you're dreaming of.
As an artist, I slightly disagree with your perspective on big media. I agree that copyright conglomerates need to be cut out of the picture. Publishers aren't helping anyone these days.
What we need to combat them isn't piracy though, it's a way to pay artists for their work, and help them gain traction, without publishers.
Hi, thanks for your ideas and insights 🙂
I'm not as well informed as you but I'm interested in mesh networks and more decentralised internet alternatives.
I've been thinking about communicating over the internet without DNS. It would be based on pairs or groups of trusted 'friends' that keep each other updated with their changing IP addresses. Your ID on the network would be a public key and you could connect to non-friends via chains of friends of friends.
Might make it have a store-and-forward behavior, so when you met your friends in meatspace your phones would exchange traffic.
Might as well encrypt the payloads immediately, or not, if you were talking to the world in general.
Adding a little bittorent for redundancy and error correction and message loss detection. Add some privacy right up front.
A thought: there are myriad other ideas that are swirling in my brain soup with reference to fixing the web and or the internet.
This is just the one that I feel is most within my ability to implement.
I’d love to hear your ideas for a better web with the internet, and a better internet with or without the web.
What internet technologies most need to be fixed? Which ones are safe?
I don’t talk about this stuff as much because I don’t feel Qualified to talk about what could be improved.
@ajroach42 You've probably heard from me, but I'll list a few things I'm doing in Odysseus.
The first thing I'm doing is working on great but loosely coupled integration into 3rd party RSS & ActivityPub clients. Simply because I'd like to see these technologies succeed.
Then I've got plans for creating a better bookmark UI that will include features to let people easily share searches of their bookmarks as RDFa. Because this could help people deal with more sites and fewer silos.
@ajroach42 But what really excites me is that I want to do is help people deal with more search engines. Because having a single search box is privacy invasive, fragile, and limits the opportunities for a great UX.
In Odysseus this means you'll be able to search multiple preselected simultaneously, possibly filtered by tag. OpenSearch + RSS/Atom is a great standard for this, even allowing some support for faceted search!
@ajroach42 (final post here)
Taking on search would also involve catologing compatible search engines, as well as providing or encouraging tools for helping people create excellent search interfaces Google can't. And to curate their catalogues. Perhaps bookmark sharing would help with that.
Then It'd be interesting to implement privacy respecting personalized recommendations, as well as Dat or IPFS-based archiving.
@ajroach42 I know this discussion is a day old, but I couldn't help but to notice that your ideas mirror mine in some surprising ways. using old style methods for circumventing the modern internet is a very interesting idea to me, but I cannot foresee wide adoption. It's not for the masses, and would only satisfy those who feel constrained by the changes brought on by the corporatization of the Web. For this reason, I feel it would be impossible to create a widely interconnected mesh
@ajroach42 however, if the internet doesn't become completely closed, it may be possible to create a semi-dark network of relatively low-tech services that traverses the established WAN. Though I could see services like Tor and I2P being blocked or even criminalized in the future if the situation deteriorates enough, and that would set a precedent for any other hopefuls
@ajroach42 classy putting words in my mouth, but it does look like you saw my second reply after the fact. I like the ideas you put forth a lot, but I feel that (ironically) there needs to be a more centralized effort toward an old internet revival, at least initially. I’ve seen efforts toward a mesh network bubble up and fizzle out or stagnate. if the movement gathers enough attention, it might even be worth having a local mesh in some areas
@ajroach42 no hard feelings at all. I'd love to see some real action come out of these discussions. I may be jaded, but I feel that the way things are going, going "underground" might be the only way through it all. I love seeing people talk about making their own networks, I just personally can't see adoption by the regular Joe. that's not a bad thing, but I think it's good to consider that it might be only certain kinds of people on these meshes and whatnot. I suppose I should have clarified
@ajroach42 I guess what I’m saying is that BitTorrent solves this particular problem. What would you need from ssb beyond seeding, downloading from multiple servers and all that – the ability to sync while you’re online in bits and pieces? I think BitTorrent can do it. What it can’t do is find the archives for you – but if the works you share are PD then indexing them is easy. PirateBay is not required.
There are two thing I'd like to add. Torrents work fine if the content is en vogue. Only die hard fans seed long enough for rare content to be available.
Second if you found the content, there is no way to connect with the sharing person. Sharing is automatic, cold, inpersonal.
The protocol does not forsee a way of human interaction. That's what set it apart from old sharing protocols like napster
@ajroach42 also, here's some notes I've taken on tech that could be useful in independent media distribution
@freakazoid I wrote that blogpost once: http://ajroach42.com/how-to-fix-new-content-discovery/
@ajroach42 You talk about locality in that post, but then you mention Kevin Bacon, which implies f2f. Are you thinking both? Or "local" in the sense of "people you know"?
RSS or RDF seem like interesting technologies for discovery. Or a microformat of some kind. Personal f2f network crawlers?
The original idea was local but federated, using locality as an artificial constraint.
At the time that I wrote this, I saw most of my friends several times a month, because I was running the location they all used as a hub to meet up.
Many of them rarely saw one another, but new things that one of them hand found would pass to the others through me.
My thought was that these folks for whom I was a hub had other folks who were also hubs for different groups. In a loose way, data was already passing between these groups, and I wanted to see it more formally, and less ephemerally.
I wanted to see that same kind of network structure play out digitally (which, incidentally, is how we landed on the mBBS. Folks would upload stuff, other folks would download when they came by. It wasn't perfect, but it worked pretty well.)
@freakazoid So yeah, originally it was supposed to be local, supplemented by long range f2f, but entirely offline.
These days, since I'm living in the middle of no where, and I'm more spoke than hub, I'm more inclined to f2f supplemented by local, instead of the other way around.
My original use case doesn't work with me living in the woods, and so I'm re-examining the whole structure of the project.
@krozruch @ajroach42 I remember hearing about this collection of disparate networks, often running different protocols on different hardware with different operating systems, communicating with one another through "gateways", which were just computers, running a common protocol.
That network was the Internet. That's how it used to be described.
@ajroach42 @krozruch Maybe that's what we need to be building. Or rather, we should be building two different kinds of things: the individual community networks, and the overarching network that connects them together. The overarching network, of course, could be the Internet itself. The Internet is already like the badlands between cities where you only go if you want to get tracked, spammed, scammed, and robbed.
In my original notes I talked about tunneling over a VPN set up directly between two nodes on the community network as a last resort when there was no other way to get a node in to the network.
Making that a common fallback is not a bad thing as long as the internet remains ... the internet, and we can do it safely.
It'd help us bootstrap.
@ajroach42 @krozruch I think the goal should be to build communities that aren't dependent on some big corporation for their existence, and to facilitate both high-bandwidth communication & sharing within those communities as well as a certain amount of safety from what's outside those communities.
That describes the structure of the Fediverse to a certain extent, though there are a lot of single-person instances, and a lot of people are on large instances that aren't really communities.
@ajroach42 that's funny, @kemonine 's lollipop project prompted me to restart my ham mesh project that was on the back burner for too long ... The gist was nodes announcing a few metadatas on 15-20Mhz ( for a wide discoverability and help orientation of antennas), and uplink on 900Mhz to still have a bit of speed and good reach. Bgp for routing and facilities like excluding bad nodes etc...
@ajroach42 Yeah but lots of programs grow far beyond their initial designs. At first UNIX was only for developing telephone technology. And Microsoft Word was originally a way to create animated anthropomorphic paper clips and eventually grew into a word processor!
@ajroach42 Tunnels need only depend on IP routing, so BGP and the physical links themselves. BGP is one of the Internet's achilles' heels, but DNS is more likely to break. We can get redundant links in place as quickly as possible, but even multiple tunnels can help in some cases, for example if you have line of sight to a neighbor using a different ISP.
We talked about using dirigibles for this long before Googles balloon networking was a thing. We were mostly joking of course.
but hitching the data on busses* ... now that would be cool.
* The large wheeled vehicles that tend to run fixed route kind, not the bundle of wires kind.
@deejoe @ajroach42 Worth noting that this has actually been done! http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.476.2591&rep=rep1&type=pdf
@ajroach42 The existence of olduse.net makes me feel some kind of way that is very hard to describe but made me exclaim "oh no." Like it's an adorable baby internet too pure too good for this world and I don't want anything to happen to it but I know things will happen. Kind of want to cry and spend the next 24 hours reading old usenet posts but then I would lose my job.