How to Build a Low-tech Internet
Reading this now, would love to know your thoughts.
The thing that intrigues me the most about this is the concept of Delay Tolerant networking (in general) and the idea of developing secure, delay tolerant analogs for existing networking bits.
Email and RSS are great examples of the potential for this. We currently do both of these things through a web service, and in the vision of delay tolerant networking presented in the article we could continue to do so, or we could replace the web service with a local application.
(Thread, of course)
Email can already be delay tolerant, if we use pop3 and smtp instead of imap.
Blogs, vlogs, podcasts, etc can already be delay tolerant if we use RSS.
Message boards can already be delay tolerant, using NNTP/usenet protocols.
Social has a few delay tolerant options, including SSB. More are possible, if we build up to them.
That leaves us with a few of the more interactive tasks that are far less delay tolerant.
Media streaming (aside from, as mentioned earlier, podcasting) tends to be an on-the-fly activity. If we designed our systems to do so, there's no reason (other than copyright, I suppose) media that was streamed couldn't be stored locally so long as there is room (google play music does this to great effect)
Once stored locally, that same media could be shared with others within range (Dat or peertube style)
This kind of thing isn't exactly suited for 100% asynchronous networks, but if you're in a situation with intermittent connectivity it works well.
Research is another activity that functions better with a synchronous connection.
But it's not entirely impossible with an asynchronous connection, depending on how you go about it.
For a while, when I had a portable computing device but not mobile internet (a palm pilot), I used a piece of software that would automatically fetch web pages, format them for the palm, and sync them. I set it to fetch every 1st and 2nd degree link from a page (so every link on that page, and every link on those pages) with some other tolerances to keep the resultant bundle of files from getting too huge (I only had 1GB of storage available for web pages, after all)
The software would strip out the images (unless I told it not to) and reformat the page for the smaller screen of the palm.
I could set the system up so that pages I wanted to visit that weren't available would be requested the next time I connected.
While it was primitive by modern standards, and it was occasionally frustrating, It was actually pretty damn functional. I used this as my primary means of web browsing throughout highschool and well in to college.
So, I guess what I'm saying is that many internet activities were originally designed to work asynchronously, and many more can be made to work asynchronously.
If we make delay tolerant networking a priority, the end result could probably be pretty fucking useful, especially as traditional online spaces get less trustworthy.
So my ideal implementation of something like the data mules (I hate this phrase, I'm going to say data trafficker, even though that's not much better) described in the article would look something like this:
1 - In a high traffic, relatively central area there is a wireless node that can be connected to from PCs, and that will automatically connect to passing data traffickers.
2 - This node serves as a local intranet for anyone in range (via long range wifi or local wifi) and works like an old school BBS. (http://ajroach42.com/a-modern-bbs/)
2.5 - It's almost certainly powered by something like Dat or IPFS (http://ajroach42.com/steps-towards-a-web-without-the-internet/) and probably gives users a local home page to serve as a jumping off point, as well as having an option for local real-time services and storing asynchronous requests to pass along to the data trafficker when the time comes.
3 - If I make a request for a piece of content that isn't stored locally, that request get's forwarded to the data trafficker when they are in range. If they have the data, they send it. If they don't, they store the request until they are in range of an internet connection or a connection to a larger network.
4 - Until a request is fulfilled, it's sent out to everyone that passes through. When/if someone passes back through with the data, the request is cleared, and that data is stored locally until the original requester picks it back up.
5 - since most of what we are doing is being done on local machines with fault tolerant applications, the nodes don't have to be particularly high powered, which means they can survive on alternative power sources, and it's acceptable if they are not always available (as a result of weather, or general impermanence, or whatever.)
6 - Whenever possible, the data we're passing around is cryptographically signed, so that we can be reasonably certain it hasn't been tampered with (this is why I like Dat or IPFS).
7 - Any machine participating in the network can be/is a gateway node, and a data trafficker.
The biggest upshot of doing things this way, IMO, is that it eliminates the problem of discovery.
That is to say, in a synchronous system you can quickly look for a thing and, if you don't find it, modify your parameters through several quick permutations until you do.
having these gateway nodes with relatively large caches of local content can eliminate the hassle of dealing with the *very* high latency of your queries, by providing indexes of known (even if not currently available) content, and by providing faster feedback to initial queries.
Executed correctly, and carefully, this kind of not-really-the-internet asynchronous network could provide a viable alternative to traditional connectivity in rural areas and poor communities (that is to say that I fully expect the system I described here to be more useful and much cheaper to implement and maintain than traditional internet in rural parts of, for example, the south-eastern US where I live.)
@ajroach42 No one actually listens to that. Your body is almost certainly saturated with encrypted waves.
The FCC doesn't even have a dozen people there.
And I agree that the protocol level should be where encryption lives. You're allowed to do signatures (as: "checksums") and that's all you need to ensure you know which node you're talking to.
And you'd be required to transmit your callsign, which is tied to a public listing of your name and mailing address in the FCC's Universal Licensing System database. (The mailing address need only be one that you can receive mail at, however.)
Zero anonymity on amateur radio, mandatory legal name policy by law.
@bhtooefr @ajroach42 @Elucidating As an amateur radio operator I can tell you that nobody who's part of the mainstream amateur community ignores the restrictions on encryption, and many of them will report those who do.
If you're going to ignore part 97 anyway you're almost certainly better off *not* getting an amateur radio license, since you are probably violating more laws by doing it as a licensed amateur than as a random member of the public.
There are a lot of people who treat amateur radio as a homeowner's association, and they take it on themselves to hound the FCC until they deal with a perceived violation. Actually transmitting encryption on amateur radio is one that the FCC could actually care about.
@seanl @ajroach42 @Elucidating Ultimately, the better bet is to do this kind of thing in Part 15 territory, between the licensing requirement (I mean, a Technician license isn't hard to get), the content restrictions, and the crypto restrictions.
Or, licensed commercial bands (things like the 3.65 GHz 802.11y band), for higher power operation. Those can carry basically whatever you want legally.
@bhtooefr @ajroach42 @Elucidating If licensing were the only problem with amateur radio we could just get node operators who run long distance links to get licenses and then use wifi for the last 100 meters. But the crypto & content restrictions have basically made amateur radio irrelevant for all but pie-in-the-sky experimentation & talking about your radio unfortunately.
Plus, with part 15 you don't have busybody hams reporting you to the FCC, since that's how part 97 tends to be enforced.
@ajroach42 @bhtooefr @Elucidating I was just reading the FaradayRF page and it looks like while it's intended for amateur radio it can probably be used under part 15. Part 15 technically requires type certification but they've been pretty lax about that, and that applies to manufacturers anyway. I think the reason they're selling this under part 97 is because it's easily modifiable by the user.
@ajroach42 @bhtooefr @Elucidating 915 MHz is shared between part 15 and part 97 users, so I have no idea how anyone would even know you were using it unless it's using a recognizable protocol or you were interfering with someone.
915 MHz can also tolerate not having line of sight better than 2.4 or 5.8 GHz. It was used for the link between Ricochet modems and the base stations, while they used 2.4 GHz with line-of-sight for the uplink from the base stations.
@ajroach42 @bhtooefr @Elucidating That's also the band used by the GoTenna and similar products to get reasonable range without needing line of sight. Line of sight is still better because you're relying on the waves bending around or reflecting off of objects rather than penetrating them, of course.
@seanl @Elucidating @bhtooefr I’ve looked at some other ~900mhz radios in the past, but I’ve been scared off by the amount of work that would be left for me to do on implementing the stack (balanced against my available free time), and the relatively low bandwidth of the radios.
This, or something like it, would work very well for sending text over reasonably long distances, but we’d still need the sneakernet/data traffickers.
@Elucidating @seanl @ajroach42 I mean, there's existing protocols that do this, too - the MBL/RLI and FBB protocols are store-and-forward protocols used pretty widely for packet BBSes and for e-mail. They're typically carried over 300-1200 bps AX.25, although 9600 bps links can be done with good equipment (emphasis on good).
The big ones that would make it awkward for current Internet usage models are commercial communications, obscene/indecent communications (AFAIK that may be unconstitutional), and music.
Cryptographic signatures on transmissions, though? The ARRL isn't the ultimate authority on amateur radio. There's plenty of groups doing signatures over amateur bands publicly, I've never heard of anyone getting in trouble for it.
@bhtooefr @ajroach42 @Elucidating I think what happened was that we specifically asked about it, after asking some fairly detailed questions about use of encryption over packet radio, and they basically had enough of us pestering them. We spoke to our lawyer about it and she advised us to let the matter drop.
over here (UK) encryption on ham bands is only alllowed if assisting Emergency Services.
it is allowed on commercial PMR systems (I have a personal license for these and look after two more for work) but these have limited range and bandwidth. Long range wifi links/community wifi are allowed here and only lightly regulated.
For USA I remember reading about an unique allocation (not amateur) somewhere in the GHz range, another possibility maybe?
alas I cannot find the PDF, it was on some random electronics link on internet archive, maybe even something @ajroach42 linked to a while back, possibly a section of L-band, but it might be old info and since the FCC may have sold off commercially what bits aren't already used by the Pentagon (and the company which was /supposed/ to use it for some "medical telemetry" went bust anyway...)
The allocation was way higher up the band than FRS, possibly in the L-band.
I was looking through all FCC bandplans at the weekend and couldn't find it again, I vaguely remember the info I saw about it being from late 1990s/early 2000s.
I suspect it was quietly sold off to commercial operators around the time USA telly went digital (and may be planned for use for internet in rural areas, as an alternative to satellite (that is laggy!)
@vfrmedia @drwho @bhtooefr @ajroach42 @Elucidating There are several ISM bands above 1 GHz but the higher you get the more line-of-sight you become. There are also ranges where you have to deal with water and molecular oxygen absorption. I'm not sure anything other than 900 MHz, 2.4 GHz, and 5.8 GHz are worth considering for RF links. Free space optical links are an interesting option for short range (few km) and high speed.
Where our new home is, the only options for traditional connectivity are satellite (high latency, low bandwidth, expensive) and cellular (low latency, high bandwidth, intermittent connectivity due to signal, data caps, very expensive.)
I will personally be installing a cell extender and getting my internet through the Calyx institute's cellular connection, but that's not an option for everyone, and it's still pretty pricey.
This kind of system, if we can support it with the right software, has a real potential to revolutionize and revitalize some communities.
And I, for one, couldn't be more excited.
@seanl As far as I know, they don't offer a monthly package? At least all I could find online was something like $500 for the first year, and $400 for each subsequent year.
It's super reasonable compared to traditional cellular, and when you break it down in to a monthly rate it's way less than I currently pay for internet.
When I say that it's still pretty pricey I'm coming from the perspective of a town with a median household income of $25k.
A related conversation: https://mastodon.social/web/statuses/99801921772823746
@ajroach42 well, maybe... the point was exploring answer to this question: can we build a better internet over a partition tollerant protocol? And how such protocol would look like?what's the pripr art?
It's pretty similar to what you call asynchronous don't you think?
Indeed while reading your thread I was surprised by the similarities...
So, how do we get started on a delay tolerant network?
Do we start with hardware or software? What existing software can be used? or repurpused? What software needs to be abandoned and replaced?
What software solutions already exist for Store and Forward? UUCP? NNTP? BBS?
What hardware already exists? How can we use existing hardware? Wifi? Xbee? LoRa?
@alphakamp @jeffcliff I assume he meant: https://en.wikipedia.org/wiki/Interplanetary_Internet
But also, IPFS is useful, I guess. .
@ajroach42 Well, back in the day FIDONET was extremely delay tolerant; it had to be.
Also, NASA's DSN is pretty darn cool, too: https://en.wikipedia.org/wiki/NASA_Deep_Space_Network
The article was never down for me, but if it is still down for you: https://web.archive.org/web/20180410163128/http://www.lowtechmagazine.com/2015/10/how-to-build-a-low-tech-internet.html
So, if you were going to design your own Delay Tolerant/asynchronous/store-and-forward network application, what would it be?
Dat tracks changes in files, so it can be used as part of a version control system. Syncing Dat is faster than syncing IPFS. Dat is specifically designed to do sync, where IPFS isn't.
IPFS must use the IPFS network. Dat doesn't care what network stack it uses.
@alcinnz @emsenn I go in to some detail about how I want to use Dat here. http://ajroach42.com/steps-towards-a-web-without-the-internet/
If you're familiar with IPFS, you'll probably see some of the key differences illustrated there. I'm less familiar with IPFS than I am with Dat, because my initial research made Dat appear to be the better choice.
the JS/Electron only aspect is worrisome, but I hope that's temporary.
But I do want to correct the latter point. IPFS does a lot of work to be independant of the underlying networking and hashing algorithms.
@ajroach42 it's always been a thing, it's just a matter of what kind of delays you want to tolerate.
FPS games tolerate delays of a few hundred ms max but can recover from larger delays because the context is very short term. But low latency on average is vital or it's worthless.
Email can handle very long delays and an email server can keep trying to send a message for days, but your minimum delay is unbounded; often a few minutes in practice.
This is not a new topic.
@ajroach42 HTTP has headers for cache control. These tell the client and server, "this is how long a delay may be before you should assume something has changed." DNS as well. Anything that deals with caching, which these days is anything big, has either a timeout, a refresh notification of some kind, or some system to turn stale data into fresh data or die trying. Even CPUs, with memory fences and such.
@ajroach42 There are multiple reasons for this. They come down to how resources are managed but there's several orthogonal resources in play here. Bandwidth, latency, ease of creation and how data is organized for human attention are orthogonal. Optimizing for one tends to have costs for the others, and technology changes fast enough that you have order-of-magnitude differences in relatively short time frames.
@icefox Yeah, unfortunately that's kind of all I got out of the discussion so far.
I figured that maybe you just didn't read all the thread, and that's why you were making points that had already been made, but that didn't explain the combativeness.
I guess everyone has cranky days. Who am I to judge?
@icefox couldn't agree more.
That's partially because it's not a single problem. It's a collection of problems, and we'll need to solve each problem in multiple ways.
But also because people's needs are different. What I need out of my network may be vastly different than what you need. so the solutions will necessarily be different.
So there's plenty of room for experimentation.
@ajroach42 The dichotomy you consider is not delay, it is remote vs local. There are programs now that can retrieve remote websites. It's just that a lot of websites also assume the existence of other remote services which you cannot easily retrieve (because they're database lookups or something). Indexing is the same; do you retrieve remote sites and index them yourself or let google do it for you?
It's not just a matter of latency because it also becomes a matter of control.
@icefox I agree that this is not a new topic, and I talk about several ways that it has been done in the past over the course of the thread.
Most modern software, even software that should otherwise be very delay tolerant and compatible with the kind of store-and forward protocols mentioned here, is written with the expectation that it will have an always available internet connection.
This isn't great for situations where an internet connection is intermittent or flaky.
@ajroach42 Most of what you described is like reverse Secure Scuttlebutt -- it's got the same properties (delay tolerant, intermittent WiFi is good enough to exchange packets). As it supports custom applications built on top of it might be a good starting point. IIRC everything is Node.js and it's pretty heavy on storage and compute though -- not a lightweight protocol but literally good enough for people who are sailing to exchange messages when near land WiFi
@insom Yeah, SSB is probably worth exploring in more detail.
Dat is conceptually very similar to SSB, the overlap between the two is why I'm leaning towards Dat over IPFS right now.
But that's just a piece of the puzzle, and a young, cumbersome, and heavy one at that.
Lots of other pieces also need to be solved, and SSB might fit in to the mix somewhere.
Haven't tried it, but a friend recommended it.
I recognize that it is possible. I further recognize that it will not suit my needs due to FCC regulations.
Plain text only transmissions with legitimate restrictions on the contents of your transmissions makes Ham a no go for me.
@ajroach42 One of the hackathon projects I participated in at Facebook was an attempt to send Wikipedia over a one-way, lossy broadcast channel using erasure coding. We were able to get it working pretty well in simulations, though it required tuning the code rate to get it right.
One could use a rateless code like online codes to do this sort of thing without the tuning. They could also replace the "rarest first" part of BitTorrent with "every seed block is useful" https://en.wikipedia.org/wiki/Online_codes
@ajroach42 Given the huge size of media compared to text, it seems like you could probably mirror a pretty large fraction of the text out there whose license allows you to do that. For media you have on-demand downloads and then just have a list of everything sitting in the local cache since often people aren't sure what they want anyway and will go for what's already available. This approach works for anything you download on request if you restrict sources & don't worry too much abt privacy.
@ajroach42 Another area I'd like to explore/see explored is broadcast. WiFi sends broadcast packets once, so it saves bandwidth if there's more than one listener. A lot of the congested links are point-to-point, but from reading that article medium range multipoint links don't seem uncommon. Could do both realtime broadcast for community programming and sending of requested files over PGM so any interested party can grab them.
The bandwidth of your average point to point wireless connection is pretty high, so I'm not worried about the redundant traffic being a problem from the gateway to the trafficker.
Having multiple traffickers request the same data could result in a bunch of redundant requests, especially in a situation where lots of traffickers only pass through an area occasionally.
But some smart defaults on rate limits and storage periods could minimize that, I'd imagine.
What do you think?
@ajroach42 Since I switched my phone from T-Mobile to Ting, I've come back more to this way of thinking. For example, setting my podcast app to download a butch of stuff for me on WiFi at night, rather than streaming stuff using phone data.
(in case you were wondering, $40-45 a month for my wife's phone and my own together, compared to $90+ for just my phone on T-Mobile. Really not much of a burden, for saving half a grand a year minimum)
@ifixcoinops My only concern with this is that phone storage is too limited, and so many phones are shipping without user expandable storage.
If I was really committing to the local storage game, I'd want 256+ GB of storage for my mobile device, and a more reliable and configurable system for syncing than just podcast and an RSS reader.
I could probably get some kind of sync system I was happy with if I had a few days, but I don't know of an android device with the kind of storage i need.
@ajroach42 This is why I get my phones from the past. Never gonna buy anything without an SD card slot or removable battery, just gonna wait 'til that whole ridiculous fad passes by before I get anything newer than a couple years old.
I use Foldersync on an OwnCloud instance - it's alright, so long as I don't work on the same file on my laptop and phone on the same day. If I do, I have to remember to sync in between devices.
@ifixcoinops I don’t know if it will be any good, but I bought a Gemini. My hope is to use it in much the same way I used my palm pilot or my HP LX.
Foldersync from owncloud would probably work pretty well, if I coupled it with some cron jobs to automatically fetch new content and expire some old stuff.
@joop For sure. I'm approaching this from the perspective of someone living in a rural area in the US, where high speed connectivity doesn't exist or is very expensive, because this is where I have the most experience.
But a lot of the same lessons translate over to places outside the US, for sure.