Hello! It is I, Lewis, with my first Tangled post. At Tangled my job involves a lot of Research & Development, by which I mean fucking around and finding out. Let me tell you about one such finding out, which turned into this new Tangled API.
Bobbin is a read-only, API-only
AppView
for Tangled. It serves the read side of the sh.tangled.*
lexicons
over XRPC, so a
client can render git repositories, issues, pulls,
comments, stars, & follows, without having to ping N
PDSes
for their data. It has no permanent storage. Everything it knows is only
ever in RAM and gets backfilled from upstream every restart.
I, along with many others, find the term "AppView" to be too abstract or perhaps even unfortunately-named. In atproto, datasets are distributed across Personal Data Servers (PDSes) that users each own, so an AppView is a single API through which one can query a dataset, in contrast to all downstream clients each maintaining a connection with every PDS. An AppView is one of the many programs in use to reduce the strain on PDSes, so that a PDS is relatively cheap and resource-efficient to run.
Let me list some problems that I find Tangled's AppView to have, such that the journey of discovery that led me to Bobbin makes sense:
- Tangled's AppView has no API; it doubles as a web client via SSR.
- It is brittle in the sense that it's not able to backfill atproto data easily.
- Its architecture from the ground up is inherently built for the kind of web program that assumes it is the source of truth of data.
- Because of all of the above, and the fact that there is some private data such as email -> user DID ties, there ought to only be one AppView for Tangled. This means that latency to Tangled is vaguely a function of distance from the server in Stockholm.
- It is hungry for RAM during a RAM crisis.
Flipping to the inverse of all of these items would yield me my ideal AppView. An API-only, easily-backfillable, easily-distributable, easy-on-RAM program.
Why "API-only"? Once it's easy to retrieve Tangled data via generic XRPC API, more programs can be built on top of it; programs such as alternative frontends, third-party clients, one's own targeted dashboards, in my case it was to enable a good CLI (though I suppose that just comes under alternative frontends). The existing Tangled AppView being its own web client means there's nothing extendable to build on, unless one is a scraping wizard; one has to figure out their own way to go grab the Tangled dataset from the world of PDSes in the wild.
While I was mulling how to go about creating such a program for fun, I have also been having a crisis about where to store state, after I've woken up in cold sweats wondering if Tranquil PDS' homegrown embedded DB could fail in new mystical ways. How could I avoid this next program having to write any durable, fallible state? Besides, AppViews are downstream of PDSes, which means the more we try and hold onto state in an AppView, the more chance there is that it'll diverge from reality. All it takes to have an AppView be wildly incorrect is missing a single record, so it is of upmost importance to have an AppView be able to reconcile what it thinks it knows with what's actually true.
Being the simpleton that I am, I think that there is no better way to make a program whose only job is to cache and serve records than to make it never touch disk in the first place. I would want it to simply backfill all of its data from scratch on every restart. In that case, reconciliation with reality is just a restart away and happens automatically! Was that feasible? What if such a backfill took a whole day every time? That would sure make deployments slow, would it anyhow be worth it?
My first tests of this concept gave promising results, I threw together a little server that could in fact backfill all of Tangled's atproto dataset in around 15 minutes, using Hydrant as its aggregator. I decided that an acceptable backfill time would be under 5 minutes or less, so that it would be feasible to switch out running instances on a blue/green deploy within a reasonable warming timeframe.
Hydrant is an aggregator for arbitrary atproto datasets that works by tailing the live firehose, backfilling every repo on the network by pulling their CARs, and serving it back over a websocket. All Bobbin has to do is open a connection, ask for cursor 0, and (re)build its index off the resulting barrage.
Massive gratitude for Dawn for having made Hydrant! Then for hearing my struggles and going and optimizing it also for fun!
Once I reached backfill in 5 minutes, I moved the goalpost for fun, that the acceptable backfill time should now be 90 seconds or less. This is around the number we are approaching today, depending on how warm the upstream Hydrant instance is; ranging from 30 seconds in best-case scenario to 20 minutes in a disaster where Hydrant is also pulling records from PDSes from scratch.
Yes, you too can spin these up & order the entirety of Tangled in 20 minutes or less, or you get a refund. (Of €0.)
There's a little conundrum that I encountered: what to have the program serve while the backfill is still warming? Do I bother having the program serve requests while aggregate/linked data is possibly incorrect? I think not. So do I have a program serving nothing in the backfill time that is possibly 20 minutes at worst? I probably could have done that, but instead we connect to Slingshot for point-lookups at least and immediately get accurate data in the meantime, even if the program itself hasn't technically seen that data yet.
Slingshot is a wonderful edge cache for atproto records & identities. Bobbin asks it for single record and identity lookups, so that a freshly-born Bobbin can answer individual queries accurately before its own index has warmed up. Slingshot is part of the microcosm project, a bundle of community-run atproto infrastructure. Thanks Fig for making it!
The program that was turning into Bobbin was able to hold all of Tangled's atproto dataset in around 200MB of RAM, which was good, but some compression led to 100MB. There is likely more I could compress that is yet to be done. "Wow, such a small amount of data for the entire social layer of a whole platform!" you remark, but on the contrary remember that this is the smallest the dataset will likely ever be from today onwards. Space-saving is of the essence.
Bobbin uses Slingshot as a record/identity resolver in real time for anything it doesn't already hold in RAM or hasn't necessarily received from Hydrant. I love Fig's work, don't get me wrong, but my lean on Slingshot instead of just writing the pings myself was a cop-out to save time; if there's one major architectural change yet to come to Bobbin, it's inlining that away such that Bobbin never has to ping Slingshot and can figure out faster ways of getting the same data.
One would be tempted to throw in an embedded instance of Hydrant to their AppView program itself, and have the serving of a particular atproto dataset be just a thin wrapper over the Hydrant data; but I want it to be easy to spin up possibly hundreds of instances of my program. Having hundreds of Hydrants all pinging PDSes all the time would be quite wasteful. One Hydrant instance can serve hundreds of Bobbins without a sweat.
Since we don't have a Hydrant/relay equivalent for Knot-related data, Bobbin proxies Knot-related requests straight through to a given knot. I think a knotstream aggregator is in order, perhaps even just a plugin to Hydrant that has it serving both. I think Bobbin and Knot Mirror should merge into one service, but that's just my opinion.
With a "stateless" AppView, we don't have to worry about databases and their woes: migrations, production massaging when things are slightly off compared to the "true" dataset, et cetera. Any time we want to change/add functionality, all we have to do is simply restart Bobbin and the operation is done. Another benefit is that Bobbin lends itself to "cloud-native deployments" as the corpos say. Tangled's own flagship instance runs on Cloudflare Containers.
All this being said, I'm not necessarily married to the idea that Bobbin never touches disk, I think that one cute improvement could be to simply dump RAM-stored-data contents as a snapshot once every hour and backfill the difference at startup; but for now I think there's headroom to continue down this path, considering Bobbin can healthily run with 200MB given to it at time of writing. Additionally, my original requirement of being easy on RAM is still fulfilled for now, but the moment it becomes too heavy is another point at which Bobbin will offload parts of itself to disk dynamically during runtime. How do I define "too heavy", you ask? I look inside my heart, which tells me that 1GB of RAM is wasteful, 10GB is unacceptable. Bobbin also must serve users, whose each API request puts more pressure on the RAM beyond just storing the Tangled dataset.
There you have it! Now there's this fun AppView with no disk storage, sub-90-second backfill, that serves API requests as a little engine that could. With it, anyone can build programs using the Tangled dataset more easily.
There is an instance of Bobbin on api.tangled.org, and hopefully you'd find it easy to host your own. Bobbin's source lives in our monorepo.