Work on Sciop Proper has slowed a bit as it is largely stabilized¹ in its pre-federation state. There are two major features, one of which is already completed, that we are working on before turning fully to federation:

• Reporting: completed in #471 - report items for a variety of concerns like validity, malicious content, rule violations, etc.
• Commenting: TODO - to be able to coordinate scrapes and discuss uploads on the site itself.

The other pieces we have been working on surrounding sciop include

• torrent-models - nice data models for working with torrents within python, as well as filling the gap where a lot of other python torrent creation/editing software doesn’t support v2.
• noob - a graph processing library that we’ll be using to implement the chains of side effects with activitypub (more on that below).
• FEP-1580 - a spec proposal for how to move objects around on activitypub. this is important for regular microblog activitypub as well as for sciop, as we will want to be able to relocate datasets and uploads between instances relatively freely (to the degree they are attached to instances, more below).
• FEP-d8c8 - A Torrent type for activitypub, which represents torrents as first-class activitypub objects and allows them to be extended within-protocol, allowing us to embed metadata that makes torrents self-describing, self-contained, and self-updating as we federate them between instances :)

There is still a bit of work to do to get these pieces up and running, particularly with noob, but we are nearly there, and will begin the work of federating sciop early next year.

Federation Plans

We plan to federate sciop using a federation overlay package that does for python² what other projects like activitypub-express and fedify have done for javascript.

In particular, we plan on writing

• a package with a set of core activitypub pydantic models
• extensions of those models to be used with sqlalchemy
• an ORM-like package that allows pydantic models to be used with oxigraph
• a set of pure function routines that implement federation and its side effects as a graph of operations
• a set of server framework wrappers that expose those routines to fastapi and litestar

As described in our federation roadmap, we plan on implementing a different kind of activitypub federation than usual: a distributed federation with FEP-c390 as a model that decouples objects and actors from instances.

Some detail that probably strays too far into the technical weeds on these …

Actually Modeling The Graph

One of the underrated achievements of ActivityPub is how it bridges the Linked Data world with the rest of the web tech world³. Collections in particular are an underrated achievement, as dealing with collections of objects with traditional RDF tools is a literal nightmare.

However the challenges of trying to jam a graph-shaped thing into a relational-shaped hole mean that for the most part we can’t really realize the full potential of what activitypub can do. The promise of building activitypub out of linked data is that we can federate any kind of object, however you can’t really store any kind of object in a relational database in any kind of coherent or performant way. This means that even minor changes in the spec require an enormous amount of labor touching all the little finicky bits of every tradweb AP server, rather than having those changes take place at the schema level, keeping model and view isolated from one another.

Part of the problem is that the alternative of just using RDF and graph databases is not exactly technically or cognitively viable either. How do you federate an unbounded graph of triples? ActivityPub gives a specific unit of graph in Objects, which is great - people think about things as discrete entities, as objects, as records; not as a big graph soup where i need to deal with the infinity of all possible properties that could be declared for something. Type annotations are great, but how can i type annotate a big blob of triples? Technically, postgres is great! wonderful! we love it! but there aren’t really comparable graph databases - they are all ancient, unmaintained, unwieldy behemoths that have the nasty habit of knocking over a server when processing even a moderately complex SPARQL query.

What to do?

We want to take a page out of bluesky’s playbook, where they can run large large services by using many single-tenant sqlite databases rather than one humongous sharded database⁴. This fits with a distributed federation model (below), where an instance is just a “delegate” that can receive and act on behalf of an account, but the account is not intrinsically tied to the instance.

To do that, we want to make the programming experience nice by making an ORM-like adapter between pydantic models and oxigraph. The python type annotation system makes a very nice analogy to triples, where an object instance - type - value maps onto a subject - predicate - object triple. From that we want to make something that looks like this

this should be considered PSEUDOCODE as an EXAMPLE

from typing import Annotated as A
from pydantigraph import GraphModel, Namespace, ObjectURI, ClassURI
from pyoxigraph import Store

Schema = Namespace("http://schema.org/")
Sciop = Namespace("http://sciop.net/")

class Person(GraphModel):
    class_uri: ClassURI = Schema.Person

    object_uri: ObjectURI
    name: A[str, Schema.name]

# create
with Store('my-data.ox') as store:
    me = Person(
      object_uri = Sciop / "accounts" / "me"
      name = "Jonny"
    )
    store.add(me)

# query
with Store('my-data.ox') as store:
    me = store.select(Person).where(name == "Jonny")

# dump to rdf
me.to_ttl()

roughly…

@prefix schema: <http://schema.org/> .

<http://sciop.net/accounts/me> a schema:Person ;
    schema:name "Jonny" .

Assuming additional methods to query additional properties, dynamic class constructors, and so on, this should let us keep the sensible modeling of relational/object-oriented world while being able to accommodate arbitrary data in the graph.

For SciOp, we want to implement the graph data as an “underlay” to the data of the sciop site - where the sciop instance will keep copies of each account’s data in a separate pyoxigraph database and create digested views against that data in its relational database. This is to let us make a gradual transition to a graph database, as well as gauge the performance consequences of this experiment so we don’t rewrite the whole backend and find out that it’s much worse. Our goal is to make it so that most of what we do with sciop is not vertically integrated and can only be used with sciop, but to make sure that we are making as much reusable technology as possible for the benefit of the broader fediverse.

Noob + Side Effects

Within activitypub, given activity will imply a number of side effects, both “officially” in the protocol and “unofficially” in how a practical implementation has to work. So e.g. when receiving a Create[Note] activity, one might have to go and fetch the replies for the post, then go and fetch the profile for the accounts that posted it, create notifications, and so on.

Noob will allow us to model these graphs of side effects in a declarative way that allows them to be run flexibly, scaling from very small instances with only a few CPUs to very large instances that can distribute those processes over compute clusters.

For (a simplified) example, we could model a side effect where we fetch a post and and download its attachments as a two-step noob pipeline like this that fetches the post, fetches its attachments, and returns the post.

noob_id: ex-fetch-post

input:
  status_uri: str

nodes:
  fetch_post:
    type: example.fetch_post
    depends:
      - uri: input.status_uri
  attachments:
    type: example.fetch_attachments
    depends:
      - post: fetch_post.post
  return:
    type: return
    depends: 
      - post: fetch_post.post

This always happens whenever we fetch a post, and we might fetch posts in lots of different cases. Noob then lets us compose pipelines so that, e.g. if we were to fetch a post as part of some other side effect chain, we could do so by reusing the pipeline:

noob_id: ex-expand-timeline

input:
  collection_uri: str

nodes:
  iterate_collection:
    type: example.iterate_collection
    depends:
      - uri: input.collection_uri
  object_map:
    type: map
    depends:
      - iterate_collection.object_uri
  fetch_post:
    type: tube
    params:
      - id: ex-fetch-post
  something_else:
    type: idk.something.else
    depends:
      - post: fetch_post.post

where we map a collection of object uris into the ex-fetch-post timeline, fetch them using that pipeline, and then use its results in the parent pipeline.

Modeling activitypub side effects as a declarative graph like this should help us keep the code very general, but we are also hoping that this paves the way to being able to federate an activity’s behavior as well as the activity itself. This is useful for a few reasons:

• One is that it makes activities self-describing - if we initially don’t know how to handle an object, we can consult its “recommended” or “default” processing chain for it, after reviewing the code of course.⁵
• It is a straightforward path for giving objects capabilities: How does an activitypub instance know how to “Like” a post? Currently by convention, but an object could provide a Like capability that refers to a pipeline of actions, and objects could provide any number of capabilities that are relevant for it.
• We eventually want to bridge federated data with federated compute - but that’s a longer term goal that’s a bit out of scope for this blog post…

Distributed Federation

I won’t repeat the relevant specs or our roadmap here, but briefly - the model of federation we will be building towards is one that decouples actor identity and objects from specific instances.

Much like how bittorrent trackers are just lightweight databases of references to the data referred to by the torrent, a sciop instance will be a lightweight database of copies and references to activitypub objects - the “always-on” server that can store and forward events when your local peer might be offline, but if you want to pick up and move your data to another instance, or host it on multiple instances simultaneously, you’re more than welcome to.

The basic outline that’s emerging is that a given identity starts as a keypair, but that identity can create one-way “derived” identities and “delegate” certain permissions for those identities to a set of instances. So, I’m little old me, and I create an ActorDelegation that says “sciop.net is allowed to do x y and z on my behalf.” The Delegation object contains a signature that lets it be validated against the public key that serves as my identity, and objects created by sciop.net are signed by this delegated identity.

This creates a chain from me -> DelegatedIdentity -> sciop that can be validated by recipients, and in the case that I want to revoke those permissions from a potentially hostile instance, allows me to publish a DelegateRevocation that severs the connection from me to DelegatedIdentity. The instance doesn’t get to hold onto my “main” private key so I retain control over the root identity, but it can act on my behalf as if it did have the key for those specified activities. Lots left to work out here re: key management, but that’s the draft so far.

We’ll also be working on a protocol extension for ActivityPub collections that makes them function like git-like merkle trees for efficient updating and change propagation, but still too early to say anything about that.

A few of the main desiderata we have here are:

• Identity and all its data (uploads, comments, etc.) can be moved and restored in case an instance suddenly goes down
• Identity can be present on many instances simultaneously
• Identities remain durable and relevant for webs of trust (as opposed to most ‘censorship resilient’ protocols with fascist undertones, where identity is cheap and abuse is rampant), but can still be deniable for external observers - balancing moderability with autonomy.
• “ownership” of objects can be shared via “group-like” delegated identities and forked with clear change in ownership

Summary

In summary, we’ve gotten started on, and will continue to work on a model of federation where

• We can use a wide variety of objects with minimal implementation labor
• We can extend our programs to accommodate new objects by federating action chains and capabilities
• Identity and data are delocalized from instances, but use instances as flexible network actors that have a level of responsibility more than passthrough relays, but less than traditional AP instances.

We call this “federated p2p” or “hybrid federated/p2p systems” that blend an always-on federated backbone that coordinates swarms of p2p actors.

Towards the mission of making archives resilient to the pressures of those with vested interests in rewriting history and keeping people confused on a sea of filtered and always-changing information.

You are of course welcome to take part in any of this work! start chatting on the issue boards of any of these projects, @ me on fedi, or email the contact address at the bottom of https://sciop.net <3

Appendix: Even More Sneak Peeks

In the even longer future we are working on merging these projects with work elsewhere on wikilike and document-like systems. A currently WIP project (that I’m unsure how much detail I can write about yet) in our lab is working to federate packs of pages between semantic mediawiki instances. On SMW instances, “wiki pages” consist of both traditional wiki-markup human-readable text, but also data schemas and forms and templates for creating and displaying structured data.

We are attempting to design sciop’s systems to eventually accommodate wikilike editing, with shared ownership of objects, edit histories, forking histories, etc.

Mediawiki is useful as a matured wiki framework, but it can be largely seen as a frontend to editing the underlying federable data. We imagine datasets and uploads being accompanied by more information needed to contextualize it that might come from multiple distinct sources: put one way, a wiki overlay to the datasets in a sciop instance, the other way, a wiki that can support a bunch of bulk data beneath it.

A bridge between MediaWiki and ActivityPub could look as simple as federating a bunch of Article objects around, but with SMW we should be able to create and structure arbitrary object types. So a small group like a lab or an archive can run a wiki, a sciop instance, and be able to organize their internal work on the wiki and then bridge work that should be shared more broadly out to a network of activitypub instances and p2p peers for hosting the bulk data: making “open data” an actually-attainable goal without needing to pay AWS a brazillionty dollars to host enormous centralized archives.

With federable processing pipelines and action chains, we imagine being able to compute over the data shared via torrent, create derived objects, populate “dashboards” on small-scale sciop/mediawiki instances, and bridge the domains of bulk data, communication, computation, and documentation..

In addition to the sciop blog existing, now it is embedded on the frontpage of sciop dot net, a routine feature for a website that has most of its core components relatively squared away.

The Pull Request, first initiated by transorsmth some 3 months ago: https://codeberg.org/Safeguarding/sciop/pulls/404

As with all new development on sciop, it is intended to be configurable by other instances aside from sciop dot net, when such things exist, and can accept multiple atom feeds for e.g. different topics, covering different projects, with different authors, and so on.

So without further ado, a screenshot of text of the other two posts that are also on this website:

A Hint Of Future Plugins

Since this is a very simple feature, and we don’t have much in the way of developer docs at the moment, it seems a good enough example to show the pattern of how most things are implemented in sciop.

It consists of

• A config model - UpdateFeedsConfig
• Some database models - models/atom.py
• A background service to update feeds - services/atom.py
• A job wrapper that links the config to the scheduler - update_atom_feeds
• An API endpoint to generate an HTML partial - /partials/whatsnew, and
• Some templates - templates/macros/whats-new.html

Pretty much “website things.”

These components (and maybe a handful more) are destined to become the basic of a plugin system, where most current functionality is moved into plugins, and most new functionality is implemented as plugins. There is still a bit of work do be done to get us there, mostly in refactoring the existing templates to support plugins modifying and adding components to them¹, and a bit of metaprogramming to wrap API endpoints and handle model migrations, but we have already been writing sciop in separable services with this in mind, so once that is done then that shall be the pattern going forward. As we become a “fediverse app,” the last thing we want to be is “a monolithic fediverse app,” and want to start with a very pluggable design rather than trying to back flexibility into the system later.

For embedding within institutions with rich histories of work patterns and homemade infrastructure, this kind of flexibility and ease of writing custom behavior is essential, and the same is true for supporting different p2p communities. One of the first plugins that we plan on writing after the plugin system lands is integrating external tracker software, where rather than relying on other public trackers, a sciop instance may want to provide its own, embed the tracker link in uploads, generate custom URLs for private torrents to track upload stats, and do all the other fun things one might want from a tracker. We also want to experiment with all the non- or sparsely-implemented FEPs floating around, and will be writing our federation layer as a generic fastapi² overlay that can be useful outside sciop.

Example

To configure feeds, stick something like this in your sciop.yaml

services:
  update_feeds:
    enabled: true
    interval: 10
    feeds:
      - name: sciop
        url: https://blog.sciop.net/feed.xml

with a URL pointing to some atom feed. That’s all that someone using the software would need to do!

That corresponds to boilerplate pydantic models:

class UpdateFeed(BaseModel):
    """
    A single feed to use for "whats new" updates
    """

    url: AnyHttpUrl
    """URL of an Atom feed"""
    name: str
    """
    Short, taglike name to use when displaying posts from this feed.
    """


class UpdateFeedsConfig(JobConfig):
    """
    A set of feeds and service config for updating them
    """

    interval: int = 30
    """
    Frequency (in minutes) to check feed for updates.
    
    If the feed has not been updated in that time, no changes are made to local objects.
    """
    feeds: list[UpdateFeed] | None = None
    """
    Potentially multiple feeds to pull updates from.
    If `feeds` is not provided, service will be disabled.
    """
    max_n_posts: int = 10
    """
    Only keep the last n posts from each configured feed.
    """

Internally, that config is used by function wrapper that schedules the service:

@scheduler.interval(
    minutes=get_config().services.update_feeds.interval,
    enabled=bool(
        get_config().services.update_feeds.enabled and 
        get_config().services.update_feeds.feeds
    ),
)
async def update_atom_feeds() -> None:
    """
    Put documentation in the real thing, but this is merely a simulacrum
    """
    await services.update_feeds()

and then the scheduler will launch a task every interval minutes to update the feed.

Simple as pie. the rest are details.

Appendix - Build and Deploy a Static Site with Codeberg/Forgejo using Webhooks

If you are unsure how one might go about making an atom feed -

We write this very blog with jekyll, generate the feed with jekyll-feed, and use a webhook to trigger a rebuild on push on one of our machines. Total time from push to deployment is ~8 seconds or so.

I wasn’t able to find another blogpost describing this exact process, since the forgejo webhook documentation is a little sparse. So if it saves anyone time…

• generate a secret with openssl rand -hex 32
• configure a webhook like this:

[
  {
    "id": "build",
    "execute-command" : "/path/to/your/build/command.sh",
    "command-working-directory": "/wherever/your/blog/repo/is",
    "response-message": "Deployed!",
    "trigger-rule": {
      "and": [
        {
          "match": {
            "type": "payload-hmac-sha256",
            "secret": "{YOUR_SECRET}",
            "parameter": {
              "source": "header",
              "name": "X-Forgejo-Signature"
            }
          }
        },
        {
          "match": {
            "type": "value",
            "value": "refs/heads/main",
            "parameter": {
              "source": "payload",
              "name": "ref"
            }
          }
        }
      ]
    }
  }
]

• for a build script /path/to/your/build/command.sh like this³:

#!/bin/bash

git pull

/home/webhook/.rbenv/shims/bundle install
/home/webhook/.rbenv/shims/bundle exec jekyll build -d /some/web/directory

• configure nginx to forward the URL to the webhook service⁴

server {
  listen 443 ssl;
  server_name your.url.here;

  root /some/web/directory;

  location / {
    try_files $uri $uri/ $uri.html =404;
  }

  location /hooks/ {
    proxy_pass http://localhost:9000/hooks/;
  }
  
  # other stuff for ssl and logging and ratelimiting and etc.
}

• Configure a webhook for your repository,
  • POSTing JSON
  • to your hooks url
  • with the secret generated above,
  • on push events
  • with a branch filter for main.

and ka blammo. push to deploy the blog, and sciop will catch up when it runs its next update.

Now you’re talking to online from your personal computer.

BitTorrent is great, but what if you, graceful and mighty but in adverse network conditions, cannot run a torrent client?

Sciop has just the feature for you! As of today¹ you can now add webseeds straight from the website! This means that we can bridge datasets archived in any number of traditional archives, efficiently use existing resources, and bring a new category of institutionalized peers into the swarm.

This work builds on one of our attempts at revitalizing the broader bittorrent tooling ecosystem, where after integrating torrent-models² we can now safely³ and transparently make serverside edits to uploaded torrents.⁴

Stick around after the ad break to learn how webseeds could enrich your life and make you more complete as a person.

How It Works

Webseeds are very simple and require no special behavior on the part of the HTTP server aside from supporting range requests. Where normally the bittorrent client would request pieces from other peers, the client instead requests ranges from the http server. Webseeds can be used along with normal peer swarms, and multiple webseeds can be used at once.

Webseed URLs are constructed like this for multi-file torrents:

{webseed_url}/{torrent_name}/{path}

and like this for single files:

{webseed_url}/{torrent_name}

where torrent_name is the value of the name field in the torrent’s infodict (not the filename). The name field is the name that’s shown in your torrent client and the folder that is created when you download a torrent.⁵

So for a torrent named tentacoli with a file named track-07.mp3, if an http server was hosting that file at https://example.com/desktop/tentacoli/track-7.mp3 then the webseed url would be https://example.com/desktop/.

Adding A Webseed

If you have hosted some data in a torrent somewhere, or are aware of some alternate source for it, you can add it to the torrent from the web interface⁶

When you are logged in to sciop, on an upload page, you will see a button to add a webseed.

If you click it, you can then type in a url

And once you submit it, it will be added to the validation queue

The server will now attempt to validate that the URL works as a webseed by requesting some subset of the pieces and checking them against the piece hashes in the torrent.⁷

If the webseed validates, if the account that added it has the upload permission scope, then it will be added to the torrent! If the account does not have the upload scope, it will enter the moderation queue and only added after manual review by a moderator.

The account that uploaded the torrent retains control of the torrent, so they are able to remove webseeds if they object to them for some reason, and webseeds from accounts without upload permissions are not displayed publicly until they are approved to avoid them being a vandalism vector.

This follows the general soft security moderation pattern of sciop - rather than gatekeeping at the level of account creation, any account can propose a change to be made, but only known accounts automatically have those changes take effect. This allows us to keep the site trustworthy while allowing everyone to participate.

Why This Is Cool

What’s the big deal, it’s just adding a url to a list in a torrent? Like objects in mirrors, webseeds in blogposts are cooler than they appear.

We are not aware of any other trackers that allow post-hoc addition of validated webseeds in a participatory way by people that aren’t the original uploader, but please let us know if we missed something.

Institutional Pipes

As much as we would love for everyone to experience the joy of a torrent swarm, many would-be collaborators have been unable to contribute because their resources are housed in some setting where bittorrent traffic is blocked, or it would otherwise be impossible for them to run a torrent client. This has been a common refrain from our colleagues at academic institutions or archives.

We are not protocol purists, and don’t use bittorrent for bittorrent’s sake, but instead use it to bring all available resources to bear from cute little raspis seeding from a flash drive to enormous Swedish Seedboxes. What are HTTP servers and S3 buckets but very big peers?

Adding webseeds via the web UI provides a new way for those in constrained circumstances to join the swarm by getting the data⁸ and rehosting it on a traditional HTTP server. We also hope to incentivize joining the swarm by providing a little pro-social gamification - having your URL listed as a webseed on sciop lets others know that you care about the preservation of cultural memory.

Bridging Archives

Archives have a problem: there are more than one of them. This is a problem for preserving threatened data, where different archival groups have been patching together dataset storage on zenodo, google drive, globus, and so on; as well as for more conventional archiving, where researchers have to search or upload their work to multiple archives that are mutually incompatible.

Aggregation and indexing overlays address this problem to some degree by collecting potentially multiple links to the potentially multiple places a dataset might be hosted, but they don’t make use of those multiple hosts, allowing the redundancy of resources to improve availability, resilience, and performance. Even if a dataset is hosted in multiple places, I still have to download it from only one of them, so the fastest and largest archives end up shouldering all the costs and smaller archives don’t have a great “path to relevance” - why would i store my data on the slow, unpopular archive?

Treating a torrent as a verifiable shorthand for a dataset and then allowing anyone to add a webseed means that now it is possible to make use of all the available resources for a given dataset - if i have previously archived something to zenodo, or on my special S3 bucket server⁹, and I see someone uploaded it to sciop, I can add in my copy as an additional source. When multiple webseeds are present, bandwidth can be shared between each of them (and the rest of the swarm), decreasing the burden on each individual host.

This allows sciop to take indexing overlays one step further - rather than just aggregating the metadata from multiple hosts, we can aggregate the hosts themselves.

Division of Labor - Scrapers & Seeds

Sciop as a project is about coordinating people with varying resources, expertise, and commitment towards the same goal - it should be possible for someone to wander in off the street¹⁰ and contribute, whether that means spending 5 minutes improving the docs or 5 months scraping alongside us.

We have observed a natural division of labor emerge, where people often fall into one or a few basic roles based on expertise or interest. One of those divisions has been between scrapers and seeders - where some people love to do the work of scraping and downloading, but don’t have the resources to actually store and upload it; and others don’t enjoy scraping but have plenty of spare storage and bandwith.

The pattern of

• low-resource scrapers downloading, hashing, and discarding data
• uploading a torrent with a webseed, and
• other seeders bootstrapping the swarm off the webseed

has proven to be ridiculously effective.

This provides a way for people without seeding resources to effectively call down a swarm of seeders onto an at-risk dataset, who then automatically¹¹ and collaboratively back it up. This process is much more respectful to the hosting servers than everyone scraping their own copy would be, as in the ideal case the webseed needs to only upload the full dataset twice,¹² and then all future downloads have bandwidth supplemented by the swarm. Scraping is additionally deduplicated by our quest system and scraping tools that turn web archiving into an all-play activity, and will be the subject of a future blog post :).

Adding the ability to add webseeds after the fact extends that by being able to adapt in the case that the dataset moves, as well as opens up new opportunities for labor division, where scouts aware of copies of data being held in other places e.g. by other archival groups can do the curation work of adding those to the swarm.

TODO

There are a number of obvious expansion points we’ll be pursuing

• The multiple-use of the name field for display and as part of the url in a webseed is a big pain in the ass. this is the reason that, e.g. when downloading the TIF archive of the NMAAHC, one ends up with a million torrents with the same name.¹³ This also poses a problem when the relevant files are not all stored under one subdirectory, or have urls for files with any other naming structure but that of the torrent. We want to create a plugin for sciop that allows you to use a sciop URL as a webseed-by-proxy that then redirects to the appropriate URL on the webseed server, and in the future we’ll be working on our own client that escapes some of the stagnation of current clients and decouples the name field.
• When a webseed is added on the server, it is not added to clients that have already downloaded the torrent. We’ll be writing a general sync function for sciop-cli that resolves this and other torrent mutability problems at the client level, updating existing torrents with new metadata, replacing torrents that have been superceded by a repack, and so on.
• Validating files in a torrent against a webseed url is almost the same operation as creating torrents from a url. We want to support that for, e.g., using spare server resources to create torrents when scraping resources are thin, as well as for “importing” datasets from other archival systems. This also opens up interesting possibilities for hybrid http/bittorrent-backed web archives for, say, replaywebpage from webrecorder, but more information on that is TBD
• We’ll also be adding stats to account pages to encourage adding webseeds, it’s good to be able to brag about doing good things!

If you’d like to get involved with these or any other sciop projects, you are more than welcome to hop on the issues, or contact me or anyone else in SRC to ask to be added to our contributor chat <3

Appendix on Malicious Use

What about privacy? isn’t making people ping a server a whole attack vector?

Adding arbitrary webseeds is no more of an information leak than being able to add arbitrary peers to a swarm. Anyone who wanted to surveil the swarm could do so far more easily by simply joining it or scraping trackers.

Sciop validates a random subset of pieces from any added webseeds before adding them into torrents, and since the selected pieces are random, it would be relatively hard to fake hosting some tiny subset of the data and yield zero reward. If a webseed were to try and serve copies of the data spiked with malware, torrent clients would reject it since it doesn’t match the piece hash and ban the webseed IP.

However if there is some security or privacy risk that we failed to consider, please submit an issue or contact us privately at contact (at) safeguar (dot) de if raising an issue could compromise sciop’s security.

This post isn’t really about anything except the blog existing, so you should probably go to a different one