Skip to content

RSS and Atom readers

Each saved feed provides both Open RSS feed and Open Atom feed, with copy buttons beside them. Both formats read the same persisted items and never trigger scraping. Their URLs end in .xml and .atom respectively. Choose whichever your reader supports.

The formats share one private read token. Reset feed links revokes both old URLs. The replacement Atom self link changes, while its feed ID and item IDs remain stable. A new recipe import creates a separate feed identity and separate reader URLs.

Each story publishes its list-page description and, when full article content is enabled, the fetched article. RSS keeps the article in description, so no reader loses content it already receives, and repeats it in content:encoded for readers that prefer that field. Atom carries the list-page description as summary and the article as content, so a reader's list view can show the short form. The extracted image is published as media:content in RSS and as an enclosure link in Atom, which is where a reader building a card or grid layout looks for a thumbnail; it is also shown inline, once: it is not added again when the list-page description already shows that image or the article body already contains it.

The RSS channel carries atom:link rel="self" for rediscovery, ttl derived from the feed's refresh interval, lastBuildDate from the last time the feed's published output changed, and generator, which names RSS Workshop without a version so upgrading does not change every feed. Every one of these comes from stored values: serialization never reads the clock, and a refresh that changes no story, including one where the source answers 304, keeps producing identical bytes, so a polling reader keeps receiving 304.

Atom responses use application/atom+xml; charset=utf-8, the Atom 1.0 namespace, absolute alternate/self links, escaped HTML content, and content-derived ETags. Conditional requests support If-None-Match, including weak tags. RSS and Atom have separate ETags. New feeds return HTTP 503 with Retry-After: 60 until their first successful refresh; failed or empty extraction retains the last successful content.

Atom entry IDs use the same stored GUIDs as RSS. Publication dates remain the saved publication/first-seen values. Entry updated is when that story's title, link, description, image or fetched article last changed, and feed updated is the latest of the feed's own change time and its entries. Stories saved before this was recorded keep the last-seen time they were already published with until they next change. These are stored timestamps, so neither reader requests nor refreshes that change nothing alter output bytes. The feed title is used as the fallback feed author, since recipes do not yet extract authors.

Reader responses also carry Last-Modified from the same change time and honour If-Modified-Since, for readers and caches that do not send If-None-Match. When a request carries both validators the ETag decides, because it hashes the bytes while Last-Modified has only one-second resolution. RSS and Atom share one change time, so a change visible in only one format, such as a list-page teaser that changes while the article body stays the same (Atom publishes the teaser as summary, RSS does not), still updates both. Changing PUBLIC_BASE_URL changes the self links and ETags but not Last-Modified, so a reader that sends only If-Modified-Since keeps its copy until the feed next changes. An upgrade that changes how saved stories are written out works the same way: the bytes and ETag change at the first request afterwards, but Last-Modified and Atom updated do not, and the changelog says when a release does this. Recipe files are covered by the export/import guide. The Atom representation follows RFC 4287.

Full article content

By default a story's description is whatever the list page shows, which is often a short teaser. Full article content in the feed editor follows each story's own link, extracts the article body from that page with a CSS or XPath selector (CSS for a feed source), and publishes that instead. The selector runs against the article page, so XPath here is absolute (//div[@class="article-body"]) rather than relative like the per-field selectors. It follows the feed's selector type, and Convert to XPath in the visual selector converts it with the rest of the recipe, unless it lists alternatives (.entry-content, article), which have to be written as one XPath by hand; after changing the type by hand, rewrite it. A CSS selector left there, such as .entry-content, would select the whole article page as XPath, so Preview items warns about it and saving it is refused.

Article pages are fetched as plain HTML through the same guarded fetcher as list pages, with the same destination checks, redirect validation, encoding detection and size limits. Rendering them with Chromium or FlareSolverr is an explicit per-feed option, because a feed has many article pages and the browser pool is shared with every other feed. A feed that renders them fetches them one at a time, so it holds a single Chromium slot or FlareSolverr slot and leaves any others free for other feeds; a deadline can then leave some of a large backlog for the next refresh.

Each refresh fetches at most 10 article pages, so a long feed fills in over several refreshes rather than in one burst; stories without a stored article come first. A story is fetched once and then left alone, except within the first hour after it is discovered, when it is re-fetched so a source that finishes an article shortly after listing it is picked up. An article that cannot be fetched or whose selector matches nothing keeps the list-page description and records a warning in diagnostics; it never fails the refresh or removes the story.

A failed article does not keep its place at the head of that queue. Stories whose article has not been tried yet go first, so articles that always fail, such as paywalled or removed pages or a template the selector does not match, cannot take all 10 requests of every refresh from the stories behind them. A story then waits before its article is tried again when the failure was its own page's: the page answered with an HTTP error status, the article could not be extracted, for example because the selector matched nothing or content cleanup emptied the body, or the article came back empty, which reports no warning. It waits an hour after the first such failure and twice as long after each further one, up to 24 hours. In its first hour a story does not wait: it is tried at each refresh, after the stories not yet tried, and those tries do not lengthen the wait. A failure that says nothing about the story never makes it wait: a connection that failed, the refresh's deadline, or Chromium or FlareSolverr being unavailable. Such a story is tried again at the next refresh, after the stories not yet tried. A list page that shows more stories than MAX_ITEMS keeps has a third kind: the oldest are removed by retention in the refresh that fetched their articles, and are listed without one again at the next. Those articles are fetched again as before, but only after the failed articles that are due, so they cannot keep one from its turn.

While stories wait, each refresh that reads the list page counts them in its diagnostics in one line, Articles waiting to be tried again after failing or coming back empty: N, which is the run's first warning; the reason is the warning in the run that tried the article. The count covers every listed story in a wait, including ones MAX_ITEMS retention does not keep, so it can be larger than the number of stories the feed stores. Refresh and resuming a paused feed end the wait, so those articles take their turn again the next time the list page is read, after the stories not yet tried, and one that fails again waits twice as long as before. A source that answers 304 Not Modified lists no stories, so behind one that is when the page next changes. Saving the feed in the editor forgets the failures altogether, because an edit may have fixed the selector, and requests the list page in full, so the next refresh tries those stories again. All of this is kept in memory, so a restart tries every story again too.

A source that answers 304 Not Modified lists no stories, and so no articles to fetch. While a refresh leaves stories without an article for later, because of that limit or because its deadline ran out, and leaves more of the listed stories with an article than before, the next refresh therefore requests the list page without the source's ETag and Last-Modified validators. Conditional requests resume after a refresh that reaches every listed story still without an article, even if some of those articles fail, or after one that adds none, for example because the remaining articles keep failing. Stories whose articles failed are tried again when the list page changes, once any wait described above is over, or after a restart. A story in such a wait is not left for later in this sense, since the next refresh would not try it either. Nor is a listed story that MAX_ITEMS retention does not keep: it is never counted as left for later. This is kept in memory, so after each start every full-content feed requests its list page once without validators. Only Static HTTP and Auto send validators.

Requesting an unchanged list page in full publishes nothing new by itself, unless the page lists more stories without a date than MAX_ITEMS keeps. Those are dated when first seen, so each full request saves again, as the newest, the ones retention removed, in place of others, and the feed changes. A source without validators does this at every refresh; behind a 304 it happens only while a backlog drains and once after each start. Otherwise only a stored article changes the feed.

Fetched bodies are stored separately from the list-page description. Re-extracting the list page every refresh therefore keeps updating a preview image published after the story went live, without discarding an article body already retrieved. The current image is published with the article body, so a late image still reaches the reader.

Preview items fetches the first 3 articles only. It runs while you wait and must not fan out a request for every story on the page, so a preview shows that the selector works rather than building the whole feed. Those previewed stories display the fetched article body, labelled, so a selector that matches the wrong block — a navigation bar, a sidebar, a cookie notice — is visible before you save.

Changing the article selector or its attribute, or clearing it to turn the feature off, discards the article bodies already stored for that feed in the same save. Later refreshes fetch them again with the new selector, or leave the list-page description in place. Nothing else about the stories changes: their identity, publication dates and list-page descriptions are untouched, so readers see no duplicates. Changing content cleanup rules discards fewer bodies, as described there.

Content cleanup

A description or article selector sometimes cannot avoid a share bar, a newsletter box or a list of related stories inside the block it selects. The sanitizer keeps such blocks: it removes their classes, scripts and styles, not their text and links. Content cleanup in the feed editor takes selectors, one per line, for parts to leave out of each description and article body before they are sanitized.

Rules use the feed's selector type, CSS for a feed source, and match only inside the selected block, which itself is never removed, so they cannot reach the rest of the page. They see the page's own class and id attributes. The sanitizer later removes class, so a rule is the only way to drop a block by its class; it keeps id. CSS rules are ordinary selectors, such as .share, aside.related or p:containsOwn("appeared first on"): :contains() and :containsOwn() ignore case, and :matches() takes a regular expression. XPath rules start with ./, such as .//div[@class='share'] or .//p[contains(., 'appeared first on')], and can also remove text, as in .//p/text()[contains(., 'Sponsored')]. A rule such as .share or . would select the block itself under XPath, so it is refused; after switching the selector type by hand, rewrite the rules. Convert to XPath in the visual selector converts them with the rest of the recipe, or converts nothing when a rule has no XPath form it supports, as :contains(), :containsOwn() and :matches() do not. Rules run in order, each on what the earlier ones left. A feed has at most 20 rules of at most 1,000 characters each, counted in the editor's 64 KiB limit, and an invalid rule is refused by its number when saving or previewing. A rule that the XPath library fails on, like any other selector, fails the refresh with selector evaluation failed on the cleanup rule 2 selector, naming the rule only by its number, and saved stories are kept. On an article page it fails only that article, as a failed article selector does: the refresh succeeds, the story keeps its list-page description, and Diagnostics shows the same message as a warning for that article.

Rules do not apply to a description or article read from an attribute, whether one named in the editor or one an XPath selector such as //meta/@content returns. They do not change a web page's story image either: automatic detection reads the card as the page has it, so an image removed from the description is shown above it instead; use the Image selector to choose another. A feed source has no Image selector, so it looks for an image in what its rules leave, and a full post they empty gives way to the entry's summary; see feed sources. A description that the rules leave with nothing to show, no text and no image, is dropped, leaving only the story's image if it has one, with the warning cleanup removed the whole description. An article body they leave empty fails as article body is empty after cleanup, and the story keeps its list-page description. A body that showed nothing before cleanup, such as a page holding only a video, is stored as it always was. Story filters match the cleaned description at each refresh. Removing already-saved stories that do not match is different: it judges each description as stored, which rules changed in the same edit have not cleaned. Preview items shows the cleaned descriptions and the first 3 cleaned articles.

On a feed without full article content, or one with an Article attribute, saving changed rules publishes nothing by itself. Like any recipe edit it makes the next refresh fetch the list page in full, and that refresh cleans the descriptions of the stories the page still lists. For a feed whose full article content has no Article attribute, the save itself discards the stored article bodies of the stories its latest stored refresh listed, in the same transaction, and readers see their list-page descriptions until later refreshes fetch the articles again, 10 at a time, even from a source that answers 304 Not Modified. A refresh lists the stories it stores, those the story filters keep, and a 304 lists the same stories as the refresh before it. Other stories keep their description and article as they were, older ones and those the filters now leave out alike, because they are not listed and could not be fetched again, and stored bodies have lost the classes the rules match. When the latest refresh that read the page kept no story at all, or found a feed source's feed with no entries, the save therefore discards nothing. On a paused feed, or one whose page stopped listing stories long ago, some of those listed stories may have left the page: they lose their article for good.

Rules are stored in the recipe as remove and travel with recipe exports. Releases before 1.3 cannot open a database that 1.3 has upgraded, so none of them can publish a feed without its rules, and they refuse a recipe file that has any.

Subscribe to every feed at once

Export OPML downloads rss-workshop.opml, an OPML 2.0 subscription list naming every feed in the library. Import it into your reader to subscribe to all of them in one step, instead of copying each URL by hand. Add ?format=atom to /api/opml for a list of Atom links; the default advertises the RSS ones, which more readers accept.

Each entry carries the feed title, its reader link as xmlUrl, and the source page as htmlUrl, so a reader can link back to the site the feed was built from. A feed source has no htmlUrl: its address is the feed it reads, which a reader would open as raw XML. Paused feeds are included, because their links keep serving the stories already saved. A feed that has not yet refreshed successfully is listed, but returns 503 until it does.

The export covers up to 1,000 feeds. A larger library is refused rather than truncated, so the file never silently omits a subscription; the recipe export has the same limit.

An OPML file lists the private read tokens for every feed at once, so it is exactly as sensitive as the reader links themselves. The export requires an active admin session, is never cached, and should be treated like a password file. Reset feed links revokes a feed's URLs; export again afterwards to refresh the list.