Comments on: Another RSS bandwidth reducing technique

By: Doug Gbison

Doug Gbison — Wed, 29 Dec 2004 13:44:22 +0000

Yeah, I agree that serving limited content on the first-time hit is not so bad, and I would propose to programatically include an entry explaining the issue and encouraging the user to use well-behaved RSS readers. My concern with doing this related to the potential for the last-checked and eTags timing out or being cleared from cache by the the RSS client. Is this a concern at all? Does anyone have any insight into how RSS readers work in that regard?

Thanks for bearing with me – It’s great to come up with countermeasures for abuse, but I am always very cautious as to not hinder the legitamate users in the process.

By: John Wilson

John Wilson — Thu, 23 Dec 2004 01:20:15 +0000

Yeah, I’ve thought about that, and I’m okay with it. The thing is, the very first time they get a new update, all of the old summary articles will change to full feed articles. Yes it’s a bit of a pain in the ass, but I think people can deal with it.

If an RSS aggregator wanted to be a smart ass, on the very first fetch it could pass in a If-Modified-Since header with a date like, oh, March 1, 1989 (the month and year that Tim Berners-Lee put forth the first paper outlining what would become today’s web). That would prove that the RSS reader was well behaved, as well as having a sense of humor.

By: Robert Sayre

Robert Sayre — Thu, 23 Dec 2004 00:48:15 +0000

Ok, but this means that first-time visitors will not receive full text feeds. No matter how well behaved, an aggregator cannot send those headers on the first visit. Do you really want to penalize new readers like that? I like the idea, but maybe start doing it after a pattern of abuse.

By: Doug Gbison

Doug Gbison — Tue, 21 Dec 2004 20:05:18 +0000

OK, so Ican do some manual testing of the headers myself. But does anyone have any insight into what determines how long client cache this sort of info?

By: John Wilson

John Wilson — Tue, 21 Dec 2004 19:31:04 +0000

Okay, let me rephrase that. This is all the headers I get from one of my RSS users:

Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*
Accept-encoding: gzip
Host: crazybutable.com
User-Agent: FeedOnFeeds/0.1.8 (+http://minutillo.com/steve/feedonfeeds/)

This is probably a problem with their setup, as all of my FOF 0.1.7 subscribers are sending me nice headers.

By: kellan

kellan — Tue, 21 Dec 2004 18:40:15 +0000

All versions of FoF are well behaved assuming they’ve been installed and configured correctly in part because they build on Magpie which ships with conditional GET, and gzip encoding support.

By: John Wilson

John Wilson — Tue, 21 Dec 2004 17:19:41 +0000

So you’re looking for programs to test with, if I understand you correctly. Hmm. I curled up in front of the fire with a copy of the HTTP/1.1 spec, crafted some test cases, and blasted them at the server by hand. Later I put in some logging code on my server to log what headers I’m getting from RSS clients themselves.

Alas, I don’t have a very big subscriber base to get stats from. As the only example of a “bad” feed reader I’ve found, it appears that FeedOnFeeds 0.17 is well behaved, but FeedOnFeeds 0.18 is not! For some reason FOF 0.18 isn’t sending me any nice headers.

Here’s a workaround using Sharpreader. The first time Sharpreader hits a feed, it doesn’t pass any nice headers, it just grabs it. Paste the URL of your feed in Sharpreader and hit enter, and that’s the example of a “bad” feedreader. Then subscribe to the feed using SR. Add something to the feed and manually refresh it. Bam! there’s your example of a well behaved feed reader. Delete the subscribed feed from SR and start again. (Note that you need a feed you don’t care about to test this with (because you have to keep editing and adding things to it to get Sharpreader to grab it on the second try.))

Nothing really works as well as writing your own program to manually send headers though, because then you can really test every case.

By: Doug Gbison

Doug Gbison — Tue, 21 Dec 2004 16:31:39 +0000

This is a great idea and something similar to what I was considering given the recent articles that sparked my interest is bandwidth optimization of RSS. One thing I would do in such a case is hard-code a reference to a post/article about “well behaved RSS Aggregators” and why a headline-only feed is being served, etc.

I do have a couple of questions/concerns though. What actually governs whether the proper headers will be sent back? I mean, some aggregators store everything in what amounts to a database, but others I assume will “cache” the info more like a web-cache. So what governs how long this info persists so that when the user revists (say after vacation) the If-Modified-Since and If-None-Match are again passed to retrieve the full RSS posts? Anyone know of a feature matrix of aggregators or RSS readers that support these features? Or a badly behaved RSS reader I can test with?

By: Regular Sucking Schedule

Regular Sucking Schedule — Tue, 21 Dec 2004 04:48:52 +0000

Give Poor Aggregators Less
John Wilson has a great idea about dealing with bad aggregators: give them less. He has reconfigured his copy of Word Press, blogging software, to provide tiny summaries for aggregators that fail to offer up the right HTTP headers that demonstrate the…