I admit that I promised a working prototype in my last update but so much has changed in what I see this project becoming I can not even say I am ready to start throwing down some code. What has changed?

The Aggregator Model is Flawed

So far aggregators were the tangible programs used to collect, aggregate (duh), and display your chosen feeds. What if you never see the aggregator. In the new model the aggregator is actually what goes on behind the scenes. It is what takes the feeds, digests them, and gives you back just items weighted, prioritized, catagorized, filtered, whatever it may be. The traditional aggregator is now really just an RSS viewer. The viewer is literally a dumb client that takes the aggregator’s output and presents it in a consistent, usable manner.

I’m not going to open up a can of worms and try to establish what the best visualization method is, but in the new model, your aggregator can connect to whatever platform you are on. A viewer for your home computer, work laptop, cell phone, television, a viewer logged on to AIM/Jabber/Etc (hello, Feed Crier) whatever you want.

The New Aggregator

This new aggregator just takes in feed items and puts them through the metrics I have already talked about. The magic is when the aggregator assigns scores to each items based on notability (not the offical word I plan on calling these metrics). One way could be just outputting one big RSS feed with aggregator generated metadata attached to each item. Another could be to just stick them into a database, both would work equally well. In both cases, whatever viewer you are using has access to exactly the same data.

Now while I am forgetting some details, this model attempts to take synchronization problems out of the picture, and eliminate any sort of platform dependency needed to read your feed items. So to summarize, we are talking about something similar to a client/server model for RSS delivery.

1. How often is the feed updated?
Some feeds are updated twice a week, some dozens of times daily. Feeds will be distinguished between low volume and high volume feeds.

2. How often is each item clicked on in the feed?
This aggregator will use average click-through rates on each feed item to establish its importance. Both high and low volume feeds will equally be effected by how often an interesting feed item is presented.

3. How popular is the link on the net?
By comparing links with sites like del.icio.us, digg, and Technorati an objective gauge of an item’s importance can be determined. Also links that are initially found to be noise can in turn become more important as they are picked up by these popular services, meaning feed items are not static but can rise to the top over time.

4. Do any of my other feeds link to it?
Obviously if an item overlaps in two or more of your feeds, its importance certainly rises. A sharing of interest creates an increase of importance.

5. Does the item share your interests?
With white list settings to include keywords, tags (assuming the feed supports them), trusted users on the popular linking sites above, and URLs, the aggregator has a better idea of what you will find interesting.

These five concepts will be the underlying rules that will govern how this aggregator works. Also each of these concepts are merely an overview and each have some more sub concepts. Some other interesting concepts include item decay where an items importance will decline over time, a vacation mode when your feeds are not checked for an extended length (severely limiting item decay), and possibly a sliding scale of importance where each one of the five core concepts are weighted differently depending on how much they correlate to your browsing needs.

I hope the next update will not come in another year, but I plan the next update to be at least a very early working prototype implementing one or two of the core concepts.

RSS Aggregators suck, literally, and figuratively. They are simple pull clients that suck data from remote webservers. But they are also simple dumb clients that just suck data from remote web servers. Why can’t your aggregator be smarter? Instead of manually pulling, who says they can’t intelligently push data at you. Your aggregator doesn’t need to be some spyware infested key logging site tracking program, it’s easier than that. Taking a page out of email among other things we can begin to lay groundwork for this smarter aggregator.

Smarter?

Yes, smarter. RSS aggregators right now are really dumb. They have two purposes; make your feeds look pretty and readable, and make sure you didn’t read the same item twice. <sarcasm> That’s some intense stuff. They could be doing more than this. They should be doing more than this. How about recommending feeds, items, and links? Filtering out the junk, and letting the best seep to the top. Your email client is doing this already with the likes of Bayesian filtering. The most popular of which is SpamBayes. By applying Bayesian reasoning to your emails, your client can get a really good idea of what is spam, and what isn’t. So dang it, let’s give your aggregator Bayesian reasoning* too.
(* as well as some statistical analysis among other things)

The Beginning of an Idea

Bayes and RSS isn’t a new idea. Matt Griffith though about it back in May 2003. where he asks, “Where is RSSBayes?” Later that month Jim O’Halloran brings up “Bayesian Filtering in Aggregators?” and continues with Part 2. Then in November Jon Udell publishes “Working with Bayesian Categorizers” where he ambitiously uses Bayes to derive categorizations of blog posts. More recently this summer, glassdog just comes out and says how much they are annoyed by aggregators in “What’s Wrong With: Feed Readers” which is also followed up by Leslie Orchard’s “Info Freako, or who’s already past arguing about syndication formats?” Which brings us to about where we are now.

Our Scope

The Smart Project’s aim is to put bayes as well as other new ideas into a “smarter” aggregator. Initially we will put the ideas together into our own aggregator as a testbed. Also this site will serve as a collection of articles, results, and links about how we are working to make your aggregator smarter.