[by Lance Robinson, Tech Evangelist]

Yesterday Jon Udell wrote about slicing and dicing, and then rewiring the web. Reusing websites as data sources is great - and Yahoo Pipes provides several such "sources" in Yahoo Search, Yahoo Local, Flickr, Google Base, and the basic URL Fetch. All these are great and provide interconnectivity and the ability to slice and dice certain parts of the web. But why stop there? The beauty of this annotating, slicing, and dicing, is in the reusability of this data - not just for reading blogs - but for transporting and exchanging real data that we can compute with. This is where Yahoo Pipes has missed the bus so far, but I have no doubt they will catch the next one.

Mark says that the power in Pipes is that it "gives you more structure and semantics to grab onto and use in the modules, building more value into standard components, rather than having to go and re-invent the wheel for each application". But does it really? It could, if it didn't rip valuable data out of the sources.

Yes, my biggest complaint about Yahoo Pipes right now is that it does things to my feed that I don't ask it to. If I source a Flickr feed and output it - I get a feed with a title, description, link, and pubDate. But I happen to know that there is a lot more rich data inside that original Flickr feed. There are so many great data feeds out there - Yahoo has some very useful ones - why rip out that useful data?

I have enjoyed playing with Yahoo Pipes, but right now I can't use it with my data feeds like the ones that RSSBus generates from my SQL Server, my QuickBooks installation, my bank account, etc. Well...I can use it, but I can't get at all the details of the feed - only what is exposed in title, description, link, and pubDate!

One of my favorite things about the RSSBus Desktop server is that it lets me slice and dice the web (feeds), but on top of that it also gives me the option of slicing and dicing it with my own private data. I know that may be extra nerdy, but it sure is cool to me.

Over the past year we have been talking a lot about connecting data from sources, in a fashion that resembles the "pipe and filter" mechanism. It turns out that we share a common vision with Yahoo when it comes to using RSS as the core technology for messaging and data transformation. Yahoo recently announced Yahoo Pipes which provides an online graphical interface allowing you to "mashup" RSS feeds in much the same way as RSSBus. 

In his blog yesterday, Tim O’Reilly highlights some of the technical features of Yahoo Pipes:

"Using the Pipes editor, you can fetch any data source via its RSS, Atom or other XML feed, extract the data you want, combine it with data from another source, apply various built-in filters..., count, truncate, union, join, as well as user-defined filters), and apply simple programming tools like for loops."

Sound familiar? If it doesn’t, take a look at the RSSBus Whitepaper. The ability to manipulate and transform data feeds is at the very essence of the RSSBus.

The first step however is feeding data to the pipe. We are encouraged by constantly emerging data sources on the Internet, Yahoo RSS Feeds, Google's API's among others, good as they are, a lot of important data is entangled in applications and IT infrastructure that are internal and proprietary. This is where RSSBus truly shines. RSSBus has a connector architecture that enables you produce feeds from a growing list of software applications or online services (browse through our growing list of connectors).

Want to subscribe to new opportunities on Salesforce.com? Use the Salesforce ops and configure the feed. What about feeds of reports from your Oracle or SQL database? Again, with RSSBus you can configure a feed of database reports in minutes. Since RSSBus is a software application and not a 3rd party service, you can be sure that the sensitive feeds that you produce are safe and secure. Moreover, easy and simple interfaces, allow you to write connectors to tap into data sources that are unique to you.

We are excited about the release of Yahoo Pipes and feel it is a terrific step forward.  Watch for demos in the near future that show how to use Yahoo Pipes to easily create and expose feeds of essential business data.

For all of the RSSBus users that have been asking for a hosted version of RSSBus to build feeds with, we're listening... keep an eye out for more announcements soon!

[by Lance Robinson, Tech Evangelist]

This post by Jeff Barr gives some interesting examples of things people might want to do with their data, and more specifically connecting their data to and from multiple sources.  The kind of things Jeff mentions can be attained using RSSBus.  It all boils down to flexibility, and how someday we should be able to take such flexibility for granted.

One of the examples Jeff gives is locating the Amazon wishlist of his top email correspondents, which is actually quite easily done with RSSBus.  RSSBus comes with what we call "connectors" for email (pop, imap, smtp), amazon web services (including wishlist searching), and lots of other data sources.  And since the RSSBus connector interface is open, new connectors for things like LinkedIn and Twitter can be created and shared.  I can take the ImapOps connector and pipe its output to the AmazonOps connector and voila, I have a list of amazon wishlists for my top email correspondents.  How?  Check this out, here's the skinny:

1. Call the RSSBus imapSearch operation and get a list of emails.
2. Call the RSSBus rsbFeedUnion operation to consolidate mails from the same sender into one item per sender, with a count of how many emails from this sender were in the box.
3. Sort this list by number of emails from each sender with the rsbSort operation.
4. Use the amazonListSearch operation to Search for an Amazon wishlist for the top 10 most active senders in the my list of emailers.

Here's what the actual script looks like (download the full script):

<rsb:call save="mail" op="imapSearch?server=myserver&searchcriteria=ALL" />
<rsb:call save="union" op="feedUnion?feed=[_feeds.mail | urlencode]&filter=imap:fromemail" />
<rsb:call op="feedSort?feed=[_feeds.union | urlencode]&sortby=rsb:count&type=numeric">
  <rsb:call op="amazonListSearch?email=[imap:fromemail]&type=wish" pagesize="10">
    <rsb:push />
  </rsb:call>
</rsb:call>

For those who want a more technical explanation, let me take you up close and personal with this RSSBus script:

1. First, I call the imapSearch operation.  I give it my IMAP server and login information and tell it I want a list of ALL my emails.  I save this list in a cached RSS feed called "mail".  Two sub-notes here:  authentication info need not be specified here if I take advantage of the RSSBus profile, and searchcriteria is extremely flexible and follows the IMAP specification, so I could do something like searchcriteria=SENTSINCE 1-Feb-2007 to get a list of only the mails sent since Feb 1.
   
2. Next, I call the feedUnion operation.  I tell it to perform a union on the "mail" feed.  Yes, I'm doing a union on one set of information, and the reason is so that I can take advantage of the filtering capabilities of the feedUnion operation to consolidate duplicate mails from the same sender.  So the result of this call is a feed of information about emails, in which each sender only appears once in the list.  I save this feed as "union".  Note that feedUnion will increment a counter for me when a particular sender is encountered more than once.  I could set the filter input to eliminate duplicate subjects, or any other attribute of the mail feed, but here I am interested in the sender.
   
3. Now I want to sort this "union" feed of emails by the number of times each sender has mailed me, so I call feedSort, give it my "union" feed, and tell it which attribute to sort by (rsb:count).  Unlike the two previous operation calls, I won't save this feed, but instead, for each of the 10 (pagesize) individual items in the feed I'll call the amazonListSearch operation.
   
4. Finally, the amazonListSearch operation returns a final feed of items containing information about the wish list(s) (if one exists) of each mail sender.  Now I have an RSS feed of Amazon wishlists for my top email contacts!

RSSTools is a set of development components that accompanies the RSSBus Engine.  It includes a full array of client access components for all major platforms and development environments.  These components are useful for use within applications other than the RSSBus framework.    

In addition to client components, the RSSTools package provides a set of embedded servers that allow you to serve RSS content from any application.

The server components are embedded web servers that run within the context of your application or device and translate RSSBus operation calls to events within your application.  You intercept these calls and provide output to the component as a set of RSS items.  The component then serializes these items into an RSS feed which gets served to the client.

The current release of RSSTools includes the following components:

• RSSClient - Used to retrieve RSS feeds with data extensions. Supports secure (SSL) communication and HTTP auhentication.
• RSSFeed - RSS feed creation and manipulation component.
• RSSServer - Specialized embeddable web server component that can serve RSS feeds from any application.

Download the RSSTools Beta for your development environment of choice here.