Sitemaps in Ruby on Rails

Simon Schoeters <blockquote cite="http://www.sitemaps.org/"> Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site. </blockquote> Sitemaps have a wide adoption including Google, Yahoo! and Microsoft so I thought it would be a good idea to integrate this in my website. A few posts later I was tired updating this XML file manually every time I changed something in the URL scheme so why not pass the task to Ruby on Rails and build the file automatically? <h3>The controller</h3> First you'll need a method to collect the data. I choose for the application controller as the Sitemap doesn't really belong anywhere else. The pages controller would be a better choice if you have one that manages all your sites URL's but that's entirely up to you. <samp>class ApplicationController < ActionController::Base def sitemap @pages = Page.find(:all) render_without_layout :template => "layouts/sitemap" end</samp> The ‘render_without_layout’ part calls the view. My view is in the views/layouts folder but again, this can be anything you want. <h3>The view</h3> As defined in the Sitemap method above we need a view that renders the data in an XML file. Create the Sitemap XML template in the folder you defined above (views/layouts in my case) and call it ‘sitemap.rxml’. Now build the structure of the Sitemap: <samp>xml.instruct! xml.urlset('xmlns'=>'http://www.sitemaps.org/schemas/sitemap/0.9', 'xmlns:xsi'=>'http://www.w3.org/2001/XMLSchema-instance', 'xsi:schemaLocation'=>'http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd') { for page in @pages xml.url { xml.loc("http://" + request.env["HTTP_HOST"] + "/" + page.permalink + "/") xml.lastmod(page.updated_at.strftime('%Y-%m-%d')) xml.changefreq("weekly") xml.priority("0.7") } end }</samp> This <a href="http://en.wikipedia.org/wiki/Code_snippets" title="Code snippets definition on Wikipedia" class="ext">snippet</a> assumes your page object has a permalink and an updated_at parameter, change these if yours looks different. There are a few things you need to know about Sitemaps: ‘loc’ is the only required element so you can drop the ‘lastmod’, ‘changefreq’ and ‘priority’ elements if you don't have any useful data for these parameters. More in detail: <ul> <li>loc - required: the full URL to the page, include your domain as well.</li> <li>lastmod - optional: last modification date for that page in the <a href="http://www.w3.org/TR/NOTE-datetime" title="W3C Datetime format specification" class="ext">W3C Datetime</a> format, probably something like YYYY-MM-DD.</li> <li>changefreq - optional: how frequently the page is likely to change. Valid values are: always, hourly, daily, weekly, monthly, yearly or never.</li> <li>priority - optional: the priority of this URL relative to other URLs in your site. Valid values range from 0.0 to 1.0.</li> </ul> See the official <a href="http://www.sitemaps.org/protocol.php" title="sitemaps.org protocol description" class="ext">Sitemap protocol definition</a> site for a full description. <h3>The route</h3> You have a automatically generated Sitemap but no way to get there. Tell Rails to call your Sitemap in the routes.rb file by adding the following mapping (change the controller if you choose a different one above): <samp>map.connect 'sitemap.xml', :controller => 'application', :action => 'sitemap'</samp> Request your new Sitemap with http://www.example.com/sitemap.xml. You may need to restart your Rails server to enable the new route. <h3>The robots.txt</h3> Almost done. The Sitemap should be working by now but how does a crawler (like the <a href="http://en.wikipedia.org/wiki/Googlebot" class="ext" title="Googlebot's page on Wikipedia">Googlebot</a>) where to look for you Sitemap? That's where the robots.txt file is for. Every crawler should request the robots.txt file first to see what it may or may not index so this is the ideal place to advertise our Sitemap. Add the following line to the robots.txt file in you public directory and make sure to use the full URL to your Sitemap (including the domain): <samp>Sitemap: http://www.example.com/sitemap.xml</samp> That's it for today! The next time a crawler visits you site it will find the Sitemap and index it. <h3>Resources</h3> <ul> <li><a href="http://www.sitemaps.org/" title="sitemaps.org" class="ext">Official Sitemap definition</a></li> <li><a href="http://www.d-jones.com/2007/4/5/google-sitemaps-in-ruby-on-rails" title="Great article I used to get started" class="ext">Google Sitemaps in Ruby on Rails</a></li> <li><a href="/sitemap.xml" title="My Sitemap as an example">My site's Sitemap</a> as an example</li> </ul>

2007-10-07T14:17:14+02:00

sitemaps Sitemaps in Ruby on Rails

2007-12-12T12:42:31+01:00

Simon Schoeters Have a look at my social graph in the sidebar on the <a href="/about" title="About Suffix">about</a> page. It shows a bunch of links to websites elsewhere on the web that are loosely related to me (e.g. my LinkedIn profile or Flickr photos) or my “friends”. These links are automatically collected based on <a href="http://gmpg.org/xfn/" title="XHTML Friends Network" class="ext">XFN</a> and <a href="http://www.foaf-project.org/" title="Friend of a Friend (FOAF) project" class="ext">FOAF</a> data with the Google Social Graph <abbr title="Application Programming Interface">API</abbr>. <h3>What?</h3> A <a href="http://en.wikipedia.org/wiki/Social_graph" title="Social graph on Wikipedia" class="ext">social graph</a> consists of who an individual is connected to based on the type of connections. In the internet world people use XFN and FOAF to describe their connections with other peoples websites. These websites can be other websites of myself (such as my Flickr photo page) or a website of a friend of mine. <samp><a href="http://fousa.be" rel="friend met">Jelle</a> <a href="http://flickr.com/photos/schoeters" rel="me">My photos</a></samp> The previous 2 XFN examples describe my connections with <a class="url fn n" href="http://www.fousa.be">Jelle Vandebeeck </a>'s website, a friend of mine, and my Flickr photo page. Now, I'm not the only one using this, thousands of links (blogs, LinkedIn profiles, Twitter profiles, etc.) have been marked up in this way. Wouldn't it be nice when we could visualize all these online connections? Google recently released their <a href="http://code.google.com/apis/socialgraph/" title="Google Social Graph API" class="ext">Social Graph API</a> which crawls the Google index for all XFN and FOAF enabled links based on a list of URL's. This returns a <abbr title="JavaScript Object Notation">JSON</abbr> response with all these links and the connection types associated with it. As the API is using the Google index you have access to a huge amount of data in a very fast way. On the downside the correctness of the data depends on the freshness of the Google index. Today my Twitter profile link still shows up in the results even though I removed my account a few months ago and the links in my previous blog post aren't found yet. <h3>Ruby on Rails implementation</h3> Let's have a look at the Ruby on Rails code to build our own social graph. First you'll need the JSON gem to parse the response from Google (the may be different in Rails 2.x, I didn't check). <samp>gem install json</samp> Great, now lets ask Google for our social graph. I built an array with all the pages on this blog but for the sake of the example I will use a static array with some sample links. <samp>require 'open-uri' require 'json' links = ['http://www.example.com/','http://www.example.com/example'] url = "http://socialgraph.apis.google.com/lookup?q=" + links.join(',') + "&fme=0&edi=1&edo=0" resp = Net::HTTP.get_response(URI.parse(url)) result = JSON.parse(resp.body) graph = Array.new for node in result['nodes'] for sub_node in node[1]['nodes_referenced_by'] graph << [ sub_node[0], sub_node[1]['types'] ] end end graph.uniq!</samp> At the end we remove the duplicate URL's and there you have it, an array with all you XFN and FOAF friends linking to the given URL's. You can play around with the 3 parameters at the end of the Google URL: <table summary="List of possible URL parameters and their values"> <tr> <th>fme</th><td>true, false, 1 or 0</td><td>Follow the ‘me’ links found on the given URL's.<td> </tr> <tr> <th>edi</th><td>true, false, 1 or 0</td><td>Return a list with pages linking to the given URL's.<td> <tr> <th>edo</th><td>true, false, 1 or 0</td><td>Return a list with pages linked from the given URL's.<td> </tr> </table> <h3>Resources</h3> <ul> <li>Google's Social Graph <a href="http://code.google.com/apis/socialgraph/docs/api.html" title="Social Graph API Documentation" class="ext">API documentation</a>.</li> <li>The very simple <a href="http://braincast.nl/samples/jsoneditor/" title="JavaScript JSON Editor" class="ext">JavaScript JSON editor</a> is an easy way to visualize the JSON tree.</li> </ul>

2008-03-03T00:07:00+01:00

google-social-graph Google Social Graph in RoR

2008-05-20T10:45:44+02:00

Simon Schoeters It's all about the map in this sites header today. The header background shows a dynamically generated map of the coordinates shown in the right top corner, or my last recorded place. I can select my current location (latitude and longitude) and the header background map is updated accordingly. So how exactly does this work?! <h3>No Google or Yahoo! maps?</h3> I integrated Yahoo! Maps first. The Yahoo! Maps look nice, the API is easy to use and well documented but the problem is that you are only allowed to play with these maps within limits placed on you by the API's terms and conditions. Below are some extracts from the <a href="http://info.yahoo.com/legal/us/yahoo/maps/mapsapi/mapsapi-2141.html" title="Yahoo! Terms Center" class="ext">Yahoo! Maps API Terms of Use</a>: <ul> <li>You understand and agree that you must retain the attribution text and links contained within all Yahoo! Maps and Yahoo! Map images without alteration.</li> <li>Your will not store or allow end users to store map imagery, map data or geocoded location information from the Yahoo! Maps APIs for any future use.</li> </ul> I would break these terms as I convert the original image to a grayscale version and store the map on the server for performance reasons so the Yahoo! Maps are out of the question. Google or Yahoo! Maps are great for out of the box functionalities but if you want more freedom you will probably have to look elsewhere (or pay for licenses if possible). <h3>OpenStreetMap</h3> Luckily for us there are other solutions out there. Let's take a look at <a href="http://www.openstreetmap.org/" class="ext">OpenStreetMap</a> for example. OpenStreetMap is a ‘wiki style’ editable map, made by people like you and me. The content is available under the Creative Commons Attribution-ShareAlike 2.0 license which means anyone can use (and remix) the data as long as you share it with a similar license. This ‘Wikipedia-like’ approach has some drawbacks as well: the map is incomplete and the data is not 100% reliable. On the other side you can correct or complete the map where needed. This may or may not be important for your project but I only needed a nice background image for this website so a general overview of the area is more than sufficient. OpenStreetMap has a RESTfull way to build a map image with a given latitude, longitude, zoom level and image size. Here is how (or see an example for <a href="http://tah.openstreetmap.org/MapOf?lat=50.8668&long=4.69714&z=12&w=200&h=200" title="OpenStreetMap map of my home town" class="ext">my city</a>): <samp>http://tah.openstreetmap.org/MapOf?lat=<lat>&long=<long>&z=<zoom>&w=<width>&h=<height>&skip_attr=1</samp> The <code>skip_attr</code> defines if you want to include an <abbr title="OpenStreetMap">OSM</abbr> attirbution image in your map or not. <h3>Start coding</h3> The process flow: <ol> <li>The current location is updated on the website,</li> <li>a map of this new location is downloaded from a map server,</li> <li>the downloaded map image is converted to grayscale and</li> <li>the generated image is saved to disk.</li> </ol> <img src="/images/map_flow.gif" alt="Map flow" /> We'll use <a href="http://rmagick.rubyforge.org/" title="Graphics Processing for Ruby and Ruby on Rails" class="ext">RMagick</a> and Ruby on Rails to convert the downloaded image to something else so make sure you have this installed first. <samp># Writes an image with a map of the location, the zoom ranges from 4 to 17 def map(latitude, longitude, width = 500, height = 500, zoom = 7) require 'RMagick' require 'open-uri' map_request_url = "http://tah.openstreetmap.org/MapOf?lat=#{latitude}&long=#{longitude}&z=#{zoom}&w=#{width}&h=#{height}" uri = URI.parse(map_request_url) map = Magick::ImageList.new map.from_blob(uri.read) map = map.quantize(256, Magick::GRAYColorspace) map.write(RAILS_ROOT + '/public/images/map.png') end</samp> That's it, this method downloads and converts the image and saves it in your images folder so from now on you can use this image in your website: <samp><img src="/images/map.png" alt="Map" /></samp> <h3>More resources</h3> <ul> <li><a href="http://www.alistapart.com/articles/takecontrolofyourmaps" title="Take Control of Your Maps" class="ext">Take Control of Your Maps</a> explains how to build you own map stack.</li> </ul>

2008-04-15T18:14:57+02:00

generated-maps Generated maps with Ruby on Rails

2008-09-19T01:45:44+02:00

Simon Schoeters I keep the latitude and longitude for each of these blog post (as you can see in the right hand column). My blog's <abbr title="Really Simple Syndication">RSS</abbr> feed is generated from my posts table. Until now it was a plain simple RSS feed but why not combine the 2 and enhance my RSS feed with the coordinates? They are there anyway and GeoRSS is widely adopted these days. <h3>What is GeoRSS?</h3> <a href="http://georss.org" title="Geographically Encoded Objects for RSS feeds" class="ext">GeoRSS</a> is a way to encode location information in RSS feeds. This can be as simple as adding one small <abbr title="Extensible Markup Language">XML</abbr> element to your existing feed. GeoRSS feeds are designed to be consumed by geographic software such as map generators and by doing so your feed can be used in new and different ways: pinpointing on a may, finding other posts close to this one, etc. <h3>What we need</h3> Generating the actual RSS feed is not in the scope of this article, we'll only add location awareness to our existing feed. Please head over to Larry Myers' <a href="http://www.myersds.com/notebook/2006/05/11/how_to_generate_rss_feeds_with_rails" title="How To Generate RSS Feeds with Rails" class="ext">How To Generate RSS Feeds with Rails</a> article if you don't generate your feed yet. We'll need to add one small XML element to each item in our feed with the latitude and longitude for the post: <samp><georss:point>50.8668 4.69714</georss:point></samp> In Ruby on Rails this looks like the following (where latitude and longitude are 2 fields in our posts table). The syntax looks a little strange as we need a way to add the ‘georss:point’ element and Ruby doesn't like the colon in there (it indicates a <a href="http://corelib.rubyonrails.org/classes/Symbol.html" title="Symbol class definition" class="ext">symbol</a>). <samp>xml.georss :point do xml.text! post.latitude.to_s + ' ' + post.longitude.to_s end</samp> Great, now we only need to add the GeoRSS namespace at the top of our feed to tell the feed parsers what we mean. <samp>xml.rss (:version => "2.0", "xmlns:georss" => "http://www.georss.org/georss")</samp> <h3>My location aware RSS feed</h3> The full view template should now resembles something like this: <samp>xml.instruct! :xml, :version=>"2.0" xml.rss (:version => "2.0", "xmlns:georss" => "http://www.georss.org/georss") { xml.channel { xml.title("My GeoRSS feed") xml.link("http://www.example.com") xml.description("My posts enhanced with location info") xml.language('en-us') for post in @posts xml.item do xml.title(post.title) xml.description(post.content) xml.author("none@none.be (#{post.author})") xml.pubDate(post.created_at.strftime("%a, %d %b %Y %H:%M:%S %z")) xml.link( post_permalink_url(post.permalink) ) xml.guid( post_permalink_url(post.permalink) ) xml.georss :point do xml.text! post.latitude.to_s + ' ' + post.longitude.to_s end end end } }</samp> That's it. Not too exciting at first but try your RSS feed URL in Google Maps and see how easy it is to pinpoint each location on a map (here is <a href="http://maps.google.com/maps?f=q&hl=en&geocode=&q=http:%2F%2Fwww.suffix.be%2Ffeeds%2Fblog.xml&safe=on&ie=UTF8" title="My GeoRSS feed in Google Maps" class="ext">mine</a>).

2008-05-23T17:00:46+02:00

georss-with-ruby-on-rails GeoRSS with Ruby on Rails

2008-05-23T18:25:04+02:00

Simon Schoeters Today we launched a new website: <a href="http://www.milkcarton.be/" title="Mac OS X applications and frameworks by milkcarton" class="ext" rel="me">milkcarton.be</a>. Last year we started developing a small Cocoa application but got stuck and never got to something ‘release worthy’, the half finished program was waiting there to be forgotten. A few months ago we decided to pick up where we left of and fix the design mistakes we made last year. We started from scratch and here we are, releasing our first private beta today! <h3>Lustro</h3> What is this mysterious app? It listens to the name Lustro and is a small Address Book exporter for Mac OS X Leopard. It exports all your contacts to <abbr title="Comma-separated values">CSV</abbr>, tab-delimited or hCard files or overwrites your Google Contacts with the ones on your Mac (so that you have access to all your Address Book contacts in GMail). If this sounds like something you could use you can <a href="http://www.milkcarton.be/contact" title="Contact milkcarton" class="ext" rel="me">drop us a note</a> as we are still looking for beta testers. Lustro is free and open source so there is no excuse to chicken out on this one. <h3>milkcarton</h3> We didn't really have a decent home for Lustro so we decided to create a new website to host these little applications, frameworks or other bits of code we designed and will design. That's <a href="http://www.milkcarton.be/" title="Mac OS X applications and frameworks by milkcarton" class="ext" rel="me">milkcarton.be</a>: hop over there to and have a look around.

2008-06-20T18:15:57+02:00

launching-milkcarton-lustro Launching milkcarton with Lustro

2008-06-20T18:15:57+02:00

Simon Schoeters Only 3 days ago we launched the first version of <a href="http://www.milkcarton.be/apps/lustro" title="Lustro, export your Address Book contacts" class="ext">Lustro</a>. It took us a week between the beta testing and the final release and we fixed and enhanced quite a few things. Lustro is our little Mac OS X Leopard app to export your Address Book contacts to <abbr title="Comma-separated values">CSV</abbr>, tab delimited, hCards and Google Contacts. <h3>Google Contacts export</h3> A few months back making an application that exports your Address Book contacts to GMail sounded like a useful idea. Little the we know that Apple was working on the same thing. A few weeks before we released the beta version Apple came out with their built-in Address Book to Google Contacts export with the release of <a href="http://support.apple.com/kb/HT1141" class="ext" title="About the Mac OS X 10.5.3 Update">Mac OS X 10.5.3</a>. We are not good enough to beat Apple... not yet. We decided to keep the Google export in Lustro for the following reason (copied from the Lustro Help): <blockquote>“Address Book also synchronizes with MobileMe, Exchange, Yahoo! and Google if you enable this in the preferences. Lustro still includes the Google Contacts export as the Address Book syncing only works if you have connected an iPhone or iPod Touch before and the syncing mechanism is not perfect at the time of writing (e.g. company cards show an empty name in the GMail contacts list).”</blockquote> So there are a few smaller reasons why you would still use Lustro today, maybe we can add the photo export in the next release as Apple skipped that one as well. <h3>hCard exports</h3> The other mayor feature are the <a href="http://microformats.org/wiki/hcard" title="hCard - Microformats wiki" class="ext">hCard</a> exports. We couldn't find any hCard exporters for Apple so why not include it? We got some indirect feedback from <a href="http://flickr.com/photos/factoryjoe/2625201023/" class="fn ext">Chris Messina</a>, <a href="http://flickr.com/photos/factoryjoe/2625201023/comment72157605921180862/" class="fn ext">Brian Suda</a>, <a href="http://flickr.com/photos/factoryjoe/2625201023/comment72157605908401795/" class="fn ext">Tantek Çelik</a>, <a href="http://microformatique.com/?p=263" class="fn ext">John Allsopp</a> and <a href="http://ma.gnolia.com/people/adactio/bookmarks/scucuwumuch" class="fn ext">Jeremy Keith</a>, whow, I'm impressed! In the end I think we released Lustro a little too late for the GMail export thing, the buzz was over and lots of GMail exporters started to pop-up but the hCard export made up for that one. <a href="http://flickr.com/photos/factoryjoe/2625201023/comment72157605921180862/" class="fn ext">Brian Suda</a>'s idea for a distributed hCard export sounds great. I don't think it fits in Lustro as it's the opposite way but I can see an Address Book plugin that ‘subscribes’ on hCard enabled webpages.

2008-07-03T20:02:40+02:00

lustro-out-now Lustro on the streets

2009-02-21T22:09:25+01:00