Additional parameters to make the tool a little more useful were a ‘minimum word length’ and a ‘minimum word occurrences’ parameter. These helped cut down the chaff from the real density results. As usual these scripts are a bit of fun and a proof of concept rather than a robust and useable utility so be ready to tweak it to meet your needs!

I ran the solution using Ruby 1.9.2 but the library includes are minimal so you should be able to get it going on most earlier versions.

Enjoy!

require 'open-uri'

URL_TO_PARSE = ARGV[0]
MINIMUM_OCCURANCES = ARGV[1].to_i
MINIMUM_WORD_LENGTH = ARGV[2].to_i

# list of stopwords
STOP_WORDS = ["a","able","about","above","abroad","according"] # etc

# read the page in
puts "\n\nOpening #{URL_TO_PARSE}..."
page = open(URL_TO_PARSE).read()

# First remove the script tags...
puts "Removing script content..."
page.gsub!(/<script.*?>[\s\S]*?<\/script>/i, "")

# then remove the markup...
puts "Removing mark-up..."
page.gsub!(/<\/?[^>]*>/, " ")

# trim the whitespace off the start and end of the lines...
puts "Tidying text..."
page.gsub!(/^[ \t]+|[ \t]+$/, " ")

# and the excess newlines...
page.gsub!(/\n{2,}/,"\n")

# then pull it all onto one line...
page.gsub!("\n"," ")

# now, strip out all punctuation...
puts "Removing punctuation..."
page.gsub!(/[\.|\,|\@|\!|\?|\-|\'|\(|\)]/, '')

# remove the excess spaces...
puts "Further tidying..."
page.gsub!(/ {2,}/," ")

# drop everything to the same case
page.downcase!

# now split it the string using spaces...
page_words = page.split(" ")

# how many words do we have?
puts "Found #{page_words.length} words, removing all words less than #{MINIMUM_WORD_LENGTH} characters in length."

# remove entries which have fewer letters than our parameter
page_words.delete_if{|word| word.length < MINIMUM_WORD_LENGTH}

# how many did we end up with?
puts "Ended up with #{page_words.length} words."

# remove the stop words from the list
puts "Removing stop words from list..."
cleaned_page_words = page_words - STOP_WORDS
puts "Ended up with #{cleaned_page_words.length} words."

# create a hash to hold the words and number of occurances
words_and_occurances = {}

# go through all the cleaned words creating a hash for each, start with no instances, just to get the keys.
cleaned_page_words.each{|word| words_and_occurances[word] = 0}

# then go through again adding up the number of occurances
cleaned_page_words.each{|word| words_and_occurances[word] +=1 }

# sort the list by number of occurances
puts "Sorting by frequency..."
sorted_words_and_occurances = words_and_occurances.sort_by{|k,v| -v }

# output the end result...
puts "Filtering words that occur less than #{MINIMUM_OCCURANCES} times.\n\n"
puts "Here's your words...\n\n"
sorted_words_and_occurances.each do |word,occurances|
	if occurances > MINIMUM_OCCURANCES then
		puts "#{occurances} occurances of #{word}"
	end
end

And when we run the script on our homepage, looking for words that appear at least 3 times and have a minimum length of 4 letters…

C:\path\to\script>ruby keyword_density.rb http://www.epiphanysolutions.co.uk 3 4

Opening http://www.epiphanysolutions.co.uk...
Removing script content...
Removing mark-up...
Tidying text...
Removing punctuation...
Further tidying...
Found 421 words, removing all words less than 4 characters in length.
Ended up with 289 words.
Removing stop words from list...
Ended up with 240 words.
Sorting by frequency...
Filtering words that occur less than 3 times.

Here are your most frequently occuring words...

14 occurrences of search
5 occurrences of google
5 occurrences of conversion
5 occurrences of media
5 occurrences of agency
5 occurrences of social
4 occurrences of marketing
4 occurrences of optimisation
3 occurrences of partner
3 occurrences of december
3 occurrences of london
3 occurrences of certified
3 occurrences of development
3 occurrences of leeds
3 occurrences of 2011
3 occurrences of paid
3 occurrences of organic
3 occurrences of dont
3 occurrences of analytics
3 occurrences of contact
3 occurrences of clients

We’ve decided to put our money where our mouth is and go head to head on a number of quick and easy parsing tools which automate a lot of the reporting and repetitive tasks of your everyday SEO analyst. We’re comparing C# on Microsoft’s .NET framework and its extensive range of web libraries with the more popular *nix based languages such as Python, PHP and Ruby.

Each month we will be presenting a number of simple utility scripts which should be easy to bundle, install, run but more importantly, easy to modify for tying into databases and attaching email alerts etc. Who knows, we may even venture into threading, making the tools seriously powerful.

Keep an eye on the blog over the next few months to pick up some nice little titbits of functionality to help automate your daily chores, whatever your platform!

Contenders will initially be myself, Andy Webb, technical architect and Drew Brigham, senior .NET developer here at Epiphany. No doubt we’ll pick up a few more contenders along the way and if you feel you want to contribute a solution, feel free to get involved!

Keep an eye open this month for the first challenge.

Let’s take a scenario where we have (generously) been given a couple of links from a few of prolific sites in our industry. To make sure we stay in favour with these sites, we run a weekly check to make sure they’re still happy to link to us. Now weekly checks aren’t that much of an effort to carry out manually but its best to automate this check for a number of reasons:

1.) Who wants to be setting reminders and reading through web pages manually?

2.) If it’s not in the usual place for whatever reason, we’ve got to trawl source code to see where they moved it to.

3.) As small a point as this is, if we needed to check more frequently, we don’t really want to be triggering their analytics tracking, just in case they’re funny about robots giving a false impression of traffic. So step in Ruby and Nokogiri.

We want to:

1.) Visit the pages in question.
2.) Parse the HTML.
3.) Pull out all the links.
4.) Find ours and read the anchor text.
5.) Make sure it’s there and they haven’t no-followed it!
6.) Report back the status of the links.

Step in Ruby/Nokigiri:

require 'rubygems'
require 'open-uri'
require 'nokogiri'

our_url = "http://www.epiphanysolutions.co.uk"
urls = ["http://www.27bslash6.com","http://www.ford.com","http://www.seomoz.org/blog/improving-reporting-efficiency-and-relevance","http://csshook.com"]
today = Time.now.localtime.strftime("%d-%m-%Y")

begin
	# cycle through the URLs in the list
	urls.each do |url|
		# read the page and parse it into Nokogiri
		page = Nokogiri::HTML(open(url))
		# select any anchors that link to our domain. (in 1 line! Good old XPath.)
		links_to_us = page.xpath("//a[substring(@href, 1, #{our_url.size}) = '#{our_url}']")
		if (links_to_us.size > 0) then
			links_to_us.each do |anchor|
				# tell the user about the link we found
				printf "On %s we found a link to %s from %s with anchor text \"%s\"", today, anchor.attr('href'), url, anchor.text()
				# and tell them whether or not its nofollowed..
				puts anchor.attr('rel') == "nofollow" ? " but it's no followed!" : ". All good on that link."
			end
		else
			printf "On %s we couldn't find a link on %s to %s\n", today, url, our_url
		end
	end
rescue Exception => e
	# ... if we can't open or parse a page for whatever reason, tell us why
	puts e.message
end

And what do we get?

On 30-08-2011 we found a link to http://www.epiphanysolutions.co.uk/ from http://www.27bslash6.com with anchor text "SEO - Epiphany Solutions". All good on that link.

On 30-08-2011 we couldn't find a link on http://www.ford.com to http://www.epiphanysolutions.co.uk

On 30-08-2011 we found a link to http://www.epiphanysolutions.co.uk from http://www.seomoz.org/blog/improving-reporting-efficiency-and-relevance with anchor text "Epiphany Solutions" but it's no followed!

On 30-08-2011 we found a link to http://www.epiphanysolutions.co.uk/ from http://csshook.com with anchor text "SEO". All good on that link.

I’m sure you can see the potential of this tool if you start threading its functionality into databases, logs and alert mailing systems, and the XPath AND jQuery style selectors give it an extra layer of accessibility and flexibility. Say what you want about these slightly more obscure open source programming languages, but they get the job done quickly and reliably! They have also come up trumps many a time with me as the right tool for the job. They come with an array of libraries to do most day to day tasks and with them being an open source, people even compete with each other building libraries that do the same job – meaning even more simplicity and power!

What are your thoughts?

Mechanize is a completely programmable browser implemented (in this instance) purely in Python. This tool is priceless for automated site health checks for everything from link presence, robots.txt rules, automated form submissions, cookie handling, you name it.

Whereas mostly we would parse a page as text and look for links, then write our own logic to follow that link and pull out information about the page, mechanize allows you to do all this at a much higher level. Here is just one neat trick:

# Make a browser.
browser = mechanize.Browser()
# Open a page.
browser.open("http://www.epiphanysolutions.co.uk/")
# Follow the second link for the keyword 'analytics'.
myResponse = browser.follow_link(text_regex=r"analytics", nr=1)
# Print the title of the page.
assert browser.viewing_html()
print browser.title()

…returns “Google Analytics IQ Consultants – Google Analytics Consulting Services”, the title of the page linked to by the second ‘analytics’ link.

The browser also has a wealth of other features such as robots.txt handling, proxy handling and redirect detection. Not only does this make short work of automated sitemap building, checking for the presence and sanity of acquired links and ensuring our pages aren’t blocked by robots.txt, but with a little extra work it can submit to forms, handle cookies and also handle various types of errors and exceptions making it invaluable for just throwing onto a regular scheduled task for the daily weeding out of quirks and problems which manual surfing may not immediately uncover.

I’m aware I’m in a bit of a niche here but if you write programs to programmatically interact with websites, give it a whirl! You’ll not be disappointed.

This made me think about my brand loyalty, very rarely do hardcore Mac users ever migrate to PC’s, usually for the above reasons. I can’t think of any Mac product or program that I’ve abandoned for a better alternative on another platform — except iTunes. This software went from being horrible, clunky and overcomplicated in the days of Winamp to pretty much the most popular music program available today. (I may have made that up.) The iTunes boom also added to the already rife office one-up-man-ship of “Oh, I have over 84 gigs of music on my computer”. “Really, I have 92 gigs of Indy alone on mine.”

These days have been cut short in our office with the introduction and popularisation of Spotify. I’m still not 100% sure on how Spotify manages its flawless sound quality and uninterrupted streaming of the most eclectic mix of music you’d ever want (for free) but Apple are sure taking a long time to react to it. One can only assume the cost of royalties are covered by the advertisements played on Spotify but can you see Apple’s response running similar advertising campaigns? Knowing their rigorous brand/image protection, I can imagine third party advertising would somehow cheapen their product, which with apple, is the biggest of the big no-no’s.

How are Apple going to quash Spotify’s growing popularity without going down a similar route and making free music economically viable? How are they going to continue earning so many billions from the humble compressed audio format (not just iTunes music store sales but iPod sales also.) As more and more record companies publish their catalogues to Spotify, it’s going to be interesting to see how Apple continues to make the format pay!

This has caused the OS giant to resort to offering cash discounts to US shoppers making purchases after coming into sites through a Live search advert. This initiative offers partial refunds on purchases through a new “cashback” system where money is deposited directly into customer’s bank accounts or optionally through their PayPal accounts.

Big hitters who have agreed to adopt the cashback scheme are book retailers Barnes & Noble, home improvement and construction products retailer Home Depot and online clearance warehouse Overstock.com amongst others.

Speaking from the annual digital ad conference in Redmond, Bill Gates said “2008 is the year that search got competitive,” along with promises of “big announcements around search to happen every six months” from the Microsoft camp.