In my previous post I wrote about finding an adaptable and configurable solution for web crawling and scraping that suits our specific needs. My chose for a crawler fell on Abot, an open source framework and now I have to add a headless browser to be able to render and execute Java Script.

Since I have decided on C# and .NET being my platform of choice I ended up with two choices; WatiN and Selenium.

WatiN is a port of the Ruby automation framework WatiR. This seems to have had a lot of traction and looks good but haven’t been updated since 2012 so I scratched this from the list.

The next is Selenium. As i mentioned in the previous post I’ve used this a little already and after trying it out I took a decision to use this as my headless browser using the PhantomJS driver. You can also use other drivers as Chrome, Firefox and so on.