View on GitHub

Site Lab

Open source website analyzer and technology tracker. Built with Ruby on Rails 4.

Download this project as a .zip file Download this project as a tar.gz file

Site Lab

Site Lab aims to be an open-source replacement for website analysis tools such as BuiltWith, NerdyData, and DataNyze.

Site Lab is a Ruby on Rails application. It uses PostgreSQL as its database and Redis + Sidekiq for background processing (background processing still in the works).




There is a live demo on Heroku at Please be a good citizen wrt the demo (e.g. try it with real-world data & submit issues if you find a problem).

The demo is currently a few commits behind as I haven't had a chance to setup Redis/Sidekiq on Heroku (now used in SiteLab for background processing).

How Does it Work?

Right now, it's fairly simple:

More complex analysis is in the works.

Why Did You Build This?

Two reasons:

  1. Current commercial SaaS products in this space are quite expensive (i.e. > $200/month). Mainly because they aim to be lead-generation services. I'm not disputing their value, I just have smaller needs and a smaller budget.
  2. I have some specialized needs (i.e. I want to search for specific/niche technologies), so I like the flexibility of being able to define my own parameters.


It's a Rails 4.1 app, so you'll need a dev environment that supports that (prolly RVM). You'll also need Redis installed and running (probably via Homebrew)

Importing Data

While you can surely add sites/URLs one-by-one in the app, most use-cases will involve importing large sets of URLs from files or external sites. With that in mind, I've started a set of Rake tasks for importing URLs. Currently, it includes:

Run a rake -T to see the tasks and required parameters

TO-DOs & Roadmap