This post is by Marc Gayle
Click here to view on the original site: Original Post
Editor’s note: Marc Gayle is a Rails developer and founder of 5KMVP, where he builds Minimum Viable Products for just $5K. Follow him on Twitter.
A few months ago, I got the idea that one way to get leads for remote freelance gigs was to scour Craigslist. So after doing the manual work of “crawling” through at least 100 job postings by hand, I wrote a Ruby script to do the heavy lifting and filtering for me.
Once I started looking through the data, some interesting things started jumping out at me. Even though I don’t actually live in the Valley (I live in Jamaica), I consume a lot of the news, blog posts, and articles that come from the Valley. Suffice to say, I am affected by the “Valley echo-chamber.” One side effect of that is an obsession with Ruby and Ruby on Rails as my development stack and a general expectation that the rest of the world has woken up to its beauty and elegance.
Alas, much to my surprise, that is not the case.
Before diving into the data, let me explain exactly what this script does. Throughout Craigslist, there are two URL subpaths that tend to have the majority of the web development freelance gigs: /cpg/ and /web/. So the script creates a list of all the cities on Craigslist (because CL doesn’t provide a clean, RESTful API that allows you to get this info easily) and then simply adds /cpg/ and /web/ to the end of that URL.
Then, on each link, it checks to see if the current link actually has gigs posted in that city. The reason for this is that whenever there is no gig posted in the current city, what CL does is shows gigs from “Nearby cities.” To prevent duplication, the script automatically checks for that and eliminates those cities that don’t have uniquely posted gigs. However, it does not eliminate a gig that has the exact text and is posted in two different cities – because, well, I hadn’t gotten there yet.
Once the script has a list of valid cities with gigs posted, then it starts to parse each of the links on the first page of those cities (i.e. up to 100 links in each city – CL does pagination by the 100 links) for keywords that I specified. The upside to only using the last 100 links in each city is that those are the most recent. The downside is that in active cities, the last 100 links aren’t always a good sample from the entire population.
For the Rails results, I have the following keywords: rails, (ruby on rails), (ruby on rails 3), (rails 3), (rails 2).
For the Ruby results, I have the following keywords: ruby, (ruby 1.8.7), (ruby 1.9.2), (ruby 1.9.3), ruby187, ruby192, ruby193.
The searches are case-insensitive, so any link containing Ruby, rUbY or RUBY will be found and included. I am trying to capture every permutation that someone would use “Rails” in a web dev sense.
The downside to this basic approach is that for technologies that share similar keywords – e.g. Ruby on Rails (the framework) and Ruby (the language) – there will be overlap. So in this case, the Ruby results contain a ton of Rails links, i.e. Rails is basically a subset of Ruby.
I have done my best to fine-tune as many obviously spam-like CL posts out of the results to really get at the legitimate posts.
So it is fair to say, I think, that these results give us a relative proxy for what the marketplace for freelance programming gigs is actually looking for.
Without further ado, here is the data and analysis.
- Cities Parsed: 720
- Total Gigs Found: 11,992-12,076 (scripts were run multiple times – for accuracy purposes – and the results returned were within this range)
- Time Script Takes to Run: 16 minutes-1.25 hours
Languages Searched For
Server-Side Languages and Frameworks
- C# (C-Sharp)
- CodeIgniter (PHP Framework)
- Django (Python Framework)
- Rails (Ruby Framework)
Client-Side Languages and MVC Frameworks
Batch 1 – Languages
On the server side, you can see that PHP wins by a long shot, with an almost Bolt-like performance, blowing everybody else out of the water. In fact, Ruby comes in at a paltry fifth place, Java comes in second, and Dot-Net and C# come in third and fourth, respectively.
Upon first glance at the results, the script says that Ember had an output of 14 gigs. However, because the keyword being searched for is “ember,” Ruby finds any string with a substring of “ember.” So it had 14 links with “Member” and “Membership” in the link title. Not one with Ember.js or the Ember we were looking for. So after I manually reviewed the links, 0 results were returned. So the only two client-side MVC frameworks that the script found that is in-demand is Backbone and Node. Both, just barely.
Below is the combined data:
This post is not meant to start a flame war between the various camps. It is just an unscientific analysis of what the general marketplace (using Craigslist as a proxy for that marketplace) is looking for in web development talent.
If you are considering learning one of these languages or frameworks, using what the marketplace requests is one factor to consider in your decision-making process. I wouldn’t necessarily encourage that, though. I certainly didn’t.
I chose Ruby and Ruby on Rails and love every minute of it. I would encourage you to try out various languages and see which you feel most comfortable with, because the vast majority of the time you spend in the language (assuming you really want to get better) will be non-billable stuff.
In the web applications I build for clients at 5KMVP, I use Ruby and Ruby on Rails because that’s what I love. Clients have been satisfied and seem to love it, too.
If you want me to do an analysis of anything else – say Mobile vs Server languages or anything else, let me know in the comments or drop me a line on Twitter.
If you found this interesting, you may find a piece I wrote about Dropbox, or a guide to understanding cashflow vs profit, similarly interesting.
You can find the Ruby script I created on GitHub, along with some sample output files that have a list of all the gigs generated for this article.
Do look around, and if you submit any pull requests for any improvements you may have, the Karma Fairy shall multiply your lineage ten-fold and your seed shall outnumber the celestial bodies.