Recently I was interested in TuneIn (http://tunein.com) radio service which provides access to radio stations around the globe. In this post I will uncover statistics behind TuneIn radio stations and show how to use the script to do some cool thinks with the TuneIn API.

Code: surfsnippets/stationdigger

Output: station_lang.json (6.9 Mb), station_genre.json (6.4 Mb)

I wrote a script stationdigger.py which uses TuneIn API http://inside.radiotime.com/developers/api/opml by traversing tree of links and collecting information about the stations.

TuneIn radio API provides both OPML (XML-like) and JSON formats. You don’t have to have any credentials to use this service. But if you start to use the service frequently you will end up getting 403 Forbidden error. Let’s try to make the first request:

http://opml.radiotime.com/Browse.ashx

Your response will be in default OPML format:

 1 <opml version="1">
 2     <head>
 3         <title>TuneIn</title>
 4         <status>200</status>
 5     </head>
 6     <body>
 7         <outline type="link" text="Local Radio" URL="http://opml.radiotime.com/Browse.ashx?c=local" key="local"/>
 8         <outline type="link" text="Music" URL="http://opml.radiotime.com/Browse.ashx?c=music" key="music"/>
 9         <outline type="link" text="Talk" URL="http://opml.radiotime.com/Browse.ashx?c=talk" key="talk"/>
10         <outline type="link" text="Sports" URL="http://opml.radiotime.com/Browse.ashx?c=sports" key="sports"/>
11         <outline type="link" text="By Location" URL="http://opml.radiotime.com/Browse.ashx?id=r0" key="location"/>
12         <outline type="link" text="By Language" URL="http://opml.radiotime.com/Browse.ashx?c=lang" key="language"/>
13         <outline type="link" text="Podcasts" URL="http://opml.radiotime.com/Browse.ashx?c=podcast" key="podcast"/>
14     </body>
15 </opml>

This is where the story begins. Traversing tree of links with type="link" you can reach links with type="audio" and collect information about the stations. For example, one branch of the link tree can look like:

http://opml.radiotime.com/Browse.ashx
+-- <outline type="link" text="Local Radio" URL="http://opml.radiotime.com/Browse.ashx?c=local" key="local"/>
    +-- <outline type="audio" text="San Diego Police Scanners: 2 (Police)" URL="http://opml.radiotime.com/Tune.ashx?id=s103434" bitrate="32" reliability="86" guide_id="s103434" subtext="English" genre_id="g2741" formats="mp3" item="station" image="http://radiotime-logos.s3.amazonaws.com/s103434q.png" now_playing_id="s103434" preset_id="s103434"/>

My script stationdigger.py just makes this task automatic and more error prone.

Results Format

I collected statistics for radio stations arranged by language and genre. The results are stored in files station_lang.json (6.9 Mb) and station_genre.json (6.4 Mb) in JSON format for stations by language and genre correspondingly.

Stations by language has the format:

 1 {<language id>:
 2     {"name": <language name>,
 3     "stations":
 4         {<station id>:
 5             {"name": <station name: {text}>,
 6             "description": <station description: {subtext}>,
 7             "url": <station url: {URL}>,
 8             "image": <station image: {image}> ,
 9             "genre": <genre id: {genre_id}>
10             },
11         // other stations
12         }
13     }
14 // other languages
15 }

For example:

 1 {"l191":
 2     {"name": "Afrikaans",
 3      "stations":
 4         {"s101967":
 5             {"name": "Radio Namakwaland (Vredendal, South Africa)",
 6              "description": "Afrikaans",
 7              "url": "http://opml.radiotime.com/Tune.ashx?id=s101967&filter=l191",
 8              "image": "http://radiotime-logos.s3.amazonaws.com/s101967q.png",
 9              "genre": "g133",
10             },
11         // ...
12         }
13     }
14 // ...
15 }

Parameter url holds the link to the station stream which you save on your machine and open content of the file with some media player.

For example, open the link in the browser:

http://opml.radiotime.com/Tune.ashx?id=s124197&filter=l1

and in the Tune.ashx file you will find station stream which looks something like:

http://lush.wavestreamer.com:3680/

Open this link, for example, in Banshee Media Player (Media/Open Location) available in Ubuntu as a default media player and enjoy the station :).

Code Analysis

As I mentioned before, stationdigger.py code traverses tree of links and collects information about the stations using DFS algorithm. At first I thought that it will be pretty straightforward to implement the algorithm, but later I realized that I have to handle several issues specific to TuneIn service to properly do it.

TuneIn radio service significantly limits number of calls without partnerId passed (http://inside.radiotime.com/developers/api/opml/security-model#limits). It also has limits for calls even with partnerId but the limit is larger than without partnerId. Once the limit is reached you will start getting 403 Forbidden error for a day. Because of the large number of calls needed to collect the data it is good to have partnerId :) or you can wait for a long time. To collect stations using partnerId it took me about 4-5 days. It is also recommended to pass the serial parameter which can be pretty much anything.

The core of the algorithm is method traverse():

 1 def traverse(self, url):
 2     "Traverses tree of links and populates stations using DFS algorithm"
 3     if not url or (self.numTotal and self.numTotal<self.num):
 4         return
 5 
 6     items = self.make_call(url=url)
 7     (audios, links) = self._separate(items)
 8     for item in audios:    # check audio links first: playable content
 9         if item.has_key("guide_id") and \
10            not self._stations.has_key(item.get("guide_id")):  # ignore duplicates
11             self._stations[item.get("guide_id")]    = self.to_station(item) # set station
12             self.num += 1
13             if self.numTotal and self.numTotal<self.num:
14                 return
15 
16     for item in links:      # check links
17         self.traverse(item.get("URL"))

In this method it makes call to specified url and returns list of items in the tag. Then it separates links and audio links. Links can later be traversed down the link tree. Audio links are what we are after: they are used to collect information about the station.

The code supports the following features.

Can dig stations both for one and all languages

Because digging stations is a relatively long process, when something happens during the data collection you need a way to restart from some intermediate step. For this purpose the program stores stations in junks: one language in a separate file in a format lang<number>.json

These files are stored in stations directory and then later merged with merge_langs.py script into a single file: station_lang.json. Besides collecting stations for all languages at once you can collect stations just for one language by passing a language tuple.

1 def dig_stations_lang(self, filename=None, lang_tuple=None):
2     "Diggs for stations with language filters"
3     if lang_tuple:
4         filters = [lang_tuple]  # one language
5     else:
6         langjs = self.get_lang() # all languages
7         filters = self.sorted_tuple(langjs)
8     self._dig_stations(filters, {"c": "lang"}, filename)

Can dig specified number of stations

You don’t need to collect all stations. Setting limit on total number of stations (self.numTotal) will collect at most this number of stations for the filter passed. See implementation of method traverse(self, url). For example, if filter has two languages then stations will get populated starting from the first language.

Digs only unique stations

To make sure that stations are not duplicated for the same language the program checks if the station has been added before:

1 def traverse(self, url):
2         ...
3         if item.has_key("guide_id") and \
4            not self._stations.has_key(item.get("guide_id")):  # ignore duplicates
5             self._stations[item.get("guide_id")]    = self.to_station(item) # set station
6         ...

Handles 403 Forbidden error

This error was a frequent companion during collecting data. This just means that limit of using the TuneIn service has been reached. It not handled properly you will end up either with non-complete stations or no stations at all. I did some experiments with this error and figured out that your program can be rejected either for one day or for a few minutes (not sure how this happens). So I developed some mechanism that handles this error which can be reduced to just one word: persistence! When program receives 403 Forbidden error it waits for some time and then tries to perform the same call again until it receives result other than this error.

 1 def make_call(self, url):
 2     "Makes call and returns response in json format"
 3     if not url:
 4         return None
 5     nurl    = self._set_format(url)
 6     if nurl in self._urls:  # Avoid circular reference
 7         return None
 8     result  = -1    # to enter the loop
 9 
10     while (result == -1):
11         try:
12             req         = urllib2.Request(nurl, headers=self._headers())
13             response    = urllib2.urlopen(req)
14             page        = self._decompress(response.read()) # decompress
15             pdict       = json.loads(page)
16             result      = pdict.get("body")    # interested in "body" section only: should be list
17             self._urls.add(nurl)    # URL is processed
18         except urllib2.HTTPError, e:
19             if e.code == 403:
20                 result = -1
21             else:
22                 result = None
23         except:
24             result = None
25 
26         if result == -1:
27             print "Waiting %s --> %s" % (self.num, nurl)
28             time.sleep(WAIT_TIME)
29     return result

Avoids circular reference

From experience I found out that some links might have circular reference: referring to itself in one or several hops. Once the program enters the circular reference it can get stuck forever! Simple check can save you out of trouble:

1 def make_call(self, url):
2     ...
3     if nurl in self._urls:  # Avoid circular reference
4         return None
5     ...

Response compression

To optimize the stream of responses I added header to request encoded content supported by most of the popular web servers and decompress it on the program’s side:

Accept-Encoding:  gzip

Here is the method which performs decompression:

1 def _decompress(self, s):
2     "Decompresses string"
3     sio     = StringIO(s)
4     file    = GzipFile(fileobj=sio)
5     return file.read()

Another issue I had related to TuneIn responses is embedded tags which looked like in xml format:

<outline text="Stations" key="stations">
    <outline type="audio" text="Radio Pretoria (Kleinfontein)" URL="http://opml.radiotime.com/Tune.ashx?id=s10616&partnerId=my_partner_id&serial=123&filter=l191" bitrate="32" reliability="96" guide_id="s10616" subtext="Tienie se musiek" genre_id="g255" formats="wma" show_id="p149910" item="station" image="http://radiotime-logos.s3.amazonaws.com/s10616q.png" current_track="Tienie se musiek" now_playing_id="s10616" preset_id="s10616"/>
</outline>

To handle this issue I traverse also these subtrees:

 1 def _traverse_outline(self, items, audios, links):
 2     "Traverses children outline elements and populate audios and links lists"
 3     if not items:   # No items
 4         return
 5 
 6     for item in items:
 7         if item.has_key("children") and item.get("children"):
 8             self._traverse_outline(item.get("children"), audios, links)
 9         elif item.get("type") == "audio":
10             audios.append(item)
11         elif item.get("type") == "link":
12             links.append(item)

Mimics headers of Chrome browser

For some reason passing browser headers with the call doesn’t reject calls as fast as without the headers. So I decided to explore Chrome browser headers and pretend that the calls are made with the browser. Here is the example of the headers presented as a Python dictionary:

1 headers = {
2     "Connection":       "keep-alive",
3     "Cache-Control":    "max-age=0",
4     "User-Agent":       "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.71 Safari/534.24",
5     "Accept":           "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
6     "Accept-Encoding":  "gzip",
7     "Accept-Language":  "en-US,en;q=0.8",
8     "Accept-Charset":   "ISO-8859-1,utf-8;q=0.7,*;q=0.3"
9 }

Using Script

Before you start running program it is recommended to set config.txt file first:

[general]
partnerId =
serial =

To collect stations for all languages you need to iterate over sorted languages:

1 sd      = StationDigger()
2 langjs  = json.load(open("generated/lang.json"))
3 lang    = sd.sorted_tuple(langjs)
4 for n in range(len(lang)):
5     sd.dig_stations_lang("stations/lang%s.json" % n, lang_tuple=lang[n])

so the session for calls without partnerId passed looks like:

Finished: Aboriginal, total: 2
Finished: Afrikaans, total: 12
Waiting 15 --> http://opml.radiotime.com/Browse.ashx?id=g373&filter=l210&render=json

If you want to just dig stations for n-th language then the script will do the work:

1 sd      = StationDigger()
2 langjs  = json.load(open("generated/lang.json"))
3 lang    = sd.sorted_tuple(langjs)
4 lang_tuple  = lang[n]
5 sd.dig_stations_lang("stations/lang%s.json" % n, lang_tuple=lang_tuple)

Other Useful Scripts

Besides implementing stationdigger.py I also implemented some useful scripts performing some operations on generated files.

merge_langs.py

This script merges files with stations by a single language into one file: station_lang.json. You need to have language files already generated from stations collections. The file structure should look like:

stations/
    lang0.json
    lang10.json
    ...

lang2genre.py

It turns out that the stations by language can be converted into stations by genre because most of the stations have also genre field. This way one can collect statistics of station distribution by genre. lang2genre.py script converts station_lang.json into station_genre.json.

status_generator.py

This script uses station_lang.json and station_genre.json to generate cumulative statistics on languages and genres. The result is later used to draw plots.

draw_plots.py

Script for generating pie and histogram plots for languages and genres.

Plotting Results

This is the final step in our analysis. We take data generated by status_generator.py script and use draw_plots.py to create plots for languages and genres. To draw plots I used matplotlib – an advanced plotting Python library extensively used by scientists. I used pie and histogram plot types to represent data in two different ways. I wasn’t happy with the default color schema so I wrote function which generates html-type colors from HSV colors.

Here are the results I obtained:

Results Analysis

It is no surprise that TuneIn radio service is a US based company so probably they are more biased to English language. English speaking stations comprise 42.7% of all provided stations. Next popular language, Spanish and Portuguese, have about 10% each.

Genre distribution is not that narrow as for languages. Majority of stations feature pop genre (Top 40-Pop: 15.4% and Adult Contemporary: 6.0%) which is also predictable. Detailed statistics can be found in status_lang_sort.txt and status_genre_sort.txt.

Number of Stations by Language

Number of Stations by Language

Number of Stations by Genre

Number of Stations by Genre