Downloading from Soundcloud

A couple years ago, downloading a song from Soundcloud used to be pretty trivial. Their server would send you the complete 128kbps MP3, and then the local embedded control would allow you to seek at will. Because the file arrived in one large chunk, it was both easy to identify in cache, and easy to copy somewhere else to play back. Sometimes this still works… You’ll know by looking at the dev console, and see if it shows a huge MP3 file transfer. If so, you’re in luck! Copy it from the cache and you’re set.

Evidently they’ve changed this practice for other tunes, possibly to improve the latency of seeking at random in tracks, or possibly because they don’t want people getting music they shouldn’t be able to get. You can get Greasemonkey scripts which put the download button back, but these simply fire the URL off to an third-party site which “somehow” reconstructs the song and then sends it back your way. Very black-box magic stuff indeed.

However – If you can stream it, you can download it, as they say. Let’s take a look at how Soundcloud actually gets a song to you, and see if we can still figure out how to download something we may not really be allowed to.

Start the browser’s Developer Console and then browse to a song you want to hear. Keep an eye on the “network” activity, it will give you clues as to what is actually going on. As the song begins playing, you’ll see a lot of small network requests to magically named files:
mini_files

This seems promising. Download one or two and run “file” on it, and you get:
$ file *
c5f47vUnF3Ow.128.mp3?f10880d39085a94a0418a7e163b03d5226edfe2317e6aa1445547d76cf23a7ca5b08b0b9169eed2c0a13f681ab93c51d8e788dcaa887622ee2905d7463e4fd982e918b5b687caf75047026a3429731c5010a16: MPEG ADTS, layer III, v1, 128 kbps, 44.1 kHz, JntStereo
c5f47vUnF3Ow.128.mp3?f10880d39085a94a0418a7e167b03d5249919aaf544816306c9a5e3ca05a129454accfdda2750c51705ac2f68f036a37b2c482058312ab10625db87a6e3ab6dc1d1631dbd883a3f38786db484e66359daf667314eb8f03: MPEG ADTS, layer III, v1, 128 kbps, 44.1 kHz, JntStereo

Okay, so Soundcloud has broken the file into parts and is playing them back in sequence. You can pop one of these into your media player and listen to a portion of the song. We’re close, but how do we know where to find all these parts and put them together in order? Easy: there’s an m3u8 that has that for you – check the Dev Console again! Soundcloud’s player is using this to fetch the data in order from various URLs, and then stream it to you. For example, something like this:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:1.985272,
https://ec-hls-media.soundcloud.com/media/0/31762/c5f47vUnF3Ow.128.mp3?f10880d39085a94a0418a7e163b03d5226edfe2317e6aa1445547d76cf23a6c6277ac4a31c0a01842ddc2964536e126c07aa07555cce1108baa06b42a8a56548b485330689f4477467fbe84e3701184db02ae8
#EXTINF:2.977908,
https://ec-hls-media.soundcloud.com/media/31763/79410/c5f47vUnF3Ow.128.mp3?f10880d39085a94a0418a7e167b03d5249919aaf544816306c9a5e3ca05a13988480bef0d85fac22fe42482ceefe72b45f93798b344c894177b5503fecfe001728ea53a521df650649905f1f820170622633f8408d974a
#EXTINF:4.989302,
https://ec-hls-media.soundcloud.com/media/79411/159240//c5f47vUnF3Ow.128.mp3?f10880d39085a94a0418a7e168b03d52f1af9dc5c031765e833721190ff39af5838b01fab4994aa8e1cbf7325eac7c17bbc32c9af8a4f995094f28154eaf2285d51dae2371e29d5df21535551f28bc9d3d6cdf11c7f30fd3
#EXTINF:9.978604,
https://ec-hls-media.soundcloud.com/media/159241/318900/c5f47vUnF3Ow.128.mp3?f10880d39085a94a0418a7e169b03d52a8b3c33b59894223455854e2f950e750c62859d445307c2908ba5ad3f2b1c73b802a1616248b94bd6c5babf25058d658ad1566709910633c2cd63c67371fbb6692e6db87e986a283f0
...
#EXTINF:9.978604,
https://ec-hls-media.soundcloud.com/media/3352449/3512109/c5f47vUnF3Ow.128.mp3?f10880d39085a94a0418a7e061b03d52eb7d518bdc510e28891131df942c66df2d11c65c5f6f65b400607098a9a36cb891691d16355d126d2ecbc6451bb48f82561d15b81d006b4104caff07fffa8361c42652b9341faefa8f4513
#EXTINF:5.642352,
https://ec-hls-media.soundcloud.com/media/3512110/3602388/c5f47vUnF3Ow.128.mp3?f10880d39085a94a0418a7e061b03d52eb7d518bdc510e28891131df942c66df2d11c65c5f6f65b400607098a9a36cb891691d16355d126d28cfc6401ebd8f8214c9a4dd73e90391e79897cc892a72bcd4883580fdef67b3cf0e33
#EXT-X-ENDLIST

An interesting aside, it seems these URLs time out after a short period of time, leading to 403 Forbidden errors if you try to access it again. No doubt these huge URL parameters point to some browser session or timestamp which becomes invalid after a while. If that happens, reload the page and start playing again to generate new files.

So to recap all this: we need to

  • take the m3u8 file,
  • retrieve each mp3 segment,
  • and concatenate them together.

Getting the m3u8 programmatically is hard, so just copy it from the browser : ) And to put these together you’ll need mp3cat installed – see http://tomclegg.ca/mp3cat for info.

#!/usr/bin/env perl
open(FP,"playlist.m3u8") or die "can't open playlist: $!";

my $piece = 0;
while( < FP > )
{
chomp;
next if ($_ =~ m/^#/);
$filename = sprintf('outdir/%02s.mp3',$piece);
print `wget --no-check-certificate -O $filename $_`;
$piece++;
}

print `cat outdir/*.mp3 | mp3cat - - > output.mp3`;

Arduino Raytracer

Every programmer worth his salt has written a raytracer of some sort. It’s one of the classic “recursion exercises”, with a fair amount of math to wrap your head around. I’m not interested in writing one to run on the PC – there are already far more capable ones that can run on pretty much any hardware, complete with optimizations and features I couldn’t hope to implement in my lifetime.

Instead, I decided to write a raytracer which targets the Atmega 328 microcontroller in my Arduino! In the end, I succeeded in banging one together that calculates ray-triangle intersection, and each triangle can have a material attached with ambient / diffuse / transparent / reflective values and an RGB color. The scene is compiled in with a .h file (stored in PROGMEM), and outputs a PPM image pixel-by-pixel over the serial port.

The same engine can be used on the PC with a few modifications, so I was able to benchmark the performance of the Arduino in comparison with a modern laptop. The “detailed scene” rendered a 640×640 image in 263 seconds on the laptop. The same scene, scaled to only 64×64, still took 4008.471 seconds… in other words, the PC outperformed the Arduino by a factor of ~1500.

I posted the code, and further write-up, on the Arduino.cc message board at this link: http://forum.arduino.cc/index.php?topic=281076.0. There are a lot of features that COULD be added here, but none that I actually intend to do. If I need a raytracer in the future, I’m just downloading POVRay : )

pfSense on Firebox X500

IMG_4881 Previously I had run pfSense as my firewall / router on a discarded Celeron 700mhz machine, with a lot of spare NICs in the PCI slots. Browsing around the local Goodwill and I stumbled upon this Watchguard Firebox X500 for $10. Finally, real rackmount gear for my 2×4 server rack!

I’m sure whatever software comes on this thing is great and wonderful, but I didn’t even try to use it. First step if you have one of these – hop on over to the pfSense Firebox wiki page and do some research. Popped open the case, swapped in a 2gb CF card with the pfSense no-VGA embedded image, and set to configuring it. The most difficult part of all this was getting a USB to Serial cable that worked… and then a Null Modem cable… and then having to do the configuration in a virtualized Windows XP machine because there are no OSX drivers for the WCH341 chip in my cheap eBay usb-serial converter.

Now that it’s all set up I can easily access via ssh or http, so serial access isn’t as important. The hardware in this is just a 1.2ghz Celeron, with 10/100 Ethernet ports and some 256mb RAM. Obsolete, yes, but that red paint job never goes out of style.

Wacom ArtPad II (KT-0405-R) to USB

1119141209 (1) Six or so years ago, I picked up a Wacom ArtPad II digitizer from Freecycle. It came with serial cable and power brick, but no pen. These devices aren’t like an iPad or whatever (which use a capacitative touchscreen) where you can use your finger or a cheap stylus. Wacom pens have built-in smarts to communicate with the tablet and relay pressure information, pen/eraser mode, button click, etc. Without the pen, it’s downright useless. A few eBay searches revealed that pens were crazy expensive to replace at the time. Into the spare bin it went.

Recently I had an interest in getting this working again. I did more research and found that Wacom pens are actually somewhat interchangeable – in this case I can use any “Penabled” pen on it, and it’ll probably work. The new Samsung Galaxy Note S tablet uses Wacom styluses and the replacements are cheap – $8 on eBay, so I ordered one. Holding the pen over the tablet causes the LED to switch from orange to green, which was good enough for me to figure it’s working.

New problem – I have no machines with a serial port any more, and even if I did, Wacom drivers don’t support serial tablets any more. The solution comes from the WaxBee project, which uses a Teensy 2.0 to sniff old-protocol tablet packets and re-encode them to emulate an Intuos2 (a modern USB tablet with driver support). Great! Ordered a Teensy and cracked open the tablet to set to work.

First, I knocked off the serial connector and voltage regulator (soldered across that instead), as the USB will provide the +5V needed to drive the tablet. Next step is removing the ADM202 TTL-to-RS232 chip, where things took a disastrous turn: my overeager Dremel application removed the chip as well as the pads it sat on, and gouged out some of the PCB to boot. Crud, now I have to follow traces to figure out where it was supposed to go. Took a couple high res photos and gave up for the night. (If you need more info – circuit board photos, etc – check out this series of posts: http://forum.bongofish.co.uk/index.php?topic=2088.msg20757#msg20757)

test The actual solution turned out not so bad: had to lift three pins on the Motorola chip and solder those to the Teensy. The hid_debug tool from pjrc (and associated WaxBee firmware) was invaluable in figuring out when I had the wires going the right way. Once that was done, superglue secured the wires and electrical tape held the Teensy. I cut a chunk off the side of the enclosure for the USB micro-port to fit, and drilled a hole for the Teensy button if I ever need to reprogram it. Covered with electrical tape, plugged it, and miraculously it all worked!

Crazy Taxi Garmin GPS Voice

Here is a project I’d been talking about for years, but never actually pulled it off until recently. It’s a voice set for a Garmin GPS, which replaces the default “narrator” with the obnoxious announcer from Crazy Taxi.
All voice clips were pulled from the PC version of the game, which had them in convenient .wav format already. Some trim / normalize with Audacity and import into Garmin Voice Studio to assemble together. I wanted to do more than just the announcer, but didn’t have any luck with that… there just weren’t enough clips to make it happen.

Here’s a video of the system in action:

I hosted all the downloadables on archive.org. You can get it from here:
https://archive.org/details/Garmin-CrazyTaxiVoice