Introduction / History

Here’s a guilty pleasure: http://dragcave.net, a “click to grow” adoptables site.  It’s a somewhat pointless and spammy website where users click on colorful pictures of eggs to collect them, then repost those images on other websites to gather unique views from anyone else who sees them.  After enough views, the egg hatches and grows up into a full-sized dragon.  So it’s got kind of a viral effect – the eggs get reposted to forum signatures etc. where other people click the eggs to find out just what the heck is the point, and then some of those people stick around to collect eggs and spread them even further.  It grows itself.  There are a ton of adoptables sites out there – Dragon Cave is just the one my wife found first, and by extension it’s the one I joined.

Somewhere in the history of adoptables sites, a user figured out that there was an easier way than spamming message boards to gather a large number of views: simply set up a huge “egg dump” area on a dedicated fan site, and ask users to come view the eggs and post their own.  For Dragon Cave three or four popular fansites had cropped up by mid-2009, designed by budding webmasters with varying degrees of aesthetic and coding skills.  That’s about the time that I decided to set up my own, just to see what was involved and how successful it would be.  So: Egg Drop Soup was born.

http://greg-kennedy.com/dragcave

The entire site is coded in PHP, and the earliest version screen-scraped everything from the bare HTML code to do the work.  It all felt like a hack, with the site design geared fully towards functionality and a minimum of flashy web content.  For whatever reason the site caught on .  A few whiz-bang features, plus the fact that several big-name fansites folded when an unannounced API change came through, boosted the site to about 5000 users at its peak.

Recent Progress

I’ve spent the past few days on a mild resurgence in EDS development.  Dragon Cave has an officially supported API which makes screenscraping obsolete, but I hadn’t ever ported over the pages to use the new features.  Since the HTML had changed in the meantime, this meant a lot of annoying bugs had crept in.  So step 1: port the code to use the API.  Once that was complete, I added a long-missing “Emergency Room” feature: this prioritizes the eggs in danger of expiring at the front of the rotation, so they gain a massive infusion of views and hopefully promote to adulthood (and immortality).  There are a couple of other features in the works, but I feel I’m nearing the end of what I set out to do with this project, and it’s going into maintenance mode unless some inspiration strikes me.

Okay, so this is a pretty boring post, but I wanted to make it for a couple of reasons.  First, it helps me to keep track of what I’ve been working on – this blog exists to categorize my projects, and since I’ve sunk a good number of hours into this one, it warrants a mention.

Second, I’m seriously considering open-sourcing the code to the site (BSD 2-clause license).  I don’t feel there is much danger in userbase loss to clone sites, because being an established name in the DC community is as much (if not more) a driver as features.  People simply gather where everyone else gathers, because that’s where the views are.  So if someone throws out “EDS 2” on stock code it’s no real threat to me.

Instead, I think this would help out a lot of struggling devs on the site by dropping a full-featured codebase on them.  I’ve seen at least one forum post where people are discussing the technical details of setting up a hatchery.  I think EDS is a perfect “skeleton code” example, because it really focuses tightly on making things work, and leaves the web design to the imagination… or the future adopters of the code.

Purpose

iPhoto Modified Importer is a tool to import digital photos to your iPhoto library as “modified” versions of original photos.  By default iPhoto only lets you import Originals.  This is a workaround to keep some existing linkage between Original and Modified versions on your disk, and bring that relationship into iPhoto.  It tricks iPhoto into thinking it is an external editor, then allows you to replace one photo with another from disk.

Background

Before I used a Mac, I organized and edited all my digital photos with Digikam on Linux.  My workflow looked something like this:

• Create a folder for the Album (e.g. “4th of July 2010”)
• Create a sub-folder beneath that called “Originals”
• Import all photos from the card to the Originals folder
• After editing an Original, save it to the top-level folder with the same filename.

By doing this, my release-ready photos were at the top level of every album, while Originals contained digital negatives and unedited photos for safekeeping.

Now that I am using a Mac I’d like to switch to iPhoto and get used to its workflow.  I have no problems letting iPhoto manage my library by itself, copying my photos into a consolidated location, etc.  But the first stumbling block I run into is: how can I preserve that linkage between digital originals, and the modified versions?

Most advice boils down to: “import them separately”.  But that leaves duplicates in the library, and worse it splits my workflow into “pre-iPhoto” and “post-iPhoto”, because any newly edited photos _will_ be linked.  Also, I can’t easily track which edited version goes with which original.  Since iPhoto uses its own database to keep relationships and photo info, there’s another problem: getting modified versions into iPhoto is hard.  You can’t just copy the files to the right location and expect them to be there.

Solution

Here is a solution that relies on iPhoto’s “Edit in External Editor” option.  Place the app somewhere on disk and then under iPhoto Preferences, change the Edit drop-down box to point to the app.  Now, when you double-click on a photo, it will start up iPhoto Modified Importer.

Use the Browse button to locate your target edited photo on disk, then hit OK.  The tool will “save” the image to the correct location, and exit.  iPhoto should then recognize that the file has changed and update the thumbnail, etc.

Warning

This tool is HIGHLY UNTESTED.  It doesn’t ask before saving over anything you’ve done, and it probably has potential to destroy already edited photos in your library.  BACK UP BEFORE YOU TRY TO USE IT.

Future Goals

There are a couple of additional items on the screen that I hope to one day turn into a useful feature.  Ideally you could set up a source directory first, and then iPhoto would report when a filename match is found, defaulting to that location.  This would speed up the import process.

Download

iPhoto Modified Importer 1.0 – zip file 2.6MB

This ZIP file contains the application, and the source code.  It requires XCode to build and… well, I really don’t even know what version of OS X it’s for.  10.6 at least.  Cocoa is a language designed by moon men for the sole purpose of being impossible to understand by Earthlings, but maybe it will make more sense to someone else than it does to me.

Overview

MIDI-Carver is a tool for extracting (partial or complete) MIDI files from a binary block of data.  It can be used against a solid data file – e.g. an ISO image – to attempt to retrieve MIDI data into standalone files.  It also can parse images of damaged filesystems and makes some attempt at recovery even when the source has damaged one or more tracks, or obliterated the MIDI header.

If your disk has crashed, and you need your music back, midi-carver may be your best bet.

Introduction

Data recovery is tricky business.  The best solution is not to have to do it at all: MAKE REGULAR BACKUPS.  But if you have not done this – or even if you have and are unlucky enough to have bad backups – then it becomes a much more painful process.

Starting with an image of the damaged source material (create using ddrescue or some other tool), the next best thing is to mount it locally.  If that fails, one should try to somehow rebuild / recover the FAT table, so the disk image may become usable again.  If that fails, the last resort is “data carving”: searching the binary image for some indicators of file types, and then extracting them to disk.  Some popular tools for doing this are foremost, scalpel, and PhotoRec.

midi-carver is a data carver specifically designed to recover MIDI files.  It has a more intelligent extractor, in that it can skip damaged information and create MIDI files with the remaining good stuff.  It can also, if you’re lucky, locate deleted items on the disk and recover them.

Features

• Attempts extraction of MIDI files from any binary blob
• Locates “orphaned” MIDI tracks (where the header is missing) and creates a new MIDI header to contain the tracks
• Copes gracefully with missing tracks by re-writing the header to the correct structure
• Detects MIDIs which were overwritten by others, and truncates / splits into two separate tracks

Usage

The hardest part of this would be compiling midi-carver.  If you’re on Linux, FreeBSD or OSX this is easy.  Download midi-carver.c, open a terminal and:

gcc -o midi-carver midi-carver.c

This will compile midi-carver into an application which can be run on a block file, like this:

./midi-carver <filename>

midi-carver will create a folder titled “mcut-out” in the same directory as <filename>, and save any MIDIs it finds into that folder.

Of course, you’ll need something to work on.  I highly recommend using ddrescue on your damaged disk to create images.  It’s very effective at getting the most undamaged data off the disk possible.  Once you’ve done that, run midi-carver as described above, and look in the mcut-out to see if anything detected was usable.

Limitations

• midi-carver does not handle source fragmentation.  This is ordinarily not a problem with iso images, but it is an issue with hard disks (and especially, floppy disk images) where files have been deleted and then added frequently.  In this case the file system may break the files into blocks and rearrange them to effectively use all the disk capacity.  midi-carver cannot read (or recreate) a file allocation table, so it cannot determine how to follow fragments.  There is hope for a “last-block fragmented” recovery feature, but its usefulness would be slim.
• midi-carver will happily terminate tracks without checking the actual MIDI events in the track.  This produces a MIDI file which is structurally valid, but which media players may choke and die on, if the MIDI events in the track are not of the proper format.
• There are a few pathological cases which may cause midi-carver to do the wrong thing – e.g. a tiny MIDI file embedded within a huge MIDI track will be missed.

Future Plans

Fix limitations 2 and 3 above!  This would mean adding a real track validator that could follow MIDI events in a track and then terminate it as soon as things appear to go sour.

Download

midi-carver.c – 12.1 kb, version 1.0.

License

It’s just 3-clause BSD.

Copyright (c) 2010, Greg Kennedy

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Neither the name of midi-carver nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.