Auditing ROM collections has been made much simpler over the years thanks to concerted efforts of many cartridge purchasers, dumpers, cataloguers, and coders. Using a combination of RomCenter (or ClrMamePro) and a No-Intro .Dat file, one can quickly fix an entire collection in one go: repairing names, verifying regions, checking for problem ROMs, producing have-lists, etc etc.
NES roms throw a particular curveball because they carry a 16-byte header with a little info about the board within the cart. Emulators need this info, but it isn’t verified by No-Intro since it’s not part of any real dump. As a result iNES headers remain missing, incorrect, or filled with crap data. A new repair tool is needed.
ines-fix.py is a Python script which can repair defunct iNES headers using an external cart database. It does this by extracting the data portion from the file, calculating a crc32, looking for the crc the xml file, and then rebuilding a header based on that info. If the old and new headers differ, the file is overwritten with a fixed version.
Be careful with this tool: it’s sort of a blunt object, and may trash your rom collection if not used carefully. Make backups. In particular it writes only iNES v1 headers, expecting emulators to cope with the shortcomings of this format.
Output samples:
-> No change to existing header
CRC check: 4318A2F8 Found CRC match: Barker Bill's Trick Shooting *** Header unchanged: not writing replacement file. ----------------------------------------------------
-> Adding a battery
CRC check: 1F6EA423 Found CRC match: Baseball Simulator 1.000 *** HEADER UPDATED *** oldHeader: 4e45531a081010000000000000000000 newHeader: 4e45531a081012000000000000000000 All done. Wrote new file roms/Baseball Simulator 1.000 (USA).nes
-> Correcting a mapper, vertical mirroring, and removing “DiskDude!”
CRC check: 9BDE3267 Found CRC match: Adventures of Dino Riki *** HEADER UPDATED *** oldHeader: 4e45531a0204004469736b4475646521 newHeader: 4e45531a020431000000000000000000 All done. Wrote new file roms/Adventures of Dino Riki (USA).nes ----------------------------------------------------
-> Standardizing on “horizontal mirroring”
CRC check: 3ECA3DDA Found CRC match: Bases Loaded 3, Ryne Sandberg Plays *** HEADER UPDATED *** oldHeader: 4e45531a101041000000000000000000 newHeader: 4e45531a101040000000000000000000 All done. Wrote new file roms/Bases Loaded 3 (USA).nes ----------------------------------------------------
-> Adding a missing header
CRC check: 50CCC8ED Found CRC match: Battleship *** HEADER UPDATED *** oldHeader: newHeader: 4e45531a020430000000000000000000 All done. Wrote new file roms/Battleship (USA).nes ----------------------------------------------------
Here’s the script:
#!/usr/local/bin/python
"""
Converts NES ROMs to iNES format, applying correct iNES header.
Usage: ines-fix <infile>
Supported infile formats are .nes, .pas (headerless .nes)
ROM data is recognized by CRC32 using BootGod's master XML database
so make sure you have a local copy
"""
import sys
import struct
from binascii import crc32
from xml.etree import ElementTree
##### String holding location of cart db
cart_xml = "NesCarts (2011-09-10).xml"
# uncomment next line to use Nestopia's DB instead
# cart_xml = "NstDatabase.xml"
# Other required vars
i_fmt = 'p'
blob = None
found = 0
oldHeader = ""
# Parse command-line
if (len(sys.argv) != 2):
print "Usage: " + sys.argv[0] + " <infile>"
sys.exit(0)
# Open rom database
#print "Attempting to open cart db " + cart_xml
tree = ElementTree.parse(cart_xml)
#print "DB opened!"
# Attempt to open supplied rom file
try:
with open(sys.argv[1], "rb") as f:
tag = f.read(4)
f.seek(0)
if (tag == "NES\x1A"):
i_fmt = 'i'
oldHeader = f.read(16)
# print "Detected " + i_fmt + " format for input file"
blob = f.read()
except IOError as (errno, strerror):
print "Error opening " + sys.argv[1] + ": "
print "I/O error({0}): {1}".format(errno, strerror)
sys.exit(2)
if (len(blob) > 0):
format_crc32 = format(crc32(blob) & 0xFFFFFFFF, '08X')
print "CRC check: " + format_crc32
# go look up crc32 in db
game_list = tree.findall("game")
for game in game_list:
cart_list = game.findall("cartridge")
for cart in cart_list:
if (cart.attrib.get('crc') == format_crc32):
found=1
break
if (found):
break
if (found == 0):
print sys.argv[1]
print "*** CART NOT FOUND IN DB"
print "----------------------------------------------------"
sys.exit(2)
print "Found CRC match: " + game.attrib.get("name").encode('ascii', 'ignore')
#ElementTree.tostring(game)
# retrieve data from game
board = cart.find("board")
mapper = int(board.attrib.get("mapper"))
prg_size = 0
prg_list = board.findall("prg")
for prg in prg_list:
prg_size = prg_size + int(prg.attrib.get("size") [:-1])
chr_size = 0
chr_list = board.findall("chr")
for chr in chr_list:
chr_size = chr_size + int(chr.attrib.get("size") [:-1])
battery = 0
wram_list = board.findall("wram")
for wram in wram_list:
if (wram.attrib.get("battery") is not None):
battery = int(wram.attrib.get("battery"))
chip_list = board.findall("chip")
for chip in chip_list:
if (chip.attrib.get("battery") is not None):
battery = int(chip.attrib.get("battery"))
mirror_4 = 0
mirror_v = 0
pad = board.find("pad")
if (format_crc32 == "CD50A092" or \
format_crc32 == "EC968C51" or \
format_crc32 == "404B2E8B"):
mirror_4 = 1
elif (pad is not None):
# the "h" pad means "v" mirror
mirror_v = int(pad.attrib.get("h"))
mapper_lo = mapper & 0x0F
mapper_hi = mapper & 0xF0
newHeader = "NES\x1a" + \
struct.pack("BBBB", \
( prg_size / 16 ), \
( chr_size / 8 ), \
(mapper_lo << 4) + (mirror_4 << 3) + (battery << 1) + (mirror_v)
, \
(mapper_hi) ) + ( '\0' * 8 )
if (newHeader != oldHeader):
print "*** HEADER UPDATED ***\noldHeader: " + oldHeader.encode('hex')
print "newHeader: " + newHeader.encode('hex')
# write new file
try:
with open(sys.argv[1], "wb") as f:
f.write( newHeader )
f.write( blob )
except IOError as (errno, strerror):
print "Error opening " + sys.argv[1]
print "I/O error({0}): {1}".format(errno, strerror)
sys.exit(2)
print "All done. Wrote new file " + sys.argv[1]
else:
print "*** Header unchanged: not writing replacement file."
print "----------------------------------------------------"
For those having problems with copy-paste, you can download a .zip containing the script here:
ines-fix
find . -type f -name ‘*.nes’ -exec ./ines-fix.py {} \;
or in a Windows command prompt use the following:
FOR /R %v IN (*.nes) DO “C:\fix\ines-fix.py” “%v”
line 71 should be sys.argv[1]
the nes roms i’ve tested so far have all been not found in the DB…they’re from the mid 90s
Thanks, made the change. You might try Nestopia’s DB xml, it is more recent than BootGod’s.
Is there a way you could include the python script file for download? If I just copy and paste it I lose the syntax completely. Thanks
Ok, I managed to do it. Thanks anyway. By the way, in case people are struggling, I couldn’t get the script to work with the latest version of python, I had to uninstall it and install the older 2.7.3 version before it started to work.
Oh that’s helpful, but how in the hell did you download the script without loosing the syntax or it being screwed up. I tried using View Source and found it doesn’t properly format when viewing that either!
I added the script in a .zip file as an attachment to the original post, give that a try.
Thank you Greg for this script! I had a complete set of NES ROM files formatted for the latest version of the MESS emulator. Each file is a .zip that contains the PRG and CHR file and there is no header. To convert such files into .nes files with header that work in the emulator BSNES I did these two steps:
step 1: unzip and binary join the PRG and CHR files. In Windows the command line command is:
copy /b “something prg” “something chr” somefilename.nes
step 2: get the nescarts xml from http://bootgod.dyndns.org:7777/ and then run your python script on the somefilename.nes file
In case anyone else comes here reading this and the bootgod site is down: I think it will be possible to adapt the script to instead use info files from the official MESS project. I haven’t tried doing so but if you, dear reader, are in a bind this might be worth a shot. Those files are here
http://git.redump.net/mess/plain/hash/nes.hsi
http://git.redump.net/mess/plain/hash/nes.xml
the .hsi includes the CRC ines-fix.py uses for identifying the game and the .xml includes the CRC and details for the PGR and CHR
Oops, a typo. The correct Windows command line must include a plus sign, like this:
copy /b “something prg” + “something chr” somefilename.nes
Any newer revisions of this script? I’m having some issues running it in Windows or in MinGW. I do have a Linux box and can try there, but I’d love to see a more up-to-date version. I have some files with the TNES header that I’d love to convert to iNES header ROMs. I was going to use this template as a program, if that was okay. It just doesn’t seem to run correctly. In MinGW-64-bit, I’m running Python 2.7.11.
I’ve temporarily deleted the header from one of the ROMs I’ve extracted from a decrypted 3DS ROM. I run the script as follows:
$ /usr/bin/python2 ./ines-fix.py
Usage: ./ines-fix.py
So that works.
Now I try passing a headerless ROM to it….
$ /usr/bin/python2 ./ines-fix.py SMBUSAtest.nes
CRC check: 80FB7E6B
Traceback (most recent call last):
File “./ines-fix.py”, line 82, in
print “Found CRC match: ” + game.attrib.get(“name”).encode(‘ascii’, ‘ignore’)
AttributeError: ‘NoneType’ object has no attribute ‘encode’
It finds the CRC checksum but the code isn’t quite compatible with the version of Python I’m using I believe. This script has a lot of potential and so far, this is the only program I’ve found that does something like this. With a little modification, I could successfully convert Virtual Console games (from the 3DS, Wii, etc) to NES ROMs with the iNES header.
Thanks!
I’m sorry, I should have added more to my post. Baby was crying and posted too quickly.
I was using the Nestopia .xml file. The error occurs because the .XML file I was using does not contain the name field. My idea was adding some error code checking, to first see if it exists. Or, because the script works with the bootGod script, include a default bootGod script and try to check if a newer version is available, and if so, offer to download it. I could help write the code in Linux, but I don’t know if it’d work in Windows. Not much of a Windows program. But I think with some proper error checking routines, perhaps the program might look a little cleaner when something like this happens….if you want, I can try submitting a patch or two.
Thanks!
I modified the script a bit. I added a simple check to make sure the ‘name’ field exists, but none of the other fields, in the XML file. I added support to convert those pesky TNES header files to iNES header files, and I added support for wildcard.
I cannot get recursive subdirectory listings working properly though. Python isn’t really my language. C is. When I execute the script from the OS using a wildcard, such as *.nes, my login script seems to be passing the filenames directly to the script. For example, instead of passing *.nes, it passes a list of all the files that end in .nes, which isn’t directories. Not sure how to handle that with Python.
Anyway, if you want a copy of my modifications, just let me know, and I’ll send them to you.
Thanks!
I’ve modified this script a lot now. I call it version 0.5. I have it working now on Python 2 and Python 3. I doubt anything under 2.7 will be supported. It does support recursive searching now. It has a nice help screen.
Making the code work with Python 2 and 3 was a real pain in the butt, but I think I worked out all the bugs. There’s a few more features I’d like to implement. I have it show a list of ROMs it couldn’t find checksums for in the database. I’d like it to show a number of ROMs that it did find but didn’t need to fix, and a number of ROMs it fixed, headerless ones, etc.
I contacted BootGod and asked if he’d create a VERSION file on his website that would contain the latest version number of the database. I’ve asked permission to use his database in my program. The idea is that there can be an auto-update feature for the database. If a newer version is available, it’ll download it automatically.
I’d also like to replace the CRC32 checksums with SHA-1 ones, but that’ll take some time.
I’ve tested my script thoroughly on Windows 10 64-bit and MSYS2. I’d imagine it’d work in straight Linux as well, but I still have to test that.
I want to detect the OS and if it’s window, display /, which seems to be a window convention. For example, in Windows, to display help, I’d like it where the user could type /? or /h or /help to get the info. In Linux, they’ll still type the -h or –help. Maybe I should just make it where all of them are supported but only the Linux ones are listed by default or something? I don’t think that’d be hard to implement.
The script depends on a few more modules. I’d like to write a function that checks if those modules are installed, and if not, tell the user nicely to install them, so they’re not wondering why nothing works.
And finally, I think it’d be nice if I broke your main code up into a few smaller functions. That’s just me though.
I need some beta testers, if anyone wants to test it and try to break it somehow, that’d be great. I think more error checking is needed as well. I only check for one variable name in the XML file, none of the others. I also got it where it supports ROM names that have funky characters in it. Before, it was removing them. So for example, Astérix would actually show up as Asterix, which I felt was wrong.
I wonder if I should add an option to rename the ROMs based on their titles….any feature requests are more than welcomed. I was gonna upload it to my github repository, if it was okay with you Greg. I was sure to leave your name in there, because without you, none of this could have been possible. You did the hard work, I just modified it a bit. I’d love for you to look at my code and see if it could be improved a bit. I don’t really know Python. I know a few other languages, but not really a Python kinda guy. I’m sure there’s a few things that could be improved!
Hey folks. I imagine Spork Schivago’s updated version above would be preferable, but as far as I can tell they never did end up posting it to their GitHub or anywhere else, so I tweaked the current one to work with Python 3:
https://gist.github.com/vaguerant/30d56467e23784f4ceef96c9b65dd735
This was really just running it through 2to3 to auto-convert it, then fixing up the couple of things it couldn’t do, e.g. Python 2 allows concatenating strings and bytes but Python 3 does not.
I really just made this for myself (hence NstDatabase.xml being uncommented on mine) but I figured it could be useful to others, so assuming I don’t get caught in the spam filter, here it is.