Suggested features of a new gfx converter

Discussion of hardware and software development for Super NES and Super Famicom. See the SNESdev wiki for more information.

Moderator: Moderators

Forum rules
  • For making cartridges of your Super NES games, see Reproduction.
User avatar
Ramsis
Posts: 341
Joined: Sun Jul 01, 2012 6:44 am
Location: Lion's den :3
Contact:

Suggested features of a new gfx converter

Post by Ramsis »

Inspired by this post, I figured it would be a good idea to gather suggestions for features an all-new, specifically SNES-targeted gfx converter tool should have. :)

Here's what I came up with for now:
  • Support for at least PCX, BMP, GIF, PNG input files (as it's a pain to convert your old BMP to PCX first just because the tool you need to use in this case doesn't support the former)
  • Support for optionally (!) optimizing the source data for the SNES tile format (i.e., by reusing redundant tiles)
  • Support for outputting tile maps and palettes (possibly even in either binary or text format, with a user-selectable .DB-like prefix) along with the raw tile data
  • Support for outputting 1bpp, 2bpp, 3bpp, 4bpp, 7bpp, 8bpp, and Mode 7 gfx data
  • Full tile map configuration support, e.g. the user may add offsets to both tile and palette numbers and/or specify to clear or set any of the X/Y flip bits
  • Support for outputting tile maps in either "interleaved" or "non-interleaved" format (i.e., for a screen of 32×32 tiles, you'd either get a single 2 KiB file with [$2118/9 16-bit data *1024] or two 1 KiB files containing [$2118 8-bit data *1024] and [$2119 8-bit data *1024], respectively)
  • Support for outputting Mode 5/6-compatible tile maps (i.e. tile maps where odd tile numbers are skipped entirely since these modes force 16×8 tiles anyway) -- or, even better -- let the user decide whether they want their tile map to be based on 8×8, 16×8, or 16×16 tiles (which should cover all BG-based cases … sprites would be a different story, of course)
  • Possibly even support for pictures using the full RGB color range, with optional dithering and what not …
Yeah, that'd be awesome. *sigh* Still, we can dream, can't we? :lol:

Feel free to add your own suggestions to this thread. Maybe this will inspire someone to tackle a project like this in the future. :)
Some of my projects:
Furry RPG!
Unofficial SNES PowerPak firmware
(See my GitHub profile for more)
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: Suggested features of a new gfx converter

Post by lidnariq »

Ramsis wrote:Support for at least PCX, BMP, GIF, PNG input files
IME/IMO, supporting formats that support truecolor encodings just means you either have obtuse error messages, or you now have to support color reduction in your converter.

My private (unreleased yet, I just haven't gotten around it) converter only deals with GIF for this reason.

It also was written before I fully understood the SNES pattern table layout, so it just emits tiles linearly left-to-right then top-to-bottom, which makes it only useful for tiles of any dimension other than 8x8 if the picture you feed it is 128 pixels wide. And I (at the time, at least) considered "converting color data into palette selectors" out of scope...
Support for outputting 1bpp, 2bpp, 3bpp, 4bpp, 7bpp, 8bpp, and Mode 7 gfx data
Where do 1bpp and 3bpp data show up in the SNES?

Maybe 1bpp could be used as cheap compression, but 3bpp just seems obtuse if used as compression, so I have to assume it's not...

(Input of 5bpp data makes sense, since that's the 2bpp data with in-line palette selectors)
Full tile map configuration support, e.g. the user may add offsets to both tile and palette numbers and/or specify to clear or set any of the X/Y flip bits
I have been wanting a variant of Shiru's NESST for the SNES screen maps...
tile maps where odd tile numbers are skipped entirely since these modes force 16×8 tiles anyway
Is it too rarely useful to use skewed tiles?
Possibly even support for pictures using the full RGB color range, with optional dithering and what not …
Color reduction and dithering is complex enough to have its own programs (such as the algorithms in mtpaint, animmerger, and Jih-Shin Ho's DISPLAY) Not to say it doesn't belong, but I've a sufficiently unixy background that I personally like the idea of separating that out into multiple programs.
Sik
Posts: 1589
Joined: Thu Aug 12, 2010 3:43 am

Re: Suggested features of a new gfx converter

Post by Sik »

lidnariq wrote:IME/IMO, supporting formats that support truecolor encodings just means you either have obtuse error messages, or you now have to support color reduction in your converter.

My private (unreleased yet, I just haven't gotten around it) converter only deals with GIF for this reason.
https://en.wikipedia.org/wiki/GIF#True_color

In all seriousness now though: 1) you can just make a tool only accept paletted images 2) but even then there isn't any guarantee the palette is usable (many programs support paletted images, but won't let you manipulate the palette yourself).

This said, color reduction is not as hard as it sounds when you're given a palette separately - and you want that, because if your tool generates the palette on its own, then that palette will only ever be usable for that graphic (because each image you convert would generate its own palette). The way I handle it in mdtiler (which does Mega Drive images) is paletted PNGs get their indices used as-is, and true color PNGs get downsampled to a palette that's provided separately in the script file.

I've tried adding optional support for checkerboard dithering before. The results were probably somewhat decent, but it didn't exactly choose the best color combinations (a checkerboard shade between two solid shades should be using those two shades, this is not what happened). I should give it a try again, I mean the idea worked quite well for Traveller Tales with their prerendered graphics after all.

Image Image

It could be that the palette I provided was crap too =D
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: Suggested features of a new gfx converter

Post by lidnariq »

Ugh. That relies on a reinterpretation of what a GIF with multiple palettes means. When GIF89 was defined, truecolor visuals were exceptionally rare, and it's almost certainly the case that a GIF containing multiple palettes would have been intended to mean "upload a replacement palette for the next packet of data" ... i.e. color cycling.
The GIF89a Specification wrote:A Local Color Table supersedes a Global Color Table, that is, if a Data Stream contains a Global Color Table, and an image has a Local Color Table associated with it, the decoder must save the Global Color Table, use the Local Color Table to render the image, and then restore the Global Color Table.
Truecolor GIFs are an awful joke originating in people relying on decoder artifacts.
even then there isn't any guarantee the palette is usable
I've definitely seen defaults of:
* sorted by luminosity
* sorted by frequency
* sorted by first-encountered

That said, the problem of "mapping an image to a tilemap" is still easier if the input is closer to fitting (if not actually fully compliant).
I've tried adding optional support for checkerboard dithering before. The results were probably somewhat decent, but it didn't exactly choose the best color combinations (a checkerboard shade between two solid shades should be using those two shades, this is not what happened). I should give it a try again, I mean the idea worked quite well for Traveller Tales with their prerendered graphics after all.
By all means, look into bisqwit's animmerger. He has quite the wealth of different dithering algorithms there, both ordered and diffusion. Imagemagick has a bunch too.
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Suggested features of a new gfx converter

Post by tepples »

Let's run your checklist against pilbmp2nes:
Ramsis wrote:
  • Support for at least PCX, BMP, GIF, PNG input files (as it's a pain to convert your old BMP to PCX first just because the tool you need to use in this case doesn't support the former)
pilbmp2nes supports any indexed color image that Pillow (Python Imaging Library) can load. This is at least PNG, GIF, and BMP.
Ramsis wrote:
  • Support for optionally (!) optimizing the source data for the SNES tile format (i.e., by reusing redundant tiles)
Tile map stuff is unsupported for two reasons. One is that I haven't yet tried to make a parser for format strings that define how a console represents horizontal flip, vertical flip, priority, subpalette number, and tile number in each entry of a tile map. Here, P means priority, V and H flip, C subpalette ("color"), and N tile number:
  • NES: NNNNNNNN. Subpalette bits are in a separate 64-byte table after the main table, stored with one level of Z ordering, and subpalette areas are larger than individual tiles.
  • NES with ExGrafix: CCNNNNNN NNNNNNNN
  • TG16: CCCCNNNN NNNNNNNN
  • Super NES: VHPCCCNN NNNNNNNN
  • Genesis: PCCVHNNN NNNNNNNN
  • GBC: PVH0NCCC NNNNNNNN (yes, bit 8 of N is separate)
  • GBA/DS: CCCCVHNN NNNNNNNN
This could be surmounted by using the above as a format string, analogous to the tile data format strings I describe below. The other, harder problem is that converting an arbitrary indexed color image with one shared color and up to eight sets of 15 colors is the "VM packing" problem, which is hard to even approximate in an automated manner, unless the source image is actually ordered like the hardware palette, where all pixels in a subpalette area with nonzero low bits share the same high bits.
Ramsis wrote:
  • Support for outputting tile maps and palettes (possibly even in either binary or text format, with a user-selectable .DB-like prefix) along with the raw tile data
This would need yet another format string for RGB systems, as only Game Boy Advance and Nintendo DS share a palette format with Super NES. Master System, Game Gear, TG16, Genesis, and Neo Geo have different formats for their RGB palettes, and NES is even weirder (HSV instead of RGB).
Ramsis wrote:
  • Support for outputting 1bpp, 2bpp, 3bpp, 4bpp, 7bpp, 8bpp, and Mode 7 gfx data
Yes. It takes a format string, where digits specify bitplanes. Consecutive digits are packed, a comma separates bitplanes interleaved by row, and a semicolon separates bitplanes interleaved by plane.
  • 0: 1bpp
  • 0,1: 2bpp, interleaved by row (Game Boy, Super NES 2bpp)
  • 0;1: 2bpp, interleaved by plane (NES)
  • 0,1;2: 3bpp, planes 0 and 1 interleaved by row, then plane 2 (Super Mario World)
  • 10 hflip little: Virtual Boy 2bpp
  • 10 little: Neo Geo Pocket 2bpp
  • 0,1,2,3: 4bpp, interleaved by row (Master System)
  • 0,1;2,3: Super NES 4bpp and TG16 background
  • 3210: Genesis 4bpp
  • 3210 hflip little: Game Boy Advance 4bpp
  • 0,1;2,3;4,5;6,7: 8bpp, Super NES mode 3/4
  • 76543210: 8bpp packed (Super NES mode 7, Game Boy Advance 8bpp)
93143
Posts: 1717
Joined: Fri Jul 04, 2014 9:31 pm

Re: Suggested features of a new gfx converter

Post by 93143 »

It should be possible, as the default or as an option, to feed the converter an image that's already been posterized to 32 levels, and have it reliably return the original result (rounded to nearest, I guess) rather than using some sort of RGB channel dither in an attempt to better preserve luminosity like pcx2snes does.
Sik
Posts: 1589
Joined: Thu Aug 12, 2010 3:43 am

Re: Suggested features of a new gfx converter

Post by Sik »

tepples wrote:This could be surmounted by using the above as a format string, analogous to the tile data format strings I describe below. The other, harder problem is that converting an arbitrary indexed color image with one shared color and up to eight sets of 15 colors is the "VM packing" problem, which is hard to even approximate in an automated manner, unless the source image is actually ordered like the hardware palette, where all pixels in a subpalette area with nonzero low bits share the same high bits.
Yeah, with the tilemapping functionality in mdtiler I handled it like follows for paletted PNGs (with true color PNGs you're stuck with a single palette, sorry ;P):
  • Bits 0-3: color index
  • Bits 4-5: palette index
  • Bit 6: priority
In other words: each 16 color group is a palette, the first four groups are low priority, the next four groups are high priority.

You could theoretically have multiple palette and priority values within a tile... if you do that then the resulting choice is just undefined[1] ( ・・) (exception: you can use index 0 for transparent, it's guaranteed to not mess with the results as long as one pixel has the valid flags) I suppose that on the SNES you'd need one more bit for the palette.

I suppose the big problem here is that you need the artist to cooperate. If you get somebody who isn't 100% intimate with the details and why it needs to work that way, they'll just keep complaining at whoever made the tool for making something seemingly so incompetent (since from their viewpoint, it's the tool's job to work around the restrictions).

[1] The tool is just doing an OR of all those bits, but don't rely on that.
zzo38
Posts: 1096
Joined: Mon Feb 07, 2011 12:46 pm

Re: Suggested features of a new gfx converter

Post by zzo38 »

Instead of everything one program, different program for different purpose and then pipe them together to get the effect you want. One program can read the various file formats, one program is reducing the colours, one program is converting the binary output to text format, etc. My Farbfeld Utilities will do some of these things (it can convert PNG input to farbfeld, and can reduce colours, and some other things; for other formats I or others could write additional programs for these purposes); for others we would write other programs. I don't know how SNES graphics format are working (I have looked but have not found a lot of stuff yet), so I don't know if any of the formats in my "ffbit" program are suitable for SNES graphics, although clearly these are not all of the formats supported by SNES anyways; we can write even different people write other program, they can be combined by pipes like I mention above. Possibly later I will add that checkerboard dithering option into ff-reduce for more completeness.
(Free Hero Mesh - FOSS puzzle game engine)
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Suggested features of a new gfx converter

Post by thefox »

Since I was looking at some SNES stuff just a few days ago, I decided to make a tile conversion tool taking in account some of the suggestions presented in the starting post.

Here it is: (Requires Python 2 and Pillow)
https://raw.githubusercontent.com/fo-fo ... le-tool.py

GitHub repo: https://github.com/fo-fo/snes-tile-tool

Currently supports:
- Bit depths 2, 4, 8
- Tile sizes 8x8, 16x16
- Direct select (only in tile data, doesn't calculate the extra 3 bits in nametable data)
- Mode 7 output (interleaved CHR/screen data)
- Optimization of duplicate tiles and mirrored duplicates
- Outputs tilemap data
- Outputs palette data

Feedback/requests are welcome. I'm not super familiar with Super Famicom, so there might be some oddities in there, although I did do some testing.

EDIT: One more thing: It only supports indexed images (unless direct select option is chosen), and the palette has to be ordered in an already SNES-compatible way.

Usage:

Code: Select all

usage: snes-tile-tool.py [-h] -i INFILE [-b {2,4,8}] [-s {8x8,16x16}] [-d]
                         [-m7] [-Od] [-Om] -o OUTPREFIX

SNES tile conversion tool

optional arguments:
  -h, --help            show this help message and exit
  -i INFILE, --infile INFILE
                        input image
  -b {2,4,8}, --bpp {2,4,8}
                        bits per pixel
  -s {8x8,16x16}, --tilesize {8x8,16x16}
                        tile size
  -d, --directselect    use CG Direct Select (8 bpp only)
  -m7, --mode7          generate data in Mode 7 format
  -Od, --optimizedupes  optimize duplicate tiles
  -Om, --optimizemirrors
                        optimize duplicate mirror tiles (needs -Od)
  -o OUTPREFIX, --outprefix OUTPREFIX
                        output file prefix
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
93143
Posts: 1717
Joined: Fri Jul 04, 2014 9:31 pm

Re: Suggested features of a new gfx converter

Post by 93143 »

Mode 7 format need not be interleaved, since you can DMA either tile data or map data by itself, and in fact this is often desirable.
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Suggested features of a new gfx converter

Post by thefox »

93143 wrote:Mode 7 format need not be interleaved, since you can DMA either tile data or map data by itself, and in fact this is often desirable.
Yeah, I was planning to add a "deinterleaved" output option later (it was mentioned in the OP as well), not only to Mode 7 output but in general.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Suggested features of a new gfx converter

Post by koitsu »

Some comments:

1. Some example usage syntaxes in the --help output (or in docs -- since it's in GitHub, a README.md would work) would be helpful. For example, --outprefix doesn't really tell me what it's going to use for a suffix, or what exactly it's going to output.

2. It appears INFILE is mandatory, not optional (re: "optional arguments"). Why does this need an argument (-i) at all? Why can't it be the first filename passed to the script? I might also suggest the same for OUTPREFIX -- I can't tell if it's mandatory or not. If it's mandatory, then that should be the 2nd argument; if it's optional, then the usage syntax needs to say what the default value is.

3. Consider using a more "GNU-like" --help usage output. This includes putting -h/--help last, and also supporting -? (yes you heard me right! You don't need to mention this in the docs/usage syntax though). I also strongly recommend not using "double letter" arguments (ex. -m7), instead stick with single-letter, otherwise you end up with parser complexities if people try to merge multiple options into a single flag (ex. -d7 should be parsed as -d and -7; this is the getopt way!). Example, including what's in #2 above:

Code: Select all

Usage: snes-tile-tool.py [OPTION]... INFILE OUTPREFIX

Options:

  -b, --bpp=BPP             bits per pixel
  -s, --tilesize=TILESIZE   tile size
  -d, --directselect        use CG Direct Select (8 bpp only)
  -7, --mode7               generate data in Mode 7 format
  -Od, --optimizedupes      optimize duplicate tiles
  -Om, --optimizemirrors    optimize duplicate mirror tiles
                              (requires --optimizedupes)
  -h, --help                display this help and exit
      --version             output version information and exit

INFILE should be the filename of an image.  Image file formats
supported are based on what Python Pillow supports; please refer
to https://python-pillow.org/ for details.

OUTPREFIX should be the prefix portion of the filename.  For example,
an OUTPREFIX of "myoutput" would result in files called "myoutput.derp"
and "myoutput.ilikesnakes".

BPP should be a value of 2, 4, or 8.

TILESIZE should be a literal string of either "8x8" or "16x16".  For
example, --tilesize=16x16.

GitHub project: https://github.com/fo-fo/snes-tile-tool
Please report bugs via GitHub's Issues tracker.
4. Along the same lines as the whole argument parsing aspect, I'd suggest maybe turning -O into some kind of multi-string for features, ex. --optimize=dupes,mirrors or -O dupes,mirrors. You can describe all the different toggles similar to what's shown below the options in my above example.

Just some ideas, as someone who has been doing this a looooooong time and gets quite annoyed when he encounters argument parsing and usage syntaxes that are weird, vague, or wonky. :-) (I almost always encounter this in Ruby and Python programs. No clue why that problem is so prevalent there...)
User avatar
Ramsis
Posts: 341
Joined: Sun Jul 01, 2012 6:44 am
Location: Lion's den :3
Contact:

Re: Suggested features of a new gfx converter

Post by Ramsis »

thefox wrote:Since I was looking at some SNES stuff just a few days ago, I decided to make a tile conversion tool taking in account some of the suggestions presented in the starting post.
Wow! :!: How awesome is this! Thank you so much! :D

I just threw some demo gfx at your script, so here's some fresh experience/feedback:
  • Python initially complained about line 92, prompting me to change tile.tostring() to tile.tobytes() (possibly because I have Python 3 installed alongside Python 2?). After making the change, the script ran smoothly.
  • I have a 512×256 file which is used with a 64×32 tile map. I had to do the following in order to get a tile map that I can DMA straight to VRAM (apart from adding palette bits, but never mind for now):

    Code: Select all

    .DEFINE SkipTileRow -128											  ; "deinterleave" tile rows as output by snes-tile-tool.py
    
    .REPEAT 32
    	.REDEFINE SkipTileRow SkipTileRow+128
    	.INCBIN "gfx/area.nam" SKIP SkipTileRow READ 64			; read even rows (i.e., for 1st half of 64×32 tile map)
    .ENDR
    
    .REDEFINE SkipTileRow -64
    
    .REPEAT 32
    	.REDEFINE SkipTileRow SkipTileRow+128
    	.INCBIN "gfx/area.nam" SKIP SkipTileRow READ 64			; read odd rows (i.e., for 2nd half of 64×32 tile map)
    .ENDR
    No big deal of course -- apparently the tool I used previously (gfx2snes) processes each 256×256 "screen" in your image as if they were aligned vertically.
  • Mode 7 palette output seems to be broken, as I got a palette file that was some 680-odd bytes in size when I tried to convert this file. Or is this because of my Python 3 installation?
  • This tool even supports direct color mode, and it works like a treat! Again, wow, and a big thumbs-up! :D
Some of my projects:
Furry RPG!
Unofficial SNES PowerPak firmware
(See my GitHub profile for more)
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Suggested features of a new gfx converter

Post by thefox »

koitsu wrote:1. Some example usage syntaxes in the --help output (or in docs -- since it's in GitHub, a README.md would work) would be helpful. For example, --outprefix doesn't really tell me what it's going to use for a suffix, or what exactly it's going to output.
Yeah, I was planning to add a README.md at some point.
2. It appears INFILE is mandatory, not optional (re: "optional arguments"). Why does this need an argument (-i) at all? Why can't it be the first filename passed to the script? I might also suggest the same for OUTPREFIX -- I can't tell if it's mandatory or not. If it's mandatory, then that should be the 2nd argument; if it's optional, then the usage syntax needs to say what the default value is.
Also considered this, but didn't want to get bogged down in the details, so I just added whatever was easiest at the time. I'll definitely consider getting rid of -i and -o.
3. Consider using a more "GNU-like" --help usage output. This includes putting -h/--help last, and also supporting -? (yes you heard me right! You don't need to mention this in the docs/usage syntax though). I also strongly recommend not using "double letter" arguments (ex. -m7), instead stick with single-letter, otherwise you end up with parser complexities if people try to merge multiple options into a single flag (ex. -d7 should be parsed as -d and -7; this is the getopt way!).
I'm not sure if this is possible with Python's argparse library. Then again, I don't think is a very important issue, so I probably won't attempt it.
4. Along the same lines as the whole argument parsing aspect, I'd suggest maybe turning -O into some kind of multi-string for features, ex. --optimize=dupes,mirrors or -O dupes,mirrors. You can describe all the different toggles similar to what's shown below the options in my above example.
In this case, again, I just chose what was easiest to throw into argparse. Not sure how easy it's to implement your suggestion with argparse. It's a fair point though.
Just some ideas, as someone who has been doing this a looooooong time and gets quite annoyed when he encounters argument parsing and usage syntaxes that are weird, vague, or wonky. :-) (I almost always encounter this in Ruby and Python programs. No clue why that problem is so prevalent there...)
What's weird, vague or wonky to some could be normal to others. :) I'm guessing it's because Python/Ruby people are using built-in facilities for command line parsing, like the argparse library in Python (which I'm using as well), where the semantics don't necessarily match getopt and friends. It's damned easy to use though, the whole argument parsing with argparse is like 20 lines of code. :)

Thanks for the feedback.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Suggested features of a new gfx converter

Post by tepples »

thefox wrote:
koitsu wrote:3. Consider using a more "GNU-like" --help usage output. This includes putting -h/--help last, and also supporting -? (yes you heard me right! You don't need to mention this in the docs/usage syntax though). I also strongly recommend not using "double letter" arguments (ex. -m7), instead stick with single-letter, otherwise you end up with parser complexities if people try to merge multiple options into a single flag (ex. -d7 should be parsed as -d and -7; this is the getopt way!).
I'm not sure if this is possible with Python's argparse library. Then again, I don't think is a very important issue, so I probably won't attempt it.
It's possible, so long as all 1-character options use one hyphen and all long options use two. See the paragraph beginning "Several short options can be joined together" in the argparse reference.
Several short options can be joined together, using only a single - prefix, as long as only the last option (or none of them) requires a value:

Code: Select all

>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-x', action='store_true')
>>> parser.add_argument('-y', action='store_true')
>>> parser.add_argument('-z')
>>> parser.parse_args(['-xyzZ'])
Namespace(x=True, y=True, z='Z')
As for -?, touch -? gives "invalid option" (as do ls -? and cp -?), but less -? gives the list of commands. Also try these commands in the (non-cmd.exe) shell and tell me what you learned:

Code: Select all

echo Gotcha > -a
ls -?
ls -- -a
rm -- -a
Post Reply