Tile encoder/decoder

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems.

Moderator: Moderators

User avatar
tokumaru
Posts: 11744
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru » Mon Jan 04, 2010 5:01 pm

What can I say? I guess I'm not the type that releases user-friendly tools. For the record, I never intended to be, I just wanted to share something I though others could possibly find interesting.

To me it really feels like a waste of time putting a lot of effort into something like this while I could be working on my game. I'll probably stick to my game from now, if I want it to ever be released.

With tools it's hard to please everybody: one is worried about licenses, the other is bothered about speed, someone else wants it to run while rendering is on. This is probably why I hate coding tools for other people. With games it's easier, people just want to play them, or not play them.

BTW, I wouldn't worry about those warnings. I have no use for the value returned by fread, if the file suddenly becomes inaccessible as the encoder runs you probably have worse problems to worry about than not being able to compress your NES tiles, such as your hard disk frying. About the two variables, they are set inside a conditional block that always executes at least once, so your smart ass compiler can complain all it wants but the variables aren't used uninitialized.

UncleSporky
Posts: 385
Joined: Sat Nov 17, 2007 8:44 pm

Post by UncleSporky » Tue Jan 05, 2010 11:49 am

In my opinion, user friendly isn't really necessary, as long as you're here to answer questions, which you appear to be. :)

This is a really cool compression scheme. I don't fully understand the method behind the tables but I can see what is happening enough to trust it to handle my graphics. I'll have to plan on implementing CHR-RAM now.

I think a buffering option would be great, although as you said, you have other things to work on.
tepples wrote:
4bit packed pixel tiles compress much better than planar (composite planar like the snes), but not sure how much this applies to the NES planar format.
Have you tried compressing 2-bit packed pixel tiles? That's what CMM does. Even on PCE and SNES, 2-bit tiles often show up in things like fonts.
I was just thinking about this. Will this compressor handle these just as efficiently as other graphics, or could a modified routine improve speed/compression further? Theoretically, I mean - we don't usually need to load and unload fonts rapidly in the middle of a level.

User avatar
Dwedit
Posts: 4328
Joined: Fri Nov 19, 2004 7:35 pm
Contact:

Post by Dwedit » Tue Jan 05, 2010 12:58 pm

1-bit tiles suck under this compression scheme.
Here come the fortune cookies! Here come the fortune cookies! They're wearing paper hats!

tepples
Posts: 22014
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples » Tue Jan 05, 2010 1:26 pm

A run of tiles with only 2 colors would get compressed with 1 bit to indicate the start of a run with new tables, 8 bits to set modes 1, 1, 0, 0, 2 to 4 bits to specify the colors, 8 bits per tile for repeats, and then 8 more bits for each row of pixels that isn't repeated. So you're right that 1-bit tiles get expanded by 12.5%. Perhaps the table definition needs an additional bit to say whether or not to use row repeating in a run.

But without arithmetic coding, is there a good way to encode 1-bit tiles at all?
What schemes have you seen in commercial NES/GB/SNES/GBA games to compress 1-bit tiles?

User avatar
tokumaru
Posts: 11744
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru » Tue Jan 05, 2010 3:36 pm

tepples wrote:What schemes have you seen in commercial NES/GB/SNES/GBA games to compress 1-bit tiles?
The quadtree method I was experimenting with before this one worked on the individual planes of the tiles (i.e. 1-bit tiles). It was slightly worse than this scheme when used on generic tiles, and the decompressor was pretty complex/large.

UncleSporky
Posts: 385
Joined: Sat Nov 17, 2007 8:44 pm

Post by UncleSporky » Tue Jan 12, 2010 8:32 pm

So TileCount is hard-coded at compression time, and if I want to be able to load in smaller chunks I will need to compress each group individually and incbin them together, with a lookup table for each starting address?

It would be nice if the compressor could do this automatically: take an argument for the size of the groups, and begin output with a table of relative pointers to each group. It makes me wonder how much that might affect the compression ratio, though.

User avatar
tokumaru
Posts: 11744
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru » Tue Jan 12, 2010 9:15 pm

Well, I don't know how other people usually handle their graphics, but I usually don't have a single CHR file. I have separate files for each character, item, font, and so on. Depending on which tiles will be used at the same time I merge the files to create blocks of tiles that I can compress.

I don't know if implementing a feature like this is worth the trouble. If you have a large CHR file but want to encode it to separate blocks, doesn't that mean that there is a reason you think they shouldn't be packed together, which means it makes sense to have them in separate CHR files to begin with?

About the pointers, I wouldn't make that automatic because I like to have options. You can incbin a compressed block anywhere in your source under a label and you have it's address. However, you might want to handle them differently, you may want to have a bank index along with each pointer (in case you have graphics scattered across different program banks). You might not need a table at all, if you have just a couple of graphic blocks and the code to read from the small table would be larger than using immediate values.

I don't like to work with tools that are too specific and limit what you can do with them. As it is now I don't think it's hard at all to include the files by hand and build your own table of pointers. Like I said before (and you agreed! :wink:) I don't care too much for user friendliness, and this qualifies as it IMO. When I'm programming I just want the tools to be usable, the main project is the game, and that's where I'll put my efforts in order to make it a polished product.

UncleSporky
Posts: 385
Joined: Sat Nov 17, 2007 8:44 pm

Post by UncleSporky » Wed Jan 13, 2010 3:52 am

Alright, that's fine. :) I know you have lots of other things to work on.

The only reason I asked was because of this:
I don't like to work with tools that are too specific and limit what you can do with them.
When I first saw TileCount, I thought, oh cool, I can tell it to replace 16 tiles starting here. When I found out it was encoded as part of the compression it just seemed limiting to me.

My natural inclination is to want to be able to switch out any number of tiles at any location, considering that's the primary advantage of CHR-RAM. Whether you want to change the boss monster while keeping regular enemies around, or just change one type of ground tile, it seems useful to me. But I suppose that could just as easily stay uncompressed and be inserted like normal.

User avatar
tokumaru
Posts: 11744
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru » Wed Jan 13, 2010 11:58 am

UncleSporky wrote:My natural inclination is to want to be able to switch out any number of tiles at any location, considering that's the primary advantage of CHR-RAM.
I'm not sure I get what you mean here, but you can't just start decompressing from arbitrary positions in the stream, because of the "blocky" nature of the scheme. Are you saying you'd like to force the start of a new block at certain locations so that you could start decompressing from that point on?
Whether you want to change the boss monster while keeping regular enemies around, or just change one type of ground tile, it seems useful to me.
To me, the natural way to do those things is to compress the basic tileset to a single file and put the boss and new ground tiles in different files. You decompress the basic tileset before the level starts, and later you "patch" the basic tileset with whatever you want, replacing whatever you want.
But I suppose that could just as easily stay uncompressed and be inserted like normal.
I haven't coded a buffered version of the decompressor yet, so you can't really decompress with rendering on yet. In my own game I don't use compression for tiles that are changed during the level, because I want them to be updated as quickly as possible. I believe the original Sonic games are like this as well.

UncleSporky
Posts: 385
Joined: Sat Nov 17, 2007 8:44 pm

Post by UncleSporky » Wed Jan 13, 2010 3:24 pm

tokumaru wrote:
UncleSporky wrote:My natural inclination is to want to be able to switch out any number of tiles at any location, considering that's the primary advantage of CHR-RAM.
I'm not sure I get what you mean here, but you can't just start decompressing from arbitrary positions in the stream, because of the "blocky" nature of the scheme. Are you saying you'd like to force the start of a new block at certain locations so that you could start decompressing from that point on?
Yes, that's what I've been saying. I figured that out about the blocky nature, and since I can't just load in as many as I want, I at least wanted the compressor to be able to stop and start a new segment at a specific length.

Right now it compresses as many tiles as it finds - in your example it's 256 tiles. I want to be able to tell it to compress 32 tiles at a time so that I can "patch in" more tiles, as you say.

Don't worry about it, I'll just make seperate chr files in 32 tile chunks and compress them individually.

tepples
Posts: 22014
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Post by tepples » Wed Jan 13, 2010 3:34 pm

UncleSporky wrote:Don't worry about it, I'll just make seperate chr files in 32 tile chunks and compress them individually.
That shouldn't be a problem. Given that runs of the same tables are only a couple tiles long anyway, the only significant overhead of each compressed segment is 1 byte for the length of the segment and 2 bytes for the starting address in your directory.

CartCollector
Posts: 122
Joined: Mon Oct 30, 2006 8:32 pm

Post by CartCollector » Thu Jan 21, 2010 10:31 pm

I made a collection of CHRs ripped from demos and homebrews. I call it the NESdev Corpus. Here's the download link:

http://www.mediafire.com/?tdilw2ghk1g

It has nearly 250k of graphics data spread across 14 CHR files. Large (>8k) CHR files have been split into 8k chunks. It also includes a text file with the original file sizes and the sizes they've been compressed to using Tokumaru's latest CHR compressor. Hope this helps with testing.
Attachments
NESdev Corpus.zip
(202.02 KiB) Downloaded 306 times

Denine
Posts: 398
Joined: Wed Feb 17, 2010 5:42 pm

Post by Denine » Sat Mar 13, 2010 1:28 pm

SO..,thanks to this it's possible to compress tiles into very small KB size?
But..it is possible to DECOMPRESS tiles from ROM,edit it,compress it again,and finally,Put compressed tiles into rom?
I'm pretty interested sice I'm hacking GDG(GO Dizzy GO),Which is Codemasters game.

User avatar
tokumaru
Posts: 11744
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Post by tokumaru » Sat Mar 13, 2010 4:51 pm

No, you can't use my application for hacking, at least not in a straightforward way. First because my format is slightly different than Codemasters'. And Codemasters used different compression algorithms, so the one used in GO Dizzy GO might not even be the same one as Bee52.

But if you're really up to the the task, you could debug the game and find the routine that decompresses tiles (not caring if it's the same one as Bee52 or not) and set up a breakpoint on calls to it, so that you can take note of where in the ROM the compressed graphics are and dump the tiles after they are decompressed.

Then you edit the tiles at will, and once you're done you can compress them with my application. Then you could replace the games decompression routine with mine (it's smaller than the one used in Bee52 at least) and replace the graphics with your own.

Not a simple task by any means, but unless there are tools specifically written for the games you hack you have to do this sort of old-school hardcore hacking.

Denine
Posts: 398
Joined: Wed Feb 17, 2010 5:42 pm

Post by Denine » Sat Mar 13, 2010 5:06 pm

As I didn't understand anything,I guess i'll drop the subject untill i learn more about NES,and 6502.

Post Reply