nesdev.com
http://forums.nesdev.com/

Including binary files that contain pointers to dynamic data
http://forums.nesdev.com/viewtopic.php?f=2&t=14413
Page 1 of 2

Author:  tokumaru [ Wed Jun 15, 2016 1:30 pm ]
Post subject:  Including binary files that contain pointers to dynamic data

I'm in a situation where I have a tool generating data and pointers to access this data, both of which will be included in my NES program via .incbin. The problem is that the tool doesn't know where in the ROM the data will be included, so it can't possibly generate the final pointers. How would you suggest I handle this issue? Ideally, the assembler would be able to automatically add an offset to the pointers contained in the file, but I don't think ca65 can do anything remotely similar to this.

The most sensible solution I came up with was to use a separate tool, that'd snoop the label file generated by the assembler in order to extract the base address of the data from it, and then patch the file that contains the pointers. Sounds simple enough, but a little clumsy because I'd need to assemble the ROM twice in a row in order to get the correct output: once using the wrong pointers so that all labels would be in the correct places, and again to include the patched files.

Any other ideas?

Author:  Bregalad [ Wed Jun 15, 2016 1:40 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

I had exactly that problem when dealing with data compression. That is exactly why I have programmed CompressTools, which generates assembly code with compressed data instead of generating an unusable binary file.
Also the input is defined as an "assembly-like" file, containing only labels and .db statements.

You could probably have the same approach, and generate an assembly file. The major problem is that it might not be compatible with all assemblers, but you can add options to support the most popular ones.

Author:  tokumaru [ Wed Jun 15, 2016 2:17 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

Interesting idea... I can generate the data as .db statements mixed with labels and generate the pointers using .dw statements followed by the labels. Thanks.

In ca65 I will have to mind scopes when doing this, though, because I scope everything now. Oh well, I can always generate long, unique, random labels and place these files in the global scope, after proper scoped labels have been used to mark the positions of these blocks of data (i.e. I won't use the generated labels to address the data in my code).

One downside is that the tool that decodes these files becomes more complex, since parsing text files is not as straightforward as reading binary files. Not a big problem, though.

Author:  tepples [ Wed Jun 15, 2016 2:28 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

I have seen three workarounds in my own projects and/or projects that I've studied:

  1. Have the compression tool output offsets relative to the first byte of the file, and use a 16-bit add at runtime to find the final pointer. In RHDE, the versions of furniture icons seen in menus use this. So do various parts of the level map in Haunted: Halloween '85. So does Solar Wars, which uses the NerdTracker II module file format described in nt2re.
  2. Devote a whole segment to the compression tool's output and have it use the hardcoded start address of that segment. The Action 53 builder uses a variant of this, based on the PRG ROM ranges marked unused in each game's entry in the config file.
  3. Have the compression tool generate an assembly language file with .byte statements instead of a binary file designed for .incbin. My music engine (Pently) and my VWF engine use this.

Author:  tokumaru [ Wed Jun 15, 2016 2:52 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

tepples wrote:
use a 16-bit add at runtime to find the final pointer

Definitely don't want to do this. This data can be used very frequently every frame, so the time taken by a 16-bit addition can certainly add up to a significant amount. Even if this wasn't the case, I generally feel like performing a task at runtime when it can be pre-computed is a poor decision, if it doesn't bring any advantage other than simplifying the generation of the data. I would, for example, consider calculating the final pointer at runtime if the offset was specified in less then 2 bytes.

Quote:
Devote a whole segment to the compression tool's output and have it use the hardcoded start address of that segment.

That's something I considered, but since I have different sets of compressed stuff sharing the same segment, I'd still need a second pass over the data to "pack" everything together.

Quote:
Have the compression tool generate an assembly language file with .byte statements instead of a binary file designed for .incbin.

Sounds like the best option to me. I'm just looking into how to generate unique names for the labels so I don't have to worry about collisions.

Author:  pubby [ Wed Jun 15, 2016 3:25 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

tokumaru wrote:
Sounds like the best option to me. I'm just looking into how to generate unique names for the labels so I don't have to worry about collisions.

I don't think you need unique names. You can use the '*' psuedo-variable or ':' labels and integer offsets to get your addresses. e.g.

Code:
:
.byt 0, 1, 2, 3, 4, 5, 6, 7
.addr :- + 5 ; pointer to 6th byte


Alternatively, don't forget that you can do stuff like this:
Code:
.scope file1
.include "data.inc"
.endscope

Author:  thefox [ Wed Jun 15, 2016 3:37 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

I exclusively use assembly files for tool output nowadays, except for things that are never going to need relocations (e.g. CHR data).

I was recently thinking about the same problem related to compression. Eventually I would like to integrate two-pass assembly into my build system by taking advantage of ld65's run/load attributes. First pass would output the uncompressed data somewhere, which would then be compressed, and included into the final binary in the second assembly pass. The compression routine would optimally receive just a binary blob and not have to care about the contents. I just haven't figured out all the details yet. (I was planning to make a thread about this actually, just didn't get around to it yet.)

Bregalad's CompressTools is nice, but of course its assembly parsing capabilities are limited compared to ca65. This is bit of deal breaker for me because I tend to go heavy on macro usage within my generated files.

Author:  rainwarrior [ Thu Jun 16, 2016 12:25 am ]
Post subject:  Re: Including binary files that contain pointers to dynamic

I only really use .incbin for data that was made by someone else's software. If I'm writing the tool to generate data, myself, I want assembly output rather than binary.

The assembler/linker already knows how to solve this problem, and is good at it. There's no need to write your own little linker just to prepare for an .incbin. ;)

Author:  na_th_an [ Thu Jun 16, 2016 1:33 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

As suggested, create a tool which outputs source code and let your assembler do the dirty work :)

Author:  tokumaru [ Thu Jun 16, 2016 2:00 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

As my programs get more complex, I sometimes worry that the assembler is going to bump into some sort of limit and will not be able to handle the huge amount of labels I'm using... I'm probably just being paranoid here, since it's very unlikely that assemblers that are still being maintained and are designed to work on modern computers would impose such a limitation.

Still, I even use dynamic symbols in ca65 as a crude form of temporary storage, before I can format the data and put it into its final location... This way I can do things like use macros to create entities (metatiles, hit boxes, objects, whatever), neatly specifying the attributes as arguments, and later dump everything as structures of arrays, which are more convenient for the 6502 to access, so this information is both easy to maintain and fast to consume. Anyway, adding data as code will definitely increase the label count by a lot, but hopefully this won't be a problem.

Author:  tepples [ Thu Jun 16, 2016 2:21 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

ca65 is written in C and has some built-in limits. For example, include files and scopes can't be nested more than 16 levels deep. If you run into any label count limits, then break out the offending file into a separate .s file that exports labels, import those labels in your other files, assemble them separately, and pass all the object files to the linker.

You say you're turning array-of-structures into structure-of-arrays. I'd like to see the macros you rigged up to do that.

Author:  3gengames [ Thu Jun 16, 2016 2:38 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

If your assembler doesn't use link lists and state variables for everything, which is basically only limited by your RAM and I'll assume is 4GB, then you need to switch to an assembler that does. I doubt any assembler worth it's sale would use static anything.

Author:  tepples [ Thu Jun 16, 2016 2:58 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

I think the limits on nested scope, include, and macro calls are there to avoid suddenly thrashing your swap and/or causing the kernel to OOM-kill processes critical to the system or UI if you accidentally cause infinite recursion.

My laptop came with 1 GB, which I upgraded to the maximum of 2 GB within the past few months.

Author:  tokumaru [ Thu Jun 16, 2016 3:31 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

tepples wrote:
You say you're turning array-of-structures into structure-of-arrays. I'd like to see the macros you rigged up to do that.

It's nothing special, I do this mostly so I can reference things by name instead of numbers, and so I can edit the definitions more easily. These are not generic macros though, I have custom code for each type of entity I use this for. For example, this is the macro I use to define hit boxes:

Code:
   ;Creates a new bounding box and gives it a name and a number.
   .macro Object_CreateBox _Name, _Top, _Bottom, _Left, _Right
      .ifndef Object::NextBox
         Object::NextType .set 0
      .endif
      Object::.ident(.sprintf("%s_BOX", .string(_Name))) = Object::NextBox
      Object::.ident(.sprintf("Box%02xTop", Object::NextBox)) = _Top
      Object::.ident(.sprintf("Box%02xBottom", Object::NextBox)) = _Bottom
      Object::.ident(.sprintf("Box%02xLeft", Object::NextBox)) = _Left
      Object::.ident(.sprintf("Box%02xRight", Object::NextBox)) = _Right
      Object::NextBox .set Object::NextBox + 1
   .endmacro

Then I can simply enter the data as an easy to edit list, and I can reference the boxes by name, without ever knowing their ids:

Code:
   Object_CreateBox PLAYER_STANDING, 16, 16, 10, 10
   Object_CreateBox PLAYER_CROUCHING, 12, 12, 10, 10
   Object_CreateBox ITEM_CONTAINER, 8, 7, 8, 7
   ;(...)

And this is the macro that outputs the data:

Code:
   ;Outputs all bounding boxes.
   .macro Object_OutputBoxes _LabelTop, _LabelBottom, _LabelLeft, _LabelRight
      .ident(.string(_LabelTop)):
      .ifdef Object::NextBox
         .repeat Object::NextBox, BoundingBox
            .byte <Object::.ident(.sprintf("Box%02xTop", BoundingBox))
         .endrepeat
      .endif
      .ident(.string(_LabelBottom)):
      .ifdef Object::NextBox
         .repeat Object::NextBox, BoundingBox
            .byte <Object::.ident(.sprintf("Box%02xBottom", BoundingBox))
         .endrepeat
      .endif
      .ident(.string(_LabelLeft)):
      .ifdef Object::NextBox
         .repeat Object::NextBox, BoundingBox
            .byte <Object::.ident(.sprintf("Box%02xLeft", BoundingBox))
         .endrepeat
      .endif
      .ident(.string(_LabelRight)):
      .ifdef Object::NextBox
         .repeat Object::NextBox, BoundingBox
            .byte <Object::.ident(.sprintf("Box%02xRight", BoundingBox))
         .endrepeat
      .endif
   .endmacro

I have similar macros for defining object types (.macro Object_Register _Name, _InitializationAddress, _LogicAddress, _DrawingAddress) and other things. I might end up doing this for metatiles and map collision data too.

Author:  Sik [ Thu Jun 16, 2016 3:48 pm ]
Post subject:  Re: Including binary files that contain pointers to dynamic

tepples wrote:
I think the limits on nested scope, include, and macro calls are there to avoid suddenly thrashing your swap and/or causing the kernel to OOM-kill processes critical to the system or UI if you accidentally cause infinite recursion.

Yep, it's in case of infinite recursion, since the assembler won't quite with an error otherwise. It's usually a setting you can change as well, since in extreme case you may indeed need higher limits.

Page 1 of 2 All times are UTC - 7 hours
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/