Including binary files that contain pointers to dynamic data
Moderator: Moderators
Including binary files that contain pointers to dynamic data
I'm in a situation where I have a tool generating data and pointers to access this data, both of which will be included in my NES program via .incbin. The problem is that the tool doesn't know where in the ROM the data will be included, so it can't possibly generate the final pointers. How would you suggest I handle this issue? Ideally, the assembler would be able to automatically add an offset to the pointers contained in the file, but I don't think ca65 can do anything remotely similar to this.
The most sensible solution I came up with was to use a separate tool, that'd snoop the label file generated by the assembler in order to extract the base address of the data from it, and then patch the file that contains the pointers. Sounds simple enough, but a little clumsy because I'd need to assemble the ROM twice in a row in order to get the correct output: once using the wrong pointers so that all labels would be in the correct places, and again to include the patched files.
Any other ideas?
The most sensible solution I came up with was to use a separate tool, that'd snoop the label file generated by the assembler in order to extract the base address of the data from it, and then patch the file that contains the pointers. Sounds simple enough, but a little clumsy because I'd need to assemble the ROM twice in a row in order to get the correct output: once using the wrong pointers so that all labels would be in the correct places, and again to include the patched files.
Any other ideas?
Re: Including binary files that contain pointers to dynamic
I had exactly that problem when dealing with data compression. That is exactly why I have programmed CompressTools, which generates assembly code with compressed data instead of generating an unusable binary file.
Also the input is defined as an "assembly-like" file, containing only labels and .db statements.
You could probably have the same approach, and generate an assembly file. The major problem is that it might not be compatible with all assemblers, but you can add options to support the most popular ones.
Also the input is defined as an "assembly-like" file, containing only labels and .db statements.
You could probably have the same approach, and generate an assembly file. The major problem is that it might not be compatible with all assemblers, but you can add options to support the most popular ones.
Re: Including binary files that contain pointers to dynamic
Interesting idea... I can generate the data as .db statements mixed with labels and generate the pointers using .dw statements followed by the labels. Thanks.
In ca65 I will have to mind scopes when doing this, though, because I scope everything now. Oh well, I can always generate long, unique, random labels and place these files in the global scope, after proper scoped labels have been used to mark the positions of these blocks of data (i.e. I won't use the generated labels to address the data in my code).
One downside is that the tool that decodes these files becomes more complex, since parsing text files is not as straightforward as reading binary files. Not a big problem, though.
In ca65 I will have to mind scopes when doing this, though, because I scope everything now. Oh well, I can always generate long, unique, random labels and place these files in the global scope, after proper scoped labels have been used to mark the positions of these blocks of data (i.e. I won't use the generated labels to address the data in my code).
One downside is that the tool that decodes these files becomes more complex, since parsing text files is not as straightforward as reading binary files. Not a big problem, though.
Re: Including binary files that contain pointers to dynamic
I have seen three workarounds in my own projects and/or projects that I've studied:
- Have the compression tool output offsets relative to the first byte of the file, and use a 16-bit add at runtime to find the final pointer. In RHDE, the versions of furniture icons seen in menus use this. So do various parts of the level map in Haunted: Halloween '85. So does Solar Wars, which uses the NerdTracker II module file format described in nt2re.
- Devote a whole segment to the compression tool's output and have it use the hardcoded start address of that segment. The Action 53 builder uses a variant of this, based on the PRG ROM ranges marked unused in each game's entry in the config file.
- Have the compression tool generate an assembly language file with .byte statements instead of a binary file designed for .incbin. My music engine (Pently) and my VWF engine use this.
Re: Including binary files that contain pointers to dynamic
Definitely don't want to do this. This data can be used very frequently every frame, so the time taken by a 16-bit addition can certainly add up to a significant amount. Even if this wasn't the case, I generally feel like performing a task at runtime when it can be pre-computed is a poor decision, if it doesn't bring any advantage other than simplifying the generation of the data. I would, for example, consider calculating the final pointer at runtime if the offset was specified in less then 2 bytes.tepples wrote:use a 16-bit add at runtime to find the final pointer
That's something I considered, but since I have different sets of compressed stuff sharing the same segment, I'd still need a second pass over the data to "pack" everything together.Devote a whole segment to the compression tool's output and have it use the hardcoded start address of that segment.
Sounds like the best option to me. I'm just looking into how to generate unique names for the labels so I don't have to worry about collisions.Have the compression tool generate an assembly language file with .byte statements instead of a binary file designed for .incbin.
Re: Including binary files that contain pointers to dynamic
I don't think you need unique names. You can use the '*' psuedo-variable or ':' labels and integer offsets to get your addresses. e.g.tokumaru wrote:Sounds like the best option to me. I'm just looking into how to generate unique names for the labels so I don't have to worry about collisions.
Code: Select all
:
.byt 0, 1, 2, 3, 4, 5, 6, 7
.addr :- + 5 ; pointer to 6th byte
Code: Select all
.scope file1
.include "data.inc"
.endscope
Re: Including binary files that contain pointers to dynamic
I exclusively use assembly files for tool output nowadays, except for things that are never going to need relocations (e.g. CHR data).
I was recently thinking about the same problem related to compression. Eventually I would like to integrate two-pass assembly into my build system by taking advantage of ld65's run/load attributes. First pass would output the uncompressed data somewhere, which would then be compressed, and included into the final binary in the second assembly pass. The compression routine would optimally receive just a binary blob and not have to care about the contents. I just haven't figured out all the details yet. (I was planning to make a thread about this actually, just didn't get around to it yet.)
Bregalad's CompressTools is nice, but of course its assembly parsing capabilities are limited compared to ca65. This is bit of deal breaker for me because I tend to go heavy on macro usage within my generated files.
I was recently thinking about the same problem related to compression. Eventually I would like to integrate two-pass assembly into my build system by taking advantage of ld65's run/load attributes. First pass would output the uncompressed data somewhere, which would then be compressed, and included into the final binary in the second assembly pass. The compression routine would optimally receive just a binary blob and not have to care about the contents. I just haven't figured out all the details yet. (I was planning to make a thread about this actually, just didn't get around to it yet.)
Bregalad's CompressTools is nice, but of course its assembly parsing capabilities are limited compared to ca65. This is bit of deal breaker for me because I tend to go heavy on macro usage within my generated files.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
- rainwarrior
- Posts: 8732
- Joined: Sun Jan 22, 2012 12:03 pm
- Location: Canada
- Contact:
Re: Including binary files that contain pointers to dynamic
I only really use .incbin for data that was made by someone else's software. If I'm writing the tool to generate data, myself, I want assembly output rather than binary.
The assembler/linker already knows how to solve this problem, and is good at it. There's no need to write your own little linker just to prepare for an .incbin.
The assembler/linker already knows how to solve this problem, and is good at it. There's no need to write your own little linker just to prepare for an .incbin.
Re: Including binary files that contain pointers to dynamic
As suggested, create a tool which outputs source code and let your assembler do the dirty work
Re: Including binary files that contain pointers to dynamic
As my programs get more complex, I sometimes worry that the assembler is going to bump into some sort of limit and will not be able to handle the huge amount of labels I'm using... I'm probably just being paranoid here, since it's very unlikely that assemblers that are still being maintained and are designed to work on modern computers would impose such a limitation.
Still, I even use dynamic symbols in ca65 as a crude form of temporary storage, before I can format the data and put it into its final location... This way I can do things like use macros to create entities (metatiles, hit boxes, objects, whatever), neatly specifying the attributes as arguments, and later dump everything as structures of arrays, which are more convenient for the 6502 to access, so this information is both easy to maintain and fast to consume. Anyway, adding data as code will definitely increase the label count by a lot, but hopefully this won't be a problem.
Still, I even use dynamic symbols in ca65 as a crude form of temporary storage, before I can format the data and put it into its final location... This way I can do things like use macros to create entities (metatiles, hit boxes, objects, whatever), neatly specifying the attributes as arguments, and later dump everything as structures of arrays, which are more convenient for the 6502 to access, so this information is both easy to maintain and fast to consume. Anyway, adding data as code will definitely increase the label count by a lot, but hopefully this won't be a problem.
Re: Including binary files that contain pointers to dynamic
ca65 is written in C and has some built-in limits. For example, include files and scopes can't be nested more than 16 levels deep. If you run into any label count limits, then break out the offending file into a separate .s file that exports labels, import those labels in your other files, assemble them separately, and pass all the object files to the linker.
You say you're turning array-of-structures into structure-of-arrays. I'd like to see the macros you rigged up to do that.
You say you're turning array-of-structures into structure-of-arrays. I'd like to see the macros you rigged up to do that.
Re: Including binary files that contain pointers to dynamic
If your assembler doesn't use link lists and state variables for everything, which is basically only limited by your RAM and I'll assume is 4GB, then you need to switch to an assembler that does. I doubt any assembler worth it's sale would use static anything.
Re: Including binary files that contain pointers to dynamic
I think the limits on nested scope, include, and macro calls are there to avoid suddenly thrashing your swap and/or causing the kernel to OOM-kill processes critical to the system or UI if you accidentally cause infinite recursion.
My laptop came with 1 GB, which I upgraded to the maximum of 2 GB within the past few months.
My laptop came with 1 GB, which I upgraded to the maximum of 2 GB within the past few months.
Re: Including binary files that contain pointers to dynamic
It's nothing special, I do this mostly so I can reference things by name instead of numbers, and so I can edit the definitions more easily. These are not generic macros though, I have custom code for each type of entity I use this for. For example, this is the macro I use to define hit boxes:tepples wrote:You say you're turning array-of-structures into structure-of-arrays. I'd like to see the macros you rigged up to do that.
Code: Select all
;Creates a new bounding box and gives it a name and a number.
.macro Object_CreateBox _Name, _Top, _Bottom, _Left, _Right
.ifndef Object::NextBox
Object::NextType .set 0
.endif
Object::.ident(.sprintf("%s_BOX", .string(_Name))) = Object::NextBox
Object::.ident(.sprintf("Box%02xTop", Object::NextBox)) = _Top
Object::.ident(.sprintf("Box%02xBottom", Object::NextBox)) = _Bottom
Object::.ident(.sprintf("Box%02xLeft", Object::NextBox)) = _Left
Object::.ident(.sprintf("Box%02xRight", Object::NextBox)) = _Right
Object::NextBox .set Object::NextBox + 1
.endmacro
Code: Select all
Object_CreateBox PLAYER_STANDING, 16, 16, 10, 10
Object_CreateBox PLAYER_CROUCHING, 12, 12, 10, 10
Object_CreateBox ITEM_CONTAINER, 8, 7, 8, 7
;(...)
Code: Select all
;Outputs all bounding boxes.
.macro Object_OutputBoxes _LabelTop, _LabelBottom, _LabelLeft, _LabelRight
.ident(.string(_LabelTop)):
.ifdef Object::NextBox
.repeat Object::NextBox, BoundingBox
.byte <Object::.ident(.sprintf("Box%02xTop", BoundingBox))
.endrepeat
.endif
.ident(.string(_LabelBottom)):
.ifdef Object::NextBox
.repeat Object::NextBox, BoundingBox
.byte <Object::.ident(.sprintf("Box%02xBottom", BoundingBox))
.endrepeat
.endif
.ident(.string(_LabelLeft)):
.ifdef Object::NextBox
.repeat Object::NextBox, BoundingBox
.byte <Object::.ident(.sprintf("Box%02xLeft", BoundingBox))
.endrepeat
.endif
.ident(.string(_LabelRight)):
.ifdef Object::NextBox
.repeat Object::NextBox, BoundingBox
.byte <Object::.ident(.sprintf("Box%02xRight", BoundingBox))
.endrepeat
.endif
.endmacro
Re: Including binary files that contain pointers to dynamic
Yep, it's in case of infinite recursion, since the assembler won't quite with an error otherwise. It's usually a setting you can change as well, since in extreme case you may indeed need higher limits.tepples wrote:I think the limits on nested scope, include, and macro calls are there to avoid suddenly thrashing your swap and/or causing the kernel to OOM-kill processes critical to the system or UI if you accidentally cause infinite recursion.