Including binary files that contain pointers to dynamic data

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems. See the NESdev wiki for more information.

Moderator: Moderators

User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Including binary files that contain pointers to dynamic data

Post by tokumaru »

I'm in a situation where I have a tool generating data and pointers to access this data, both of which will be included in my NES program via .incbin. The problem is that the tool doesn't know where in the ROM the data will be included, so it can't possibly generate the final pointers. How would you suggest I handle this issue? Ideally, the assembler would be able to automatically add an offset to the pointers contained in the file, but I don't think ca65 can do anything remotely similar to this.

The most sensible solution I came up with was to use a separate tool, that'd snoop the label file generated by the assembler in order to extract the base address of the data from it, and then patch the file that contains the pointers. Sounds simple enough, but a little clumsy because I'd need to assemble the ROM twice in a row in order to get the correct output: once using the wrong pointers so that all labels would be in the correct places, and again to include the patched files.

Any other ideas?
User avatar
Bregalad
Posts: 8056
Joined: Fri Nov 12, 2004 2:49 pm
Location: Divonne-les-bains, France

Re: Including binary files that contain pointers to dynamic

Post by Bregalad »

I had exactly that problem when dealing with data compression. That is exactly why I have programmed CompressTools, which generates assembly code with compressed data instead of generating an unusable binary file.
Also the input is defined as an "assembly-like" file, containing only labels and .db statements.

You could probably have the same approach, and generate an assembly file. The major problem is that it might not be compatible with all assemblers, but you can add options to support the most popular ones.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: Including binary files that contain pointers to dynamic

Post by tokumaru »

Interesting idea... I can generate the data as .db statements mixed with labels and generate the pointers using .dw statements followed by the labels. Thanks.

In ca65 I will have to mind scopes when doing this, though, because I scope everything now. Oh well, I can always generate long, unique, random labels and place these files in the global scope, after proper scoped labels have been used to mark the positions of these blocks of data (i.e. I won't use the generated labels to address the data in my code).

One downside is that the tool that decodes these files becomes more complex, since parsing text files is not as straightforward as reading binary files. Not a big problem, though.
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Including binary files that contain pointers to dynamic

Post by tepples »

I have seen three workarounds in my own projects and/or projects that I've studied:
  1. Have the compression tool output offsets relative to the first byte of the file, and use a 16-bit add at runtime to find the final pointer. In RHDE, the versions of furniture icons seen in menus use this. So do various parts of the level map in Haunted: Halloween '85. So does Solar Wars, which uses the NerdTracker II module file format described in nt2re.
  2. Devote a whole segment to the compression tool's output and have it use the hardcoded start address of that segment. The Action 53 builder uses a variant of this, based on the PRG ROM ranges marked unused in each game's entry in the config file.
  3. Have the compression tool generate an assembly language file with .byte statements instead of a binary file designed for .incbin. My music engine (Pently) and my VWF engine use this.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: Including binary files that contain pointers to dynamic

Post by tokumaru »

tepples wrote:use a 16-bit add at runtime to find the final pointer
Definitely don't want to do this. This data can be used very frequently every frame, so the time taken by a 16-bit addition can certainly add up to a significant amount. Even if this wasn't the case, I generally feel like performing a task at runtime when it can be pre-computed is a poor decision, if it doesn't bring any advantage other than simplifying the generation of the data. I would, for example, consider calculating the final pointer at runtime if the offset was specified in less then 2 bytes.
Devote a whole segment to the compression tool's output and have it use the hardcoded start address of that segment.
That's something I considered, but since I have different sets of compressed stuff sharing the same segment, I'd still need a second pass over the data to "pack" everything together.
Have the compression tool generate an assembly language file with .byte statements instead of a binary file designed for .incbin.
Sounds like the best option to me. I'm just looking into how to generate unique names for the labels so I don't have to worry about collisions.
User avatar
pubby
Posts: 583
Joined: Thu Mar 31, 2016 11:15 am

Re: Including binary files that contain pointers to dynamic

Post by pubby »

tokumaru wrote:Sounds like the best option to me. I'm just looking into how to generate unique names for the labels so I don't have to worry about collisions.
I don't think you need unique names. You can use the '*' psuedo-variable or ':' labels and integer offsets to get your addresses. e.g.

Code: Select all

:
.byt 0, 1, 2, 3, 4, 5, 6, 7
.addr :- + 5 ; pointer to 6th byte
Alternatively, don't forget that you can do stuff like this:

Code: Select all

.scope file1
.include "data.inc"
.endscope
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Including binary files that contain pointers to dynamic

Post by thefox »

I exclusively use assembly files for tool output nowadays, except for things that are never going to need relocations (e.g. CHR data).

I was recently thinking about the same problem related to compression. Eventually I would like to integrate two-pass assembly into my build system by taking advantage of ld65's run/load attributes. First pass would output the uncompressed data somewhere, which would then be compressed, and included into the final binary in the second assembly pass. The compression routine would optimally receive just a binary blob and not have to care about the contents. I just haven't figured out all the details yet. (I was planning to make a thread about this actually, just didn't get around to it yet.)

Bregalad's CompressTools is nice, but of course its assembly parsing capabilities are limited compared to ca65. This is bit of deal breaker for me because I tend to go heavy on macro usage within my generated files.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
User avatar
rainwarrior
Posts: 8732
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Including binary files that contain pointers to dynamic

Post by rainwarrior »

I only really use .incbin for data that was made by someone else's software. If I'm writing the tool to generate data, myself, I want assembly output rather than binary.

The assembler/linker already knows how to solve this problem, and is good at it. There's no need to write your own little linker just to prepare for an .incbin. ;)
na_th_an
Posts: 558
Joined: Mon May 27, 2013 9:40 am

Re: Including binary files that contain pointers to dynamic

Post by na_th_an »

As suggested, create a tool which outputs source code and let your assembler do the dirty work :)
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: Including binary files that contain pointers to dynamic

Post by tokumaru »

As my programs get more complex, I sometimes worry that the assembler is going to bump into some sort of limit and will not be able to handle the huge amount of labels I'm using... I'm probably just being paranoid here, since it's very unlikely that assemblers that are still being maintained and are designed to work on modern computers would impose such a limitation.

Still, I even use dynamic symbols in ca65 as a crude form of temporary storage, before I can format the data and put it into its final location... This way I can do things like use macros to create entities (metatiles, hit boxes, objects, whatever), neatly specifying the attributes as arguments, and later dump everything as structures of arrays, which are more convenient for the 6502 to access, so this information is both easy to maintain and fast to consume. Anyway, adding data as code will definitely increase the label count by a lot, but hopefully this won't be a problem.
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Including binary files that contain pointers to dynamic

Post by tepples »

ca65 is written in C and has some built-in limits. For example, include files and scopes can't be nested more than 16 levels deep. If you run into any label count limits, then break out the offending file into a separate .s file that exports labels, import those labels in your other files, assemble them separately, and pass all the object files to the linker.

You say you're turning array-of-structures into structure-of-arrays. I'd like to see the macros you rigged up to do that.
3gengames
Formerly 65024U
Posts: 2284
Joined: Sat Mar 27, 2010 12:57 pm

Re: Including binary files that contain pointers to dynamic

Post by 3gengames »

If your assembler doesn't use link lists and state variables for everything, which is basically only limited by your RAM and I'll assume is 4GB, then you need to switch to an assembler that does. I doubt any assembler worth it's sale would use static anything.
tepples
Posts: 22708
Joined: Sun Sep 19, 2004 11:12 pm
Location: NE Indiana, USA (NTSC)
Contact:

Re: Including binary files that contain pointers to dynamic

Post by tepples »

I think the limits on nested scope, include, and macro calls are there to avoid suddenly thrashing your swap and/or causing the kernel to OOM-kill processes critical to the system or UI if you accidentally cause infinite recursion.

My laptop came with 1 GB, which I upgraded to the maximum of 2 GB within the past few months.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: Including binary files that contain pointers to dynamic

Post by tokumaru »

tepples wrote:You say you're turning array-of-structures into structure-of-arrays. I'd like to see the macros you rigged up to do that.
It's nothing special, I do this mostly so I can reference things by name instead of numbers, and so I can edit the definitions more easily. These are not generic macros though, I have custom code for each type of entity I use this for. For example, this is the macro I use to define hit boxes:

Code: Select all

	;Creates a new bounding box and gives it a name and a number.
	.macro Object_CreateBox _Name, _Top, _Bottom, _Left, _Right
		.ifndef Object::NextBox
			Object::NextType .set 0
		.endif
		Object::.ident(.sprintf("%s_BOX", .string(_Name))) = Object::NextBox
		Object::.ident(.sprintf("Box%02xTop", Object::NextBox)) = _Top
		Object::.ident(.sprintf("Box%02xBottom", Object::NextBox)) = _Bottom
		Object::.ident(.sprintf("Box%02xLeft", Object::NextBox)) = _Left
		Object::.ident(.sprintf("Box%02xRight", Object::NextBox)) = _Right
		Object::NextBox .set Object::NextBox + 1
	.endmacro
Then I can simply enter the data as an easy to edit list, and I can reference the boxes by name, without ever knowing their ids:

Code: Select all

	Object_CreateBox PLAYER_STANDING, 16, 16, 10, 10
	Object_CreateBox PLAYER_CROUCHING, 12, 12, 10, 10
	Object_CreateBox ITEM_CONTAINER, 8, 7, 8, 7
	;(...)
And this is the macro that outputs the data:

Code: Select all

	;Outputs all bounding boxes.
	.macro Object_OutputBoxes _LabelTop, _LabelBottom, _LabelLeft, _LabelRight
		.ident(.string(_LabelTop)):
		.ifdef Object::NextBox
			.repeat Object::NextBox, BoundingBox
				.byte <Object::.ident(.sprintf("Box%02xTop", BoundingBox))
			.endrepeat
		.endif
		.ident(.string(_LabelBottom)):
		.ifdef Object::NextBox
			.repeat Object::NextBox, BoundingBox
				.byte <Object::.ident(.sprintf("Box%02xBottom", BoundingBox))
			.endrepeat
		.endif
		.ident(.string(_LabelLeft)):
		.ifdef Object::NextBox
			.repeat Object::NextBox, BoundingBox
				.byte <Object::.ident(.sprintf("Box%02xLeft", BoundingBox))
			.endrepeat
		.endif
		.ident(.string(_LabelRight)):
		.ifdef Object::NextBox
			.repeat Object::NextBox, BoundingBox
				.byte <Object::.ident(.sprintf("Box%02xRight", BoundingBox))
			.endrepeat
		.endif
	.endmacro
I have similar macros for defining object types (.macro Object_Register _Name, _InitializationAddress, _LogicAddress, _DrawingAddress) and other things. I might end up doing this for metatiles and map collision data too.
Sik
Posts: 1589
Joined: Thu Aug 12, 2010 3:43 am

Re: Including binary files that contain pointers to dynamic

Post by Sik »

tepples wrote:I think the limits on nested scope, include, and macro calls are there to avoid suddenly thrashing your swap and/or causing the kernel to OOM-kill processes critical to the system or UI if you accidentally cause infinite recursion.
Yep, it's in case of infinite recursion, since the assembler won't quite with an error otherwise. It's usually a setting you can change as well, since in extreme case you may indeed need higher limits.
Post Reply