It is currently Mon Oct 23, 2017 12:07 pm

All times are UTC - 7 hours





Post new topic Reply to topic  [ 27 posts ]  Go to page 1, 2  Next
Author Message
PostPosted: Wed Jun 15, 2016 1:30 pm 
Offline
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10067
Location: Rio de Janeiro - Brazil
I'm in a situation where I have a tool generating data and pointers to access this data, both of which will be included in my NES program via .incbin. The problem is that the tool doesn't know where in the ROM the data will be included, so it can't possibly generate the final pointers. How would you suggest I handle this issue? Ideally, the assembler would be able to automatically add an offset to the pointers contained in the file, but I don't think ca65 can do anything remotely similar to this.

The most sensible solution I came up with was to use a separate tool, that'd snoop the label file generated by the assembler in order to extract the base address of the data from it, and then patch the file that contains the pointers. Sounds simple enough, but a little clumsy because I'd need to assemble the ROM twice in a row in order to get the correct output: once using the wrong pointers so that all labels would be in the correct places, and again to include the patched files.

Any other ideas?


Top
 Profile  
 
PostPosted: Wed Jun 15, 2016 1:40 pm 
Offline
User avatar

Joined: Fri Nov 12, 2004 2:49 pm
Posts: 7235
Location: Chexbres, VD, Switzerland
I had exactly that problem when dealing with data compression. That is exactly why I have programmed CompressTools, which generates assembly code with compressed data instead of generating an unusable binary file.
Also the input is defined as an "assembly-like" file, containing only labels and .db statements.

You could probably have the same approach, and generate an assembly file. The major problem is that it might not be compatible with all assemblers, but you can add options to support the most popular ones.


Top
 Profile  
 
PostPosted: Wed Jun 15, 2016 2:17 pm 
Offline
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10067
Location: Rio de Janeiro - Brazil
Interesting idea... I can generate the data as .db statements mixed with labels and generate the pointers using .dw statements followed by the labels. Thanks.

In ca65 I will have to mind scopes when doing this, though, because I scope everything now. Oh well, I can always generate long, unique, random labels and place these files in the global scope, after proper scoped labels have been used to mark the positions of these blocks of data (i.e. I won't use the generated labels to address the data in my code).

One downside is that the tool that decodes these files becomes more complex, since parsing text files is not as straightforward as reading binary files. Not a big problem, though.


Top
 Profile  
 
PostPosted: Wed Jun 15, 2016 2:28 pm 
Offline

Joined: Sun Sep 19, 2004 11:12 pm
Posts: 19120
Location: NE Indiana, USA (NTSC)
I have seen three workarounds in my own projects and/or projects that I've studied:

  1. Have the compression tool output offsets relative to the first byte of the file, and use a 16-bit add at runtime to find the final pointer. In RHDE, the versions of furniture icons seen in menus use this. So do various parts of the level map in Haunted: Halloween '85. So does Solar Wars, which uses the NerdTracker II module file format described in nt2re.
  2. Devote a whole segment to the compression tool's output and have it use the hardcoded start address of that segment. The Action 53 builder uses a variant of this, based on the PRG ROM ranges marked unused in each game's entry in the config file.
  3. Have the compression tool generate an assembly language file with .byte statements instead of a binary file designed for .incbin. My music engine (Pently) and my VWF engine use this.


Top
 Profile  
 
PostPosted: Wed Jun 15, 2016 2:52 pm 
Offline
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10067
Location: Rio de Janeiro - Brazil
tepples wrote:
use a 16-bit add at runtime to find the final pointer

Definitely don't want to do this. This data can be used very frequently every frame, so the time taken by a 16-bit addition can certainly add up to a significant amount. Even if this wasn't the case, I generally feel like performing a task at runtime when it can be pre-computed is a poor decision, if it doesn't bring any advantage other than simplifying the generation of the data. I would, for example, consider calculating the final pointer at runtime if the offset was specified in less then 2 bytes.

Quote:
Devote a whole segment to the compression tool's output and have it use the hardcoded start address of that segment.

That's something I considered, but since I have different sets of compressed stuff sharing the same segment, I'd still need a second pass over the data to "pack" everything together.

Quote:
Have the compression tool generate an assembly language file with .byte statements instead of a binary file designed for .incbin.

Sounds like the best option to me. I'm just looking into how to generate unique names for the labels so I don't have to worry about collisions.


Top
 Profile  
 
PostPosted: Wed Jun 15, 2016 3:25 pm 
Offline
User avatar

Joined: Thu Mar 31, 2016 11:15 am
Posts: 200
tokumaru wrote:
Sounds like the best option to me. I'm just looking into how to generate unique names for the labels so I don't have to worry about collisions.

I don't think you need unique names. You can use the '*' psuedo-variable or ':' labels and integer offsets to get your addresses. e.g.

Code:
:
.byt 0, 1, 2, 3, 4, 5, 6, 7
.addr :- + 5 ; pointer to 6th byte


Alternatively, don't forget that you can do stuff like this:
Code:
.scope file1
.include "data.inc"
.endscope


Top
 Profile  
 
PostPosted: Wed Jun 15, 2016 3:37 pm 
Offline
User avatar

Joined: Mon Jan 03, 2005 10:36 am
Posts: 2963
Location: Tampere, Finland
I exclusively use assembly files for tool output nowadays, except for things that are never going to need relocations (e.g. CHR data).

I was recently thinking about the same problem related to compression. Eventually I would like to integrate two-pass assembly into my build system by taking advantage of ld65's run/load attributes. First pass would output the uncompressed data somewhere, which would then be compressed, and included into the final binary in the second assembly pass. The compression routine would optimally receive just a binary blob and not have to care about the contents. I just haven't figured out all the details yet. (I was planning to make a thread about this actually, just didn't get around to it yet.)

Bregalad's CompressTools is nice, but of course its assembly parsing capabilities are limited compared to ca65. This is bit of deal breaker for me because I tend to go heavy on macro usage within my generated files.

_________________
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: kkfos.aspekt.fi


Top
 Profile  
 
PostPosted: Thu Jun 16, 2016 12:25 am 
Offline
User avatar

Joined: Sun Jan 22, 2012 12:03 pm
Posts: 5735
Location: Canada
I only really use .incbin for data that was made by someone else's software. If I'm writing the tool to generate data, myself, I want assembly output rather than binary.

The assembler/linker already knows how to solve this problem, and is good at it. There's no need to write your own little linker just to prepare for an .incbin. ;)


Top
 Profile  
 
PostPosted: Thu Jun 16, 2016 1:33 pm 
Offline

Joined: Mon May 27, 2013 9:40 am
Posts: 351
As suggested, create a tool which outputs source code and let your assembler do the dirty work :)

_________________
http://www.mojontwins.com


Top
 Profile  
 
PostPosted: Thu Jun 16, 2016 2:00 pm 
Offline
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10067
Location: Rio de Janeiro - Brazil
As my programs get more complex, I sometimes worry that the assembler is going to bump into some sort of limit and will not be able to handle the huge amount of labels I'm using... I'm probably just being paranoid here, since it's very unlikely that assemblers that are still being maintained and are designed to work on modern computers would impose such a limitation.

Still, I even use dynamic symbols in ca65 as a crude form of temporary storage, before I can format the data and put it into its final location... This way I can do things like use macros to create entities (metatiles, hit boxes, objects, whatever), neatly specifying the attributes as arguments, and later dump everything as structures of arrays, which are more convenient for the 6502 to access, so this information is both easy to maintain and fast to consume. Anyway, adding data as code will definitely increase the label count by a lot, but hopefully this won't be a problem.


Top
 Profile  
 
PostPosted: Thu Jun 16, 2016 2:21 pm 
Offline

Joined: Sun Sep 19, 2004 11:12 pm
Posts: 19120
Location: NE Indiana, USA (NTSC)
ca65 is written in C and has some built-in limits. For example, include files and scopes can't be nested more than 16 levels deep. If you run into any label count limits, then break out the offending file into a separate .s file that exports labels, import those labels in your other files, assemble them separately, and pass all the object files to the linker.

You say you're turning array-of-structures into structure-of-arrays. I'd like to see the macros you rigged up to do that.


Top
 Profile  
 
PostPosted: Thu Jun 16, 2016 2:38 pm 
Offline
Formerly 65024U

Joined: Sat Mar 27, 2010 12:57 pm
Posts: 2257
If your assembler doesn't use link lists and state variables for everything, which is basically only limited by your RAM and I'll assume is 4GB, then you need to switch to an assembler that does. I doubt any assembler worth it's sale would use static anything.


Top
 Profile  
 
PostPosted: Thu Jun 16, 2016 2:58 pm 
Offline

Joined: Sun Sep 19, 2004 11:12 pm
Posts: 19120
Location: NE Indiana, USA (NTSC)
I think the limits on nested scope, include, and macro calls are there to avoid suddenly thrashing your swap and/or causing the kernel to OOM-kill processes critical to the system or UI if you accidentally cause infinite recursion.

My laptop came with 1 GB, which I upgraded to the maximum of 2 GB within the past few months.


Top
 Profile  
 
PostPosted: Thu Jun 16, 2016 3:31 pm 
Offline
User avatar

Joined: Sat Feb 12, 2005 9:43 pm
Posts: 10067
Location: Rio de Janeiro - Brazil
tepples wrote:
You say you're turning array-of-structures into structure-of-arrays. I'd like to see the macros you rigged up to do that.

It's nothing special, I do this mostly so I can reference things by name instead of numbers, and so I can edit the definitions more easily. These are not generic macros though, I have custom code for each type of entity I use this for. For example, this is the macro I use to define hit boxes:

Code:
   ;Creates a new bounding box and gives it a name and a number.
   .macro Object_CreateBox _Name, _Top, _Bottom, _Left, _Right
      .ifndef Object::NextBox
         Object::NextType .set 0
      .endif
      Object::.ident(.sprintf("%s_BOX", .string(_Name))) = Object::NextBox
      Object::.ident(.sprintf("Box%02xTop", Object::NextBox)) = _Top
      Object::.ident(.sprintf("Box%02xBottom", Object::NextBox)) = _Bottom
      Object::.ident(.sprintf("Box%02xLeft", Object::NextBox)) = _Left
      Object::.ident(.sprintf("Box%02xRight", Object::NextBox)) = _Right
      Object::NextBox .set Object::NextBox + 1
   .endmacro

Then I can simply enter the data as an easy to edit list, and I can reference the boxes by name, without ever knowing their ids:

Code:
   Object_CreateBox PLAYER_STANDING, 16, 16, 10, 10
   Object_CreateBox PLAYER_CROUCHING, 12, 12, 10, 10
   Object_CreateBox ITEM_CONTAINER, 8, 7, 8, 7
   ;(...)

And this is the macro that outputs the data:

Code:
   ;Outputs all bounding boxes.
   .macro Object_OutputBoxes _LabelTop, _LabelBottom, _LabelLeft, _LabelRight
      .ident(.string(_LabelTop)):
      .ifdef Object::NextBox
         .repeat Object::NextBox, BoundingBox
            .byte <Object::.ident(.sprintf("Box%02xTop", BoundingBox))
         .endrepeat
      .endif
      .ident(.string(_LabelBottom)):
      .ifdef Object::NextBox
         .repeat Object::NextBox, BoundingBox
            .byte <Object::.ident(.sprintf("Box%02xBottom", BoundingBox))
         .endrepeat
      .endif
      .ident(.string(_LabelLeft)):
      .ifdef Object::NextBox
         .repeat Object::NextBox, BoundingBox
            .byte <Object::.ident(.sprintf("Box%02xLeft", BoundingBox))
         .endrepeat
      .endif
      .ident(.string(_LabelRight)):
      .ifdef Object::NextBox
         .repeat Object::NextBox, BoundingBox
            .byte <Object::.ident(.sprintf("Box%02xRight", BoundingBox))
         .endrepeat
      .endif
   .endmacro

I have similar macros for defining object types (.macro Object_Register _Name, _InitializationAddress, _LogicAddress, _DrawingAddress) and other things. I might end up doing this for metatiles and map collision data too.


Top
 Profile  
 
PostPosted: Thu Jun 16, 2016 3:48 pm 
Offline

Joined: Thu Aug 12, 2010 3:43 am
Posts: 1589
tepples wrote:
I think the limits on nested scope, include, and macro calls are there to avoid suddenly thrashing your swap and/or causing the kernel to OOM-kill processes critical to the system or UI if you accidentally cause infinite recursion.

Yep, it's in case of infinite recursion, since the assembler won't quite with an error otherwise. It's usually a setting you can change as well, since in extreme case you may indeed need higher limits.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 27 posts ]  Go to page 1, 2  Next

All times are UTC - 7 hours


Who is online

Users browsing this forum: DRW, lidnariq and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group