Dynamically generating ld65 config files

Discuss technical or other issues relating to programming the Nintendo Entertainment System, Famicom, or compatible systems. See the NESdev wiki for more information.

Moderator: Moderators

Post Reply
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Dynamically generating ld65 config files

Post by tokumaru »

I've been feeling annoyed at the limitations of ld65, more specifically at its inability to automatically align arbitrarily sized blocks to the upper end of PRG banks. Also, working on an MMC3 project, the number of MEMORY and SEGMENT entries in the config file is kinda massive, and the fact that banks can be mapped to 2 different slots means that I have to manually keep the source code and the config file in sync according to what banks I'll use where.

Then it occurred to me that I had been using ca65/ld65 to create text files containing debug information for a while, so I figured I could automate much of the ROM configuration using macros, and run ld65 once to generate a dynamic config file, and then a second time to generate the final binary. The first config file would be static, with large placeholder MEMORY areas and SEGMENTS just to gather the sizes of each segment, which are exported via the "define = yes" feature. This information can then be injected into the dynamic config file, which will be used for the final ld65 pass.

So yeah, that's the idea. I haven't implemented any of this yet, but the idea of being able to target any mapper without having to bother with config files, and at the same time extending their versatility (e.g. end-of-bank alignment), sounds great to me. RAM configuration doesn't really change from game to game (unless you're using something more exotic, like MMC5's ExRAM), so those settings could be constant, or you could use the same idea and do more advanced RAM mapping, to implement something cool like overlays.

I'll report back after I give this a shot, but in the mean time, feel free to share your thoughts.
User avatar
rainwarrior
Posts: 8735
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Dynamically generating ld65 config files

Post by rainwarrior »

Probably the ideal solution for that would be to add the capability to ld65 itself for "right-aligned segment".

An alternative to generating CFG files directly is to use the SYMBOLS section to import symbols with the addresses you want to use. Symbols could be exported from an extra object file created for your second pass, or possibly from the command line, but it'd probably let you use the same CFG file for both passes.

If you can generate the information you need for the address during the assemble pass and export it as a symbol, you might even be able to do the whole thing in one step.

For an example, in my ZENSF project I have a single CFG file that can be configured for several different ROM sizes via generated symbols. (Not doing a "right align" but it might give an idea of how to do it.)
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: Dynamically generating ld65 config files

Post by tokumaru »

I tried using SYMBOLS, but turns out they can only really be used for MEMORY parameters, not for a SEGMENT's "start" attribute, for example, which limits their use... If I want to place things in different dynamic positions of the same bank, I have to create multiple MEMORY areas for the same bank, which's extremely verbose if you have a lot of banks and doesn't represent well the actual structure of the ROM (a bank is supposed to be a single thing, otherwise the redundant "bank" attributes might cause inconsistencies, etc.).

Also, it seems that the size of a MEMORY area isn't known until it's output, so you don't have access to the size of the right-click block in order to know how much to pay the area preceeding it.

To overcome all these problems, I could only think of doing a preliminary pass to gather the sizes of everything, and inject dynamic values where they aren't normally allowed (SEGMENT's "start" attribute).

I would absolutely love a native "right-align" functionality, but I don't have the skills to implement it myself. But even that wouldn't solve all my problems. I constantly struggle with looking for a decent solution for RAM overlays too.

Recently I noticed that a new segment type showed up in ld65, the "overwrite" type. I immediately thought it would help me implement all the stuff I wanted, but being able to define attributes using nothing more than literal values quickly killed my hopes of doing anything useful with them. Unless I use a preliminary pass to create a new config file where the "hardcoded" values are actually dynamic, which is what I'll be trying now.
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Dynamically generating ld65 config files

Post by thefox »

I also chose to dynamically generate the linker configuration file for similar reasons. I decided to use Python to generate the file, though (https://github.com/fo-fo/ngin/blob/mast ... figurer.py), as I already had dependencies to it in my project.

I believe Movax (the person who made the ca65 HL macros) also did something similar to what you're suggesting by generating the linker configuration files using cc65 itself.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
User avatar
rainwarrior
Posts: 8735
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Dynamically generating ld65 config files

Post by rainwarrior »

thefox wrote:I believe Movax also did something similar to what you're suggesting by generating the linker configuration files using cc65 itself.
That's an interesting idea. I often use ca65 to build binary files (i.e. no 6502 instructions, only .byte statements etc.), but I suppose it could build text files reasonably well too, and you could use exported symbols from the first pass link to fill in the needed values?
tokumaru wrote:I tried using SYMBOLS, but turns out they can only really be used for MEMORY parameters, not for a SEGMENT's "start" attribute...
Oh, I didn't realize that was a limitation. That's a bit of a pain. Why wouldn't it work with SEGMENTs? :(
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: Dynamically generating ld65 config files

Post by tokumaru »

rainwarrior wrote:I often use ca65 to build binary files (i.e. no 6502 instructions, only .byte statements etc.), but I suppose it could build text files reasonably well too, and you could use exported symbols from the first pass link to fill in the needed values?
That's exactly what I proposed. I already generate text files with debug information, so I figured I could just as well generate a config file. The tricky part of writing text files is formatting numbers that aren't known at assembly time, because you can't use .sprintf. You have to use macros (like the one in the end of this post) to turn the numbers into expressions that can be evaluated later.
Oh, I didn't realize that was a limitation. That's a bit of a pain. Why wouldn't it work with SEGMENTs? :(
I have no idea why SYMBOLS don't work with SEGMENTs, but I simply couldn't make it work... ld65 says the values must be constant, which as far as I can tell, they are, since the values of the symbols are known at link time...
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: Dynamically generating ld65 config files

Post by tokumaru »

I swear, sometimes I spend so much time trying to work around problems/limitations in existing assemblers that I could very well code my own assembler and put all the features I need built into the program itself, rather than using tons of macros, preprocessors and other convoluted solutions.

I'm not saying mine would be better than what's already out there (it'd certainly have way less features than ca65), but it'd be shaped according to my own needs.
User avatar
Banshaku
Posts: 2417
Joined: Tue Jun 24, 2008 8:38 pm
Location: Japan
Contact:

Re: Dynamically generating ld65 config files

Post by Banshaku »

tokumaru wrote: I have no idea why SYMBOLS don't work with SEGMENTs, but I simply couldn't make it work... ld65 says the values must be constant, which as far as I can tell, they are, since the values of the symbols are known at link time...
I could be wrong but I think the reason it complain about it is because cc65 may allows during the chain of compiling to update the symbol at some point, making it not "constant" even thought the value itself is constant while in use. If this is the case, it could have impact on what you are trying to do, if you could change that value for each file, for example.

I could be wrong but that would be my educated guess.
russellsprouts
Posts: 53
Joined: Sun Jan 31, 2016 9:55 pm

Re: Dynamically generating ld65 config files

Post by russellsprouts »

Thanks everyone for the idea of using a multistage process to right-align blocks. I had this issue in my Forth project, and I've been solving it by just reserving as many bytes as possible at the start and updating it each time I get an overflow error. It's not ideal to say the least.
User avatar
thefox
Posts: 3134
Joined: Mon Jan 03, 2005 10:36 am
Location: 🇫🇮
Contact:

Re: Dynamically generating ld65 config files

Post by thefox »

tokumaru wrote:
Oh, I didn't realize that was a limitation. That's a bit of a pain. Why wouldn't it work with SEGMENTs? :(
I have no idea why SYMBOLS don't work with SEGMENTs, but I simply couldn't make it work... ld65 says the values must be constant, which as far as I can tell, they are, since the values of the symbols are known at link time...
Might be just a limitation of the implementation. Open an issue at GitHub if you want to find out whether there's a "real" reason behind the limitation.
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi
Oziphantom
Posts: 1565
Joined: Tue Feb 07, 2017 2:03 am

Re: Dynamically generating ld65 config files

Post by Oziphantom »

just going to remind people about TASS64 again http://tass64.sourceforge.net/ it has an internal linker and is N pass so you can do stuff where it assembles stuff, then you tell it where you want it to put it all in one operation. It also has an option to export a makefile for you. But in not 65816 mode you can't set the PC to anything outside of 16bits which can be a slight pain.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: Dynamically generating ld65 config files

Post by tokumaru »

I'm seriously designing my own assembler at this moment. I don't think it will be better than what's already out there, but it'll have everything I need and I'll have full control over it. I'm going for something closer to ASM6, which I really admire, but with a few things it's missing: functions, bank attribute for labels, bidirectional anonymous labels, better handling of local labels, a directive to print text without aborting assembly, stuff like that.
Rahsennor
Posts: 479
Joined: Thu Aug 20, 2015 3:09 am

Re: Dynamically generating ld65 config files

Post by Rahsennor »

tokumaru wrote:I'm seriously designing my own assembler at this moment.
And I just started writing one with no particular design in mind. :roll: If you need a spare code monkey, feel free to shoot me a PM.
User avatar
tokumaru
Posts: 12427
Joined: Sat Feb 12, 2005 9:43 pm
Location: Rio de Janeiro - Brazil

Re: Dynamically generating ld65 config files

Post by tokumaru »

Rahsennor, I just started a topic explaining what I have in mind. Check it out if you're interested!
Post Reply