Makefile, how to make it react to any content included?

You can talk about almost anything that you want to on this board.

Moderator: Moderators

User avatar
rainwarrior
Posts: 7642
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Makefile, how to make it react to any content included?

Post by rainwarrior » Fri Sep 14, 2018 5:15 pm

I only suggested it as a slightly better alternative to being able to accidentally forget to clean. ;P Not a serious suggestion, obviously using real dependencies is better.
gauauu wrote:But in small NES stuff? Yeah, it's what I do. I can live with an extra 3 seconds of compiling every time I change a header.
Yeah, I considered Make when I started my larger project, and decided to think about it again later if compilation ever got slow. It never did, the full build is still like 1 second. (This is even with like 400k of .byte data in there.)

It doesn't use the CC65 C compiler though, not sure if that is significantly slower? (I've never found it to have noticeable compile times when I have used it.)

Only time I've ever had significant compile time with CC65 was when Movax12 made that Super Mario Bros. disassembly that was composed almost entirely of complicated "high level" macros.

User avatar
gauauu
Posts: 640
Joined: Sat Jan 09, 2016 9:21 pm
Location: Central Illinois, USA
Contact:

Re: Makefile, how to make it react to any content included?

Post by gauauu » Fri Sep 14, 2018 6:54 pm

rainwarrior wrote:I only
Yeah, I considered Make when I started my larger project, and decided to think about it again later if compilation ever got slow. It never did, the full build is still like 1 second. (This is even with like 400k of .byte data in there.)
My cc65 builds can be a few (2-4) seconds for a full rebuild, particularly if has to rebuild generated assets. Fast enough that I don't care enough to properly set up something to manage the header dependency graph.

What really kills you is you have a bunch of rom data as C files. In my GBA game, each room map generated a C source file of data. Compiling all those could take a couple minutes.

User avatar
thefox
Posts: 3141
Joined: Mon Jan 03, 2005 10:36 am
Location: Tampere, Finland
Contact:

Re: Makefile, how to make it react to any content included?

Post by thefox » Sat Sep 15, 2018 1:53 am

I set up my asset compilation tools (mostly written in Python) to recompile when the source assets (like levels made in Tiled) change. There the compilation times start to stack up enough so that it starts to be a good idea to compile only what's needed.

I threw up a quick example of the cc65 CMake toolchain at https://github.com/fo-fo/cc65-toolchain-example
Download STREEMERZ for NES from fauxgame.com! — Some other stuff I've done: fo.aspekt.fi

User avatar
Banshaku
Posts: 2325
Joined: Tue Jun 24, 2008 8:38 pm
Location: Fukuoka, Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku » Sat Sep 15, 2018 9:52 am

@Jarhmander

I just did a quick test and the d files are created but the makefile doesn't react yet to the deps. Right now it is 2h in the morning so it may be the reason that I cannot find the cause ^^;; I think it may be that my $(DEPS) variable may not contain any value and when it does -include $(DEPS) fails to react. Once I find why I think it should work.

Does the location on -include makes a difference? I remember some example in GNU that was putting it more at the top but I don't think that's the reason.

I guess I should try tomorrow after sleeping.

edit:

Need to find at the least the value of DEPS. There was an error, which caused it to be empty but now it contains the list of d files but even if I modify a .h file make doesn't react on it. I tried with -include or include and no difference. This is the only thing left, it's almost working.

User avatar
Jarhmander
Formerly ~J-@D!~
Posts: 488
Joined: Sun Mar 12, 2006 12:36 am
Location: Rive nord de Montréal

Re: Makefile, how to make it react to any content included?

Post by Jarhmander » Sun Sep 16, 2018 6:45 am

Banshaku wrote:@Jarhmander

I just did a quick test and the d files are created but the makefile doesn't react yet to the deps. Right now it is 2h in the morning so it may be the reason that I cannot find the cause ^^;; I think it may be that my $(DEPS) variable may not contain any value and when it does -include $(DEPS) fails to react. Once I find why I think it should work.

Does the location on -include makes a difference? I remember some example in GNU that was putting it more at the top but I don't think that's the reason.

I guess I should try tomorrow after sleeping.

edit:

Need to find at the least the value of DEPS. There was an error, which caused it to be empty but now it contains the list of d files but even if I modify a .h file make doesn't react on it. I tried with -include or include and no difference. This is the only thing left, it's almost working.
Maybe the easiest way to diagnose this is to try my repo. It's small and uncomplicated. These makefiles works so if it doesn't when you try it, maybe it's related to the cc65 suite. I remember that there was an issue related to --create-dep in older versions of cc65, though I don't remember what was the problem exactly. If it does work however, it's certainly in how you wrote it.

Attach your modified makefile here and I'll look at how you did it. I cannot test it now because I'm on my puny Pinebook*, maybe this night (EDT).

* It's a nice machine, but certainly not a workhorse. And considering the fact that cc65 is not in this Ubuntu's repo, well I can build it, but only if I have time on my hands. Or cross-compile it, but it's a bit of a pain in its own...
((λ (x) (x x)) (λ (x) (x x)))

User avatar
Banshaku
Posts: 2325
Joined: Tue Jun 24, 2008 8:38 pm
Location: Fukuoka, Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku » Sun Sep 16, 2018 7:40 am

I will attach the latest version of my make file. As you will see, not much changed compared to the one I pasted in a few messages above.

The only difference is that I use deps for C files so I do not know if cc65 have issue, like you mentioned. I'm using the latest version, 2.17 unless mistaken.
Attachments
Makefile_20180916.zip
(2.35 KiB) Downloaded 144 times

User avatar
Jarhmander
Formerly ~J-@D!~
Posts: 488
Joined: Sun Mar 12, 2006 12:36 am
Location: Rive nord de Montréal

Re: Makefile, how to make it react to any content included?

Post by Jarhmander » Mon Sep 17, 2018 8:58 pm

I have found the problem. It's quite simple actually when we see the Makefile in action.

gcc and clang both compile "c" files to "o" directly; cc65 cannot. Instead, it outputs an assembly file which can be assembled. Because of this, the generated dep file looks like this:

Code: Select all

build/src/test.s:	src/test.c src/test.h

src/test.c src/test.h:

That's logical: it cannot know yet of the real output file, so dependency is expressed from the point of view of its real output, the "s" file. And here lies the entirety of the problem. Your Makefile does this with "c" files:

Code: Select all

### Main target files to compile ###
$(C_OBJ_FILES): $(BUILD_DIR)/%.o : %.c
	@mkdir -p $(@D)
	$(CC) $(CCFLAGS) $(CREATE_DEP) $< -o $(@:.o=.s)
	$(AS) $(@:.o=.s) $(ASFLAGS) -l $(@:.o=)_listing.txt -o $@
Here, the rules says that for every C_OBJ_FILES, each "o" files depends on "c" files. But nothing tells make about the "s" files generated along the way. Well, the dep file mentions some dependencies, yes, but make doesn't know that "o" files are generated from "s" files from "c" files; the recipe assembles "s" files behind his back.

There are two valid solutions: split this rules into two rules, one doing "c" → "s" and the other from "s" → "o", OR use cl65.
Yes, cl65, with the -c flag, will compile, assemble but not link source files. The dep file will nonetheless reflect the expected dependencies: now things depends on the "o" file, which the Makefile knows about. This is the solution I prefer, because it acts more like the other compilers, and it simplify the Makefile a (tiny) bit. There are the only things to watch for with cl65:
  • The order of options vs argument is important; it's better to put the source file at the end of the command line, because what's after it will not be taken into account;
  • The default target is C64, use -t none to counter this.
  • Most options are understood by cl65 directly, but if for some reason you need a specific compiler/assembler option that cl65 doesn't know about, then, and only then use the -Wc/-Wa option to pass that option through cl65.
So here's the diff which is quite small, and attached is the modified Makefile.

Code: Select all

diff --git a/Makefile b/Makefile
index 875a24b..2ffd942 100644
--- a/Makefile
+++ b/Makefile
@@ -44,8 +44,8 @@ TARGET = $(APP_NAME).nes
 
 ##################
 # CC65 related
-AS := ca65
-CC := cc65
+AS := cl65
+CC := cl65
 LD := ld65
 AR := ar65
 
@@ -95,8 +95,8 @@ FT_DEFINES += -D FT_SFX_STREAMS=4    # Set all 4 channels for SFX
 
 ###################
 # Compiler/linker flags
-ASFLAGS = -g -I $(SRC_DIR) $(INCLUDE_LIBS_FLAGS) $(INCLUDE_DATA_FLAGS) $(FT_DEFINES)
-CCFLAGS = --add-source -Oi -Cl -g $(INCLUDE_LIBS_FLAGS) 
+ASFLAGS = -c -t none -g -I $(SRC_DIR) $(INCLUDE_LIBS_FLAGS) $(INCLUDE_DATA_FLAGS) $(FT_DEFINES)
+CCFLAGS = -c -t none --add-source -Oi -Cl -g $(INCLUDE_LIBS_FLAGS)
 LDFLAGS = -Ln $(BUILD_DIR)/$(APP_NAME)_labels.txt -m $(BUILD_DIR)/$(APP_NAME)_map.txt --dbgfile $(APP_NAME).dbg -vm
 INCLUDE_DATA_FLAGS := $(addprefix -I ,$(DATA_DIR_LIST)) $(addprefix --bin-include-dir ,$(DATA_DIR_LIST)) $(addprefix --bin-include-dir ,$(SRC_DIR_LIST))
 INCLUDE_LIBS_FLAGS := $(addprefix -I ,$(SRC_DIR_LIST)) $(addprefix -I ,$(LIB_DIR_LIST))
@@ -128,12 +128,11 @@ print-%  : ; @echo $* = $($*)
 ### Main target files to compile ###
 $(C_OBJ_FILES): $(BUILD_DIR)/%.o : %.c
        @mkdir -p $(@D)
-       $(CC) $(CCFLAGS) $(CREATE_DEP) $< -o $(@:.o=.s)
-       $(AS) $(@:.o=.s) $(ASFLAGS -l $(@:.o=)_listing.txt -o $@
+       $(CC) $(CCFLAGS) $(CREATE_DEP) -l $(@:.o=)_listing.txt -o $@ $<
 
 $(S_OBJ_FILES): $(BUILD_DIR)/%.o : %.s 
        @mkdir -p $(@D)
-       $(AS) $< $(ASFLAGS) $(CREATE_DEP) -l $(@:.o=)_listing.txt -o $@
+       $(AS) $(ASFLAGS) $(CREATE_DEP) -l $(@:.o=)_listing.txt -o $@ $<
 
 ##########################
 ### Build runtime only ###
Attachments
Makefile.zip
(2.38 KiB) Downloaded 142 times
((λ (x) (x x)) (λ (x) (x x)))

User avatar
Banshaku
Posts: 2325
Joined: Tue Jun 24, 2008 8:38 pm
Location: Fukuoka, Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku » Mon Sep 17, 2018 11:25 pm

I see. I don't remember why I didn't use cl65 directly, maybe because I wanted to keep the .s file for some reason but today that wouldn't be an issue. Thank you for going that far to find the cause!

I did a quick compile and it's almost working. For some reason some include paths are not working anymore for s file. Once it compile and deps works as expected, I will let you know. The extension is .inc but that shouldn't be an issue. Must be a little something, will figure it out soon.

User avatar
Jarhmander
Formerly ~J-@D!~
Posts: 488
Joined: Sun Mar 12, 2006 12:36 am
Location: Rive nord de Montréal

Re: Makefile, how to make it react to any content included?

Post by Jarhmander » Wed Sep 19, 2018 8:05 pm

Have you found what's causing your issue? I'm not sure how the modifications break compilation of "s" files; if anything, I changed how the "c" files are compiled, so it would expect that if I broke something, that must be it. Assembly files are compiled through cl65, and I only made a slight reordering of the command-line flags.

That might be a bit extreme, but if you don't find what's causing this, you can attach the console output of a clean compile (e.g. with make | tee output.log), one with the old Makefile and the other with the new. I might figure out something from those outputs alone.
((λ (x) (x x)) (λ (x) (x x)))

User avatar
Banshaku
Posts: 2325
Joined: Tue Jun 24, 2008 8:38 pm
Location: Fukuoka, Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku » Wed Sep 19, 2018 9:03 pm

I didn't work on it recently but I do not think what you updated is the cause. From the way it's done, everything seems fine. My guess is (but I need to confirm by testing) maybe the include list is too long and cl65 could be failing on that because of the error received. Since the workflow didn't change except to switch from as/cc to cl, this seems to be the current logical explanation. It could be a bug in the parsing of -i but until I can prove it, this is just speculation.

The way my project work is, instead of putting relative path in every file, I just list all folder in src and data and include them on every item compiled. This avoid issue with relative path when using files from other projects and allow to move file without worrying about the include. The compromise to be done is that all include file name must be unique, if not then you have to write the real path in that case.

Except for the include path, I don't think what you wrote is the cause. Maybe what I will do is test this makeFile with a sample with less include and see if it does work. I updated the sfx test sample that glutock gave me with my makeFile so it would be the perfect test sample for that since only a few folder exist. If it works then test an s file with inc files and if it does work then maybe what I'm assuming is the cause is right.

I will see if I can test since it should be quite fast with a small sample.

User avatar
Banshaku
Posts: 2325
Joined: Tue Jun 24, 2008 8:38 pm
Location: Fukuoka, Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku » Wed Sep 19, 2018 9:41 pm

Sorry for the double post. Here's the error log. I provided a sample so it would be easier to figure out.

Code: Select all

cl65 -c -t none --add-source -Oi -Cl -g -I src -I libs -I libs/famitone/ -I libs/ft_drv/ -I libs/neslib/ --create-dep build/src/example.d -l build/src/example_listing.txt -o build/src/example.o src/example.c
cl65 -c -t none -g -I src -I src -I libs -I libs/famitone/ -I libs/ft_drv/ -I libs/neslib/ -I data --bin-include-dir data --bin-include-dir src -D FT_DPCM_ENABLE=1     -D FT_SFX_ENABLE=1      -D FT_THREAD=1          -D FT_PAL_SUPPORT=1     -D FT_NTSC_SUPPORT=1    -D FT_SFX_STREAMS=4     --create-dep build/src/crt0.d -l build/src/crt0_listing.txt -o build/src/crt0.o src/crt0.s
src/crt0.s(240): Error: Constant expression expected
src/crt0.s(257): Error: Cannot open include file `ft_drv/driver.s': No such file or directory
src/crt0.s(258): Error: Cannot open include file `neslib.s': No such file or directory
src/crt0.s(272): Error: Constant expression expected
src/crt0.s(279): Error: Constant expression expected
Makefile:139: recipe for target 'build/src/crt0.o' failed
make: *** [build/src/crt0.o] Error 1

my guess was wrong, it only fails with .s file actually so it cannot be the length of include paths. I will continue to test with that sample since it is easier to figure out from a simple set.

Small notes:
- MakeFile.old is the makeFile without update
- I added AS2 in new one since I wanted to do a quick test and wasn't sure the proper parameters to add for building lib. I will figure out later

edit:

The error was similar (I think) with my code, the constant expression I'm not sure but I will check during my lunch but I think it is almost the same. I will retest with my code later and add extra comment to this post and fix the sample is there is something wrong in it.

edit3:

removed edit2 since the doc as an example later for .s file. I was just mislead by how the top explanation was written. Continue to check possible causes. For the "Constant expression expected", it's like it doesn't recognize the -D parameters somehow. It was not an issue with as65.

edit4:

For c files, deps is working fine and was able to see it in action for h file, which is nice. Now I just need to figure out what it doesn't like with the parameters for that s file since executing that same command (minus -c -t none) with ca65 works fine. hmmm...

edit5:

Hopefully my last edit. My current solution to the problem is:

- CC uses cl65
- AS uses ca65

If used that way, both recognise when files are updated and compile properly! I don't know why cl65 fails with parameters for .s file since the same parameters works with ca65. It could be a bug in 2.17, who knows. For now, unless you think there is an issue to do it that way, I'm more than happy with the result.

Thanks!!!!
Attachments
shiru-ft-sfx-issue2_makefile_deps_tests.zip
(313.77 KiB) Downloaded 141 times

User avatar
Jarhmander
Formerly ~J-@D!~
Posts: 488
Joined: Sun Mar 12, 2006 12:36 am
Location: Rive nord de Montréal

Re: Makefile, how to make it react to any content included?

Post by Jarhmander » Wed Sep 19, 2018 11:14 pm

Hmm, this one is quite funny. From cl65's source code, -D and -I are only handed to cc65, not ca65. In the resulting ASFLAGS, substituting --asm-define for -D and --asm-include-dir for -I will make it work, though I don't know if it's really an improvement over just using ca65 at this point?

gcc's behavior is different:

Code: Select all

% gcc -c -x assembler-with-cpp -DFOO=42 -o test.o - <<.
    .global Foo

    Foo = FOO
.
% nm test.o
000000000000002a A Foo
I don't know if it deserves a bug report for cl65?
((λ (x) (x x)) (λ (x) (x x)))

User avatar
Banshaku
Posts: 2325
Joined: Tue Jun 24, 2008 8:38 pm
Location: Fukuoka, Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku » Wed Sep 19, 2018 11:43 pm

I see. So I was right about what I was suspecting but I usually try to think that "it must" be a user error so don't blame the tool until you're 100% sure. I make so many error that I usually blame myself first over everything else :lol:
Jarhmander wrote:I don't know if it deserves a bug report for cl65?
Since the doc doesn't mention that -I and -D doesn't work for ca65 then yes, I think it would be a good idea to create one.

If you read the doc from the top, the explanation gave me the false impression that the main target was for C after reading that part:
The cl65 compile and link utility may be used to convert, compile, assemble and link files. While the separate tools do just one step, cl65 knows how to build object files from C files (by calling the compiler, then the assembler) and other things.
But it could be that sometime I try to read "too much" between the lines and it didn't mean that ^^;;; After reading the doc further, they give some example with file extention so this is when I was sure that it was for both.

Where should the bug report be written, on github?

edit:

Oh, and thanks again for going that far to confirm, I really appreciate it. It now works so that will save a lot of time!

Post Reply