Makefile, how to make it react to any content included?

You can talk about almost anything that you want to on this board.

Moderator: Moderators

User avatar
Banshaku
Posts: 2417
Joined: Tue Jun 24, 2008 8:38 pm
Location: Japan
Contact:

Makefile, how to make it react to any content included?

Post by Banshaku »

I'm really happy with my current makeFile that allows me to include file from anywhere in the project without writing the relative path as long the file name is unique, which is a good compromise. There is one last itch to scratch:

Included files, either h when used as headers in C, .inc for asm or binary files, etc doesn't make a specific file to be rebuilt since I think the makeFile is not aware of them (they are not part of any rules since they are included in target files).

Since I'm only using makeFile personally and not during work, I rarely have to go deep into the subject. I think what could help fix this issue is something related to DEPS (?) but I don't know if this is really the solution, don't know much about DEPS too and if the linker for cc65 works with that.

Once I figure out that last itch, I don't have to be aware of those file and forcefully clean the project to include those updated files. On a fast computer with SSD it is almost instantaneous so I didn't mind much but my "dinosaur" takes a more time to do so I would prefer to recompile only the files necessary.
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: Makefile, how to make it react to any content included?

Post by lidnariq »

Make syntax isn't too bad:

Code: Select all

resultingfile: dependency1 dependecy2 dependency3
<TABnotSpace>line to run to generate resultingfile from dependencies
I'm ignoring things like "should you use variables to enumerate the dependencies" or "Make has some convenient automatic variables like $@ and $<"...

Are you asking if there's an automated way to generate the dependency listing? If so, GCC has the -MM option to generate that. ... only for C files though, I think not assembly. You could make something using perl or grep and cut to extract a dependency list from assembly source.
User avatar
Banshaku
Posts: 2417
Joined: Tue Jun 24, 2008 8:38 pm
Location: Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku »

lidnariq wrote:Are you asking if there's an automated way to generate the dependency listing? If so, GCC has the -MM option to generate that. ... only for C files though, I think not assembly. You could make something using perl or grep and cut to extract a dependency list from assembly source.
This is a good question. The problem is, I don't know what to ask since I don't know what I'm supposed to do. From your questions it seem what I want to do is manages the dependencies. Your question is giving me ideas already and now I may be able to ask the appropriate ones ;)

Nothing is static in my make files regarding targets. This means I do not have to specify files one by one: it will finds all c/asm files from the current folder/sub folder and parse them. This mean all paths for includes are parsed and added to -I automatically.

So from your question, I think what I want to do is automatically manages the dependencies but is DEPS what I'm looking for? I never used it yet so I'm not sure. I mean, if I populate DEPS (right now let's forget how the list is made), this would make the compiler/linker (not sure which one use it yet) react and recompile specific c/asm files is some DEPS were modified (bin files, h files, included text/asm files tec)?

You are talking about GCC but I'm using ca65, cc65 (etc). Would this means that DEPS could be created with GCC and is not specific to ca65/cc65?

If I have to create a script to create that list then I'm fine with that, once I understand the flow and how it works. Since I never used DEPS yet and do not remember the details (I read briefly on the subject many months ago), I guess it is something that create a relationship between a target file and it's dependencies. If a dependency is modified, the target will be rebuilt.

edit:

usual spelling mistakes (...) and added extra words to clarify some phrases.
lidnariq
Posts: 11432
Joined: Sun Apr 13, 2008 11:12 am

Re: Makefile, how to make it react to any content included?

Post by lidnariq »

Banshaku wrote:Nothing is static in my make files regarding targets. This means I do not have to specify files one by one: it will finds all c/asm files from the current folder/sub folder and parse them. This mean all paths for includes are parsed and added to -I automatically.
There's some magic here, and this is probably where you want to look into dependencies. Make by itself doesn't do this.
So from your question, I think what I want to do is automatically manages the dependencies but is DEPS what I'm looking for? I never used it yet so I'm not sure. I mean, if I populate DEPS (right now let's forget how the list is made), this would make the compiler/linker (not sure which one use it yet) react and recompile specific c/asm files is some DEPS were modified (bin files, h files, included text/asm files tec)?
DEPS is not a magic variable in Make; something else is populating it. As to whether it's the right solution... I can't know.
You are talking about GCC but I'm using ca65, cc65 (etc). Would this means that DEPS could be created with GCC and is not specific to ca65/cc65?
Very likely. I don't know how GCC's -MM option works, so I don't know if a c file that compiles with cc65 will work with GCC -MM ...

(edit) quickly testing with na_th_an's Miedow, I get:

Code: Select all

$ gcc -I. -MM game.c
In file included from game.c:100:
engine/bolts.h:46: error: unterminated #if
 #if defined (ENABLE_TOUCHPLATES) || defined (ENABLE_LIGHTABLE)
 
game.o: game.c neslib-CNROM.h definitions.h assets/palettes.h \
 assets/precalcs.h assets/chr-rom-maps.h assets/enems0.h assets/enems1.h \
[... blah blah blah ...]
 engine/enemmods/enem_precalc_fanty.h engine/enemmods/enem_type_7.h \
 engine/enemmods/enem_saw.h engine/game.h engine/mainloop/flick_screen.h
i.e. both a valid dependency list and also a fatal error.
User avatar
Banshaku
Posts: 2417
Joined: Tue Jun 24, 2008 8:38 pm
Location: Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku »

From your answers it seems to make the targets be updated based on their dependencies is not something easy after all, since it requires to parse the target to find those dependencies.

I will need to understand more on the subject before getting rid of that itch. For now, to clean the project when those files may have been updated is the easiest solution but a brute force one.

edit:

When I think about it, it may be quite difficult to find the dependencies since the path of the file is not relative but based of the include paths, which complicate things. That is an interesting puzzle to solve but right now I do not have the time to figure it out unless the tool that uses the include path can creates the deps, hmmm...

ca65/cc65 have "--create-dep name" in their parameters. Could it be what I'm looking for? Need to test it.
calima
Posts: 1745
Joined: Tue Oct 06, 2015 10:16 am

Re: Makefile, how to make it react to any content included?

Post by calima »

$(OBJS): $(wildcard *.inc)
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Makefile, how to make it react to any content included?

Post by koitsu »

Suggested reading (not skimming), which includes examples: https://www.gnu.org/software/make/manua ... #Wildcards

One thing to be aware of that the wildcard() function is a GNU make-specific function. It does not exist on the BSDs (possibly including OS X), Solaris, etc.. Not that you can't get GNU make on those systems, but they do not use GNU make, they use their own make. Sometimes similar functionality to GNU make features, but syntactically completely different, is available in BSD makes. You just have to read the man page slowly. Otherwise, I second calima's recommendation.

I always suggest people read through the GNU make manual. Decades ago it sucked -- badly. These days, it contains lots of great examples and is actually quite helpful. Honestly Makefiles are not hard to create, but a lot of people do them badly and this (for decades) has proliferated. When I see a well-written one, it really does make me smile.

If said methodology isn't a viable option, then the "proper" way to do it is -- yup, you guessed it -- to statically declare the relevant filenames as dependencies for each thing. You may end up having to "redesign" how you do all of your includes/etc. in your source code to make things "a bit saner" Makefile-wise, but that's OK.

Otherwise you end up having to make a program that figures out what source files rely on what things, build the list of dependencies + put them into a file in proper make-compliant syntax that your main Makefile can utilise. In C, there is makedepend or gcc -M (I think most people prefer the latter these days; here's an example). You would need an equivalent for the assembler you're using. Results might be something like this (there are several ways to do it though; see the SO link for some ideas):

Code: Select all

deps.inc (this is the file that gets generated by whatever program you write):

thing_deps = file1.inc file2.inc
thing2_deps = file1.inc file3.inc others.inc
thing3_deps = file2.inc others.inc

Code: Select all

Makefile:

.include deps.inc

thing: $(thing_deps)
        commands_here

thing2: $(thing2_deps)
        commands_here

thing3: $(thing3_deps)
        commands_here
This is entirely just my opinion: I tend to find projects that have boatloads of included files make for very hard reading. They usually require an editor that can (must) have several files open at once just to "jump around" and figure out what all is declared where. I don't tend to write software this way. The belief is that "it provides organisation", which is true, but I've seen way too many projects in my time get overzealous with that approach and end up with absurd numbers of files. I've even seen some old Apple II-era source like this (things consisting of ~70 files and you can't figure out what's in what, so you're constantly doing things like grep -i foo * to find out where it is). People start out with good intentions, but by the end have a rat's nest.
User avatar
Banshaku
Posts: 2417
Joined: Tue Jun 24, 2008 8:38 pm
Location: Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku »

I do not know if my current makeFile would make you smile or cringe but because of this file, I'm now active again on nesdev :lol: I never made some for work since these days they only work with the latest hipster things so makeFiles doesn't quite fit in ^^;;

The only time I made some was for personal projects and a few times only. The first time was when I started on nesdev. I was using batch files and I found that not very convenient so I made my first one with static files. It did the job but was a pain to update everytime you added a new thing to the project. It was maybe based from one of Tepples example but it's been so long that I don't remember.

The second time was when I tried wxWidgets on a mac so I used a their makefile and updated it for my needs. Now I did use wildcards to find all the files but it was still very basic. I may have made a few extra ones based on this to compile other wxWidgets tests.

Then finally I tried to make one when I was trying to figure out how to compile the latest c runtime for cc65. My goal was to use C samples with the latest runtime and see how convenient coding in C was. Then when it was at last compiling, I started to use it with many C samples but everytime I was including X sample here or Y sample there in my test files, there was always issues with relative path in include that failed. After a while I got annoyed and since I found about include path, I decided to figure out how to avoid writing the path in the include to make it easier to manage. I did found a way but at the cost of having all you files have unique names. It's a compromise but it's quite useful.

So my goals were:
- a makeFile that I don't need to update unless I'm adding some new task in it
- allow include to be located arbitrary inside the project folder and remove the need to write the relative path to it, allowing to move files easily when refactoring

And this part is working quite well for my needs. The only issue is the dependencies that are not created but I didn't know much about deps: I just guessed that maybe this is what I need based on the name.

My directory structure looks like this:

Code: Select all

.
├── build -> all object, map, list files ends here (except main built is in root) same structure as src,
             omitted to make the list smaller
├── build_runtime -> c runtime library. Same structure has runtime
├── config -> cc65 config files
├── data  -> data folder, outside of src. This folder can be inside src if necessary
│   ├── chr
│   ├── intro
│   ├── logo
│   ├── maps
│   ├── menu
│   ├── metatiles
│   ├── music
│   ├── select
│   ├── stage
│   └── title
├── libs -> external libraries that could be included in some parts of the project
│   ├── famitone_sfx 
│   └── ft_drv -> famitracker
├── runtime -> C runtime library. Copy so the project is not affected if new version comes out
│   └── libsrc
│       ├── common
│       └── runtime
└── src -> main source folder. All files are dynamically added to the main build (same thing with data)
    ├── common
    ├── crt
    ├── data
    ├── libs
    ├── manager
    ├── nmi
    ├── stage
    ├── state
    └── utils
With my current makeFile, I just need to create a new c/s file in my src folder and the compiler will find it, check the include and even if the include is not in the same path, all path in the projects are extracted at the beginning then added on every file compile so unless the file is outside the project folder, it will find it. The same thing is done for incbin too. No more need to write ../../data/myFile.bin to include you file, "myFile.bin" is enough. And if in one week you move the file then no problem, just clean/recompile and everything will be found.

This is quite useful if you have not much time and may want to refactor the paths, name later. If everything was static then every small changes would be a pain to update. Maybe for a work related project it make sense but for a hobby project that you have very little time to work on it doesn't.

Included, my current makeFile. Opinion about what is interesting in it or what should be improved is always appreciated. I had to zip it because of phpbb restriction on MakeFile
Attachments
Makefile.zip
(2.23 KiB) Downloaded 234 times
3gengames
Formerly 65024U
Posts: 2284
Joined: Sat Mar 27, 2010 12:57 pm

Re: Makefile, how to make it react to any content included?

Post by 3gengames »

I think Meson would be able to make it work, I don't see why it wouldn't. I moved my projects too it, MUCH less headache over all. You can list deps of an executable even if they're not a .c file, afaik.
User avatar
pubby
Posts: 583
Joined: Thu Mar 31, 2016 11:15 am

Re: Makefile, how to make it react to any content included?

Post by pubby »

As lidnariq said, pass the C files to GCC using the -MM flag to generate the dependency graph, then -include the generated files into your makefile.

Here's how I did it in an old C++ project:

Code: Select all

DEPS:=$(SRCS:.cpp=.d)

-include $(DEPS)

%.o: %.cpp
	$(CXX) $(CXXFLAGS) -c $< -o $@

%.d: %.cpp
	$(CXX) $(CXXFLAGS) -MM -MP -MT '$(<:.cpp=.o) $@' -o $@ $<
User avatar
rainwarrior
Posts: 8734
Joined: Sun Jan 22, 2012 12:03 pm
Location: Canada
Contact:

Re: Makefile, how to make it react to any content included?

Post by rainwarrior »

If it's a CC65 project though, probably should use their --create-dep function instead of GCC's. (Both cc65 and ca65 have this.)
User avatar
Banshaku
Posts: 2417
Joined: Tue Jun 24, 2008 8:38 pm
Location: Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku »

Yes, it's a cc65 project since it's my current makeFile for building my engine ;) I saw those parameters in the configuration of cc65/ca65 yesterday and I remember briefly looking at them a few months ago when I was building my latest makefile. But... since I was not knowledgeable about dependencies but from the look of it, it seemed like something that may be useful later. So I kept it on my todo list to figure out later.

Basically that message was to see how to clear out that point on my todo list, by explaining my issue since my current issue seems related to that and making sure that deps was the solution. I guess after everyone comments I should try it, it just I'm not sure how to launch it yet. It seems like it would be an extra task in my current makefile, similar to the runtime library task, that needs to be run before the final target is executed.

What confuse me a little bit is cc65/ca65 is run on every files so I guess that it just append the results to the file defined by the "name" parameter of "--create-dep name". I need to check how to add that to my current makeFile.

And thanks everyone for the other deps example, it may come in handy for other projects than the nes one!
User avatar
koitsu
Posts: 4201
Joined: Sun Sep 19, 2004 9:28 pm
Location: A world gone mad

Re: Makefile, how to make it react to any content included?

Post by koitsu »

It sounds like we need a (simple/concise) Makefile example that uses cc65/ca65's --create-dep feature. I don't know what this flag outputs, so I can't even begin to do such an example (I still use other assemblers).

If pubby's (gcc-based) example is hard to understand, I can explain it line by line if that would help explain how to write a Makefile that could be potentially used with cc65/ca65? There's nothing wrong with his example, so I'm speaking generally here: for "make newbies", "magic variable" usage (ex. $@ $<) and string translations are never immediately obvious or self-explanatory. (Thankfully the GNU make docs are better today than ages ago!)
User avatar
Banshaku
Posts: 2417
Joined: Tue Jun 24, 2008 8:38 pm
Location: Japan
Contact:

Re: Makefile, how to make it react to any content included?

Post by Banshaku »

@Koitsu

I think I learned about those ($@ $<) while reading the gnu doc if I'm not mistaken but now without checking my makeFile and since I rarely use those... I don't remember what they means except that it's for either a list of files or a file based if it's on the target side or the one ^^;; Did you check the file? How much did you cringe? :lol:

I'm now fully awake ( :D ) and I remember another reason I made that makeFile: for beginners that may not know how to compile the runtime and to specify files in it. When I started to make it, there was some questions on the subject of neslib and the runtime not working with 2.16, I think. That got my interest and started to check how to do it. At that time, I still had no intention to work back on my projects: it was just something that got my interest and wanted to learn more about makefile too.

I started to look at the files and make a simple makeFile. Once it worked, since there was some questions on how to compile Shiru's example, I started to check neslib examples and how they work. From that point I thought "why not make some makeFile that does everything for the user?". My goal was to create a simple file that the user doesn't have to touch: you just put your files in src, the data in data and any library (neslib, famitone etc) in the libs folder since usually those ones are not compiled, they are included as-is. The makefile search all the subfolders and compile all for you. So you don't have to touch the file at all (maybe for symbolic links now but that's another story!).

Then the last point I thought was, a new user may create some kind of weird structure and move the file often. They may have issue with include path and don't know much about relative/absolute etc. So I searched how to make it easier and decided to make it so you don't have to write any path: the makeFile just add them all for you.
I wrote some simple doc at top that explain how to use it.

Once I finished and was happy by the result (geeky pleasures, I guess ^^;;;) I posted it to help such people and... Nobody used it :lol: Surprised? Not really, but I had fun doing it.

So this is why this file is so "convoluted" (? not sure this is the proper word), since my goal was to make a file that "auto-magically" compile your project. From there, when the users become knowledgeable enough, they can make their own anyway they want. I was sure it would help, it seems I was wrong.

After that, I got interested back to work with C code, tested more neslib, checked out all the methods one by one and figuring out how to interface with C (and I'm still asking question on the subject on nesdev because I have a tendency to do it half asleep and forgot most of it :lol:). Then I tried to take my old project and check how to use C code with it and this is how I got back to work on some nes coding.

So this makefile, even though the intention was for beginners, is actually very useful and I use it everyday now. After using it a lot, the only problem I found was dependencies may bite you once in a while if you forget to do a full recompile (your project only, the runtime is not affected) after modifying a H/bin/inc file: it makes for very interesting bugs :lol:

I will try to test deps for cc65/ca65 and share my results later.
User avatar
Jarhmander
Formerly ~J-@D!~
Posts: 569
Joined: Sun Mar 12, 2006 12:36 am
Location: Rive nord de Montréal

Re: Makefile, how to make it react to any content included?

Post by Jarhmander »

Looking at my old Makefile, here how I did it — and it works correctly:

Code: Select all

include main.mk

LD := ${CC}

OBJS := $(addprefix ${BUILD_DIR}/,${SOURCES:.s=.o})
DEPS := $(addprefix ${BUILD_DIR}/,${SOURCES:.s=.d})

.PHONY: all
all: ${TARGET}

.PHONY: clean
clean:
	rm -rf ${BUILD_DIR}
	${TGT_POSTCLEAN}

.PHONY: test
test : ${TARGET}
	mednafen ${TARGET}

${TARGET} : ${OBJS}
	${LD} ${TGT_LDFLAGS} ${OBJS} -o ${TARGET}

${BUILD_DIR}/%.o : %.s
	@mkdir -p $(dir $@)
	${AS} ${SRC_ASFLAGS} -o $@ ${CREATE_DEP} -c $<

${OBJS} : Makefile main.mk


-include ${DEPS}
The main.mk contains this:

Code: Select all

TARGET := out.nes

AS := cl65
CC := cl65

CREATE_DEP = --create-dep $(@:.o=.d)

SRC_ASFLAGS = -t none --asm-include-dir include -g --debug-info -Wa -l,$(@:.o=.lst)

TGT_LDFLAGS := -t nes -C linker/mapper69.ld -m $(TARGET:.nes=.map) -Wl --dbgfile,$(TARGET:.nes=.dbg)

TGT_POSTCLEAN := rm -f *.map *.dbg

BUILD_DIR := .build


SOURCES := foo.s
SOURCES += src/init.s
... other sources ...
There's a bit of useless complication here, because before rolling my own I used a Makefile I found on GitHub, and later when I wrote the Makefile I just kept the original main.mk I wrote. That explains why there are two files. Verified to work with GNU Make, on a Linux system, with cc65 V2.17 - Git 6c320f7d6 (there are certainly earlier versions that works fine). Note all things here are exactly portable — I see I used the -p flag with mkdir, for instance.

From what I can see, you can do the same with cl65 as with gcc: compile/assemble while outputting to a dependency (dep) file. It happens on the ${AS} ${SRC_ASFLAGS} -o $@ ${CREATE_DEP} -c $< line. All dep files are created along with ".o" files, and they are -include'd right at the end of the Makefile, so the automatic tracking of dependencies work as intended.

Why I use cl65 instead of ca65? Well, gcc is not the compiler, it's the compiler driver, a front-end to the actual compilers and linker. Might as well utilize cl65, the cc65's equivalent, if it works and simplify things a little...


RE an earlier question:
lidnariq wrote:Are you asking if there's an automated way to generate the dependency listing? If so, GCC has the -MM option to generate that. ... only for C files though, I think not assembly. You could make something using perl or grep and cut to extract a dependency list from assembly source.
gcc can generate the dep files from any file it preprocesses, really. The key point is that it must involve the preprocessor. So if you assemble an ".S" file, which is an "assembly file that must be preprocessed", gcc will happily run the preprocessor on it. However, to actually get dependency output with -MD (actually, -M), you have to use #include, not the GNU as (GAS) equivalent .include: it won't work otherwise, because with .include that's the assembler that includes the file, not the preprocessor. You'd guess correctly if you think I was bitten by this some time ago...
((λ (x) (x x)) (λ (x) (x x)))
Post Reply