For subroutines I've been doing this strange form of calling convention, if you can call it that, where I use the first sixteen bytes of RAM for arguments
As do I, incidentally. Because subroutines call other subroutines, I document which bytes in $0000-$000F are overwritten by any particular subroutine.
and return data by using a pointer in X.
Could you give an example?
I don't really like this much, and it makes recursion basically impossible.
What recursion are you trying to accomplish in 27,000 cycles? The only thing I can anticipate needing recursion for would be a flood fill in a game like Rampart
or Magical Drop
or Puzzle League
, and that can be done as easily with a circular buffer of next things to check.