Assessment for Judging

Moderator: Moderators

Post Reply
M_Tee
Posts: 430
Joined: Sat Mar 30, 2013 12:24 am
Contact:

Assessment for Judging

Post by M_Tee »

Being a teacher, assessment methods (rubrics, grading scales, and the like) are all of particular interest to me.

How are you each approaching judging?

What factors are you including in the overall categories, etc.?
na_th_an
Posts: 558
Joined: Mon May 27, 2013 9:40 am

Re: Assessment for Judging

Post by na_th_an »

It's hard to tell. I usually begin from top to bottom. I usually pick up muy most favourite in each category and give it full marks if I think it deserves it, or almost full marks. Then I pick up my next favourite, and measure how much "worse" it is so I give it the same, 1 less or 2 less or ...

When I'm done I reconsider my decisions. Maybe I was too harsh with my least favourite. Maybe the entries awarded the less points should have higher marks.

So it is quite relative. I don't know if this is a good or bad method, but it's the method I use most of the time in every task where I need to qualify or simply pick an arbitrary order to stuff.

On top of that, I think each one has a different approach to what topic or category is most important. For example, I don't care if a game is original as long as it is fun. Well, I do care that it is original, but that's less important. I don't care if it's technically top-notch as long as it looks good and plays great, and feels polished and complete.
User avatar
FrankenGraphics
Formerly WheelInventor
Posts: 2064
Joined: Thu Apr 14, 2016 2:55 am
Location: Gothenburg, Sweden
Contact:

Re: Assessment for Judging

Post by FrankenGraphics »

One way, though it requires a bit of prequisite legwork, is to set up atomized and universally functioning merits, and assign them a value (nominally 1). Each merit which you find true rewards a point (or a half, or whatever value you've assigned to that merit). Since some categories can reward more points than others, you'd either have to come up with more merits or attribute them with more points each.

Ie.
the apple was:
-rinsed (1pt)
-good for baking (1pt)
it had:
-an appetizing colour (1pt)
-a nice scent (1pt)
-a distinct flavour (1pt)
-a nice texture (1pt)

Maybe if the apple had a really nice scent, it might be worth 2pts.

Or the merits could be accumulative, perhaps like this:
-the student has grasped the elementary concepts needed to pass the course and continue with the next (threshold 1pt)
-the student has been able to reflect on the concepts and relate them to other related fields of study (threshold 3pt)
-the student has achieved a confident and profound understanding of the subject. (threshold 5pts)
User avatar
dougeff
Posts: 3078
Joined: Fri May 08, 2015 7:17 pm

Re: Assessment for Judging

Post by dougeff »

I ordered them by how much I like them, then adjusted their scores until the numbers came out the way I wanted. Unfair? maybe.

I also subtracted points for bugs at submission time. Sorry. I know alot of them had been fixed before voting, but deadlines are deadlines. I'm a stickler.
nesdoug.com -- blog/tutorial on programming for the NES
User avatar
gauauu
Posts: 779
Joined: Sat Jan 09, 2016 9:21 pm
Location: Central Illinois, USA
Contact:

Re: Assessment for Judging

Post by gauauu »

dougeff wrote: I also subtracted points for bugs at submission time. Sorry. I know alot of them had been fixed before voting, but deadlines are deadlines. I'm a stickler.
I think it was mentioned somewhere that we should be rating them based on the version available at the deadline/submission, and not based on later improvements, so I agree with this.
na_th_an
Posts: 558
Joined: Mon May 27, 2013 9:40 am

Re: Assessment for Judging

Post by na_th_an »

That's why I haven't still commited to introduce any of the enhancements / corrections suggested for my entry. I will do when the results are posted, for the cart.
User avatar
toggle switch
Posts: 139
Joined: Fri Sep 30, 2016 8:57 pm

Re: Assessment for Judging

Post by toggle switch »

i am also holding off on updates until voting is completed, and will be voting on submissions, not edited versions.
M_Tee
Posts: 430
Joined: Sat Mar 30, 2013 12:24 am
Contact:

Re: Assessment for Judging

Post by M_Tee »

I feel bad voting on games I haven't been able to beat.

Is it appropriate to PM requesting savestates for endings or that show essential content a developer might want included in the judging of their game?

Did anyone get stuck in Grunio? If so, I'd gladly PM a password list and a savestate that shows the all-carrots ending.
calima
Posts: 1745
Joined: Tue Oct 06, 2015 10:16 am

Re: Assessment for Judging

Post by calima »

I only got about two thirds in on Grunio, at that point I just couldn't figure out any solutions. No need to send a password list though.
na_th_an
Posts: 558
Joined: Mon May 27, 2013 9:40 am

Re: Assessment for Judging

Post by na_th_an »

There's a cheat in Miedow you can use to skip levels. It's not that hard to find (specially if you check the sources ;) ), but if somebody is interested just PM me.

The ending sequence (which goes from the moment Cheril "awakes" after level 10 to the end and which is playable) is quite nice to experience.
Erockbrox
Posts: 397
Joined: Sun Nov 23, 2014 12:16 pm

Re: Assessment for Judging

Post by Erockbrox »

I am unfamiliar with voting on homebrews or the contest in general. Can anyone vote? When is the voting? Are there rules explaining the voting and contest process?
M_Tee
Posts: 430
Joined: Sat Mar 30, 2013 12:24 am
Contact:

Re: Assessment for Judging

Post by M_Tee »

http://nesdevcompo.nintendoage.com/rules/2017/#judging
Although judging is not open for anyone (essential for preventing it from becoming an internet popularity contest, IMO), I'd still love to hear your assessment of our entry at least.
na_th_an
Posts: 558
Joined: Mon May 27, 2013 9:40 am

Re: Assessment for Judging

Post by na_th_an »

I think it would be nice if people posted their own ratings just for fun, specially if commentary was done. I always like to know what people think about my design choices because, as some of you will agree, working for some time in your project basicly blindfolds you.
M_Tee
Posts: 430
Joined: Sat Mar 30, 2013 12:24 am
Contact:

Re: Assessment for Judging

Post by M_Tee »

Thanks for the insight into your approaches.

Judging was definitely enjoyable. It's nice to be motivated to take the time to play and analyze these games more than I may have gotten around to otherwise.

Łukasz and I assessed separately and then planned to submit the average of our scores. I was caught a bit off guard by being restricted to whole number scores. At first, I'd just planned to round up any category entry that fell between whole numbers. But then I realized some games were getting total score boosts of 1.5 points or more (as they had multiple categories being rounded up). So, then we decided to round each (non whole number) category up or down to ensure that no game received more than a half point advantage from rounding.

As for my half of our judging, I assessed each game independently of each other, using the attached sheet which broke down most categories into smaller subcategories and a scale for each.
NesDevMTeeScorecard.pdf
(23.08 KiB) Downloaded 657 times
Anyway, I printed these and penciled them in fairly messily during my last play session, but if anyone wants my individual assessment of their entry, I'll gladly type up a legible copy with notes when I get a chance and shoot it your way.
Post Reply