It is currently Mon Feb 18, 2019 1:30 am

All times are UTC - 7 hours





Post new topic Reply to topic  [ 6 posts ] 
Author Message
PostPosted: Tue Jan 29, 2019 4:56 am 
Offline

Joined: Fri Feb 24, 2012 12:09 pm
Posts: 812
The RTC in NDS/DSi handhelds forgets the time & date whenever disassembling the console & removing the battery, so I've been wondering to fix the time via internet.

DHCP provides several options for obtaining IP addresses for time server / ntp, and to for obtaining timezone info. I don't know how far it would be recommended to trust that information, or if one would often end up with wrong timezone or nonfunctional server IPs... but I gave it a try... and the result was worse than expected: My router doesn't seem to support any of that DHCP options at all : /
I've added time related options (2, 4, 42, 100, 101) to the list of requested options (via option 55) in the DHCP discover/request messages - and the router does simply ignore that requests. I've tried requesting a handful of other options, which were ignored, too. And tried NOT to request DNS (by removing option 6), which was ignored, too (ie. I still received the DNS address).

The router used is called "o2 HomeBox 6641" (which seems to be rebadged Zyxel hardware). Looking at its internal browser GUI, it does know the correct local time, but it seems to be unwilling to share the time & timezone info via DHCP. So it looks as if I could forget about using the router and DHCP for time. Or is there some trick? Like needing to enable time option requests via another option/version number in the DHCP packet? Or some general rule like always being able to use the router's IP as time server (eg. via 192.168.1.1:port123)?

The other idea would be using a list of known ntp IPs, or somethine called "pool.ntp.org" (which should work as "wildcard" for finding nearby time servers).
The downside would be missing timezone and daylight saving info. And it won't work if the hardcoded servers were shutdown someday.

Hmmmm, or much easier: I could transfer the RTC time from PC to NDS/DSi (alongsides with wifiboto uploads). Thinking about it... at the moment that looks like the best and most reliable method; avoiding all expected and unexpected problems with time servers.


Top
 Profile  
 
PostPosted: Tue Jan 29, 2019 11:26 am 
Offline

Joined: Sun Apr 13, 2008 11:12 am
Posts: 8136
Location: Seattle
nocash wrote:
And it won't work if the hardcoded servers were shutdown someday.
As long as you use the name instead of an IP address, it should be fine.

There are many stories of people getting that wrong, e.g. https://en.wikipedia.org/wiki/NTP_serve ... _and_abuse


Top
 Profile  
 
PostPosted: Tue Jan 29, 2019 2:48 pm 
Online
User avatar

Joined: Sun Sep 19, 2004 9:28 pm
Posts: 3861
Location: A world gone mad
Your Zyxel router's DHCP server doesn't seem to care (i.e. ignores / does not send back to the client) many different DHCP options.

You added (presumably to the DHCP client) the ability to request the following DHCP options:

2 = Time Offset in Seconds from UTC (note: deprecated by options 100 and 101)
4 = Time Server (this is not NTP, this is the classic RFC 868 time server; literally nothing uses this any more)
42 = NTP Servers (this is the correct DHCP option for NTP server transferral)
100 = PCode / IEEE 1003.1 TZ String -- refer to RFC 4833, DHCPv4
101 = TCode / Reference to the TZ Database -- refer to RFC 4833, DHCPv4

So you can remove inclusion of 2 and 4 in your client. Honest. You probably should not use 100 or 101 either -- let the user of the device choose their own timezone (more on that in a moment), which includes in this case (contextually) locale, thus whether or not their area honours DST or not. (NTP always uses UTC universally)

You ask "is there a trick (to making this work)" and the answer is no. The problem is that your router's DHCP server is not delegating these options to the client. Because you're using a router with a firmware that's black box / closed source, your only option is really to open a support ticket with Zyxel asking them for these features in their DHCP server.

You **CANNOT AND SHOULD NOT** assume by default that the NTP server is the same IP address as the DHCP server. If you are doing that (you don't say you are, but you ask if there's some other way to get it to work), then that's wrong. In fact, I have very, VERY rarely ever seen consumer or open-source routers running ntpd (NTP daemon/server). What may not be known to you is that most routers do not have any kind of battery for RTC clock time -- every time they start up, their clock time is 0 (the epoch, which depending your timezone, is usually December 31 1969 @ 23:59:59). They do not keep track of time very well either (lots of clock drift very rapidly, usually within a few minutes). Cable modems are like this too. I'm familiar with all of this from the Tomato/TomatoUSB firmware project, as I was one of the people who was quite familiar with NTP and the nuances, plus Broadcom crap (and it is crap).

For time synchronisation on the router (i.e. the router having the correct date/time itself), odds are Zyxel uses NTP but what servers they're using is unknown. Vendors often hard-coding this, rather than letting people configure it, and sometimes they even hard-code their own IP addresses of their own NTP server. lidnariq's link to the NTP abuse Wikipedia article is highly relevant here -- see the D-Link section. Respectfully, Chinese vendors literally do not care about proper implementation or netiquette, which is how stuff like this happens.

My general opinion is that black-box routers with closed-source firmwares are pretty awful. It's why I was part of Tomato/TomatoUSB for so long. Now I use Ubiquiti equipment (ex. EdgeRouter X SFP) which is Linux-based, but the rest of the OS is closed-source. Their products are highly configurable, so you can end up restarting daemons etc. and look at lower-level logs, plus use tcpdump etc.. It's all stuff I find absolutely necessary given my line of work and need to have insights into things when they misbehave. The DHCP server on their products, BTW, is ISC's DHCP server, so you can configure it easily to delegate NTP servers (by IP address) to DHCP clients -- and it works, because I actually use it for exactly that.

Now let's talk about NTP servers and general setup:

You should be wary using pool.ntp.org. It is a common go-to default for people who aren't familiar with NTP. The mindset is: "oh wow! A magic hostname that I can use with an NTP client and it'll just work!" While that's true, there are major problems with the overall model/implementation that nobody thinks about:

1. pool.ntp.org requires functional DNS to resolve. This means whatever is acting as the NTP client has to ensure that its network stack is functional (it has a proper public IP address, etc.) *before* attempting any NTP queries, otherwise they'll fail. Whether or not the client retries is entirely up to the client -- and you'd be surprised how often they don't. The same goes for NTP servers that use another NTP server a source of NTP data (e.g. an NTP server that uses ntp.pool.org to sync its time, and then answers queries from NTP clients on a LAN).

Hard-coding IP addresses from pool.ntp.org isn't a wise idea, cuz the list changes all the time. I talk more about more "stable" NTP servers later on, but you should not be hard-coding those IPs in a firmware/product either (for reasons I just covered above, re: D-Link section); you would need permission from those NTP server maintainers too, despite them being public servers.

2. pool.ntp.org will return a dynamically changing list of A records of supposed NTP servers that are on the Internet. The DNS record has a 150 second TTL, which means every 150 seconds the list of servers returned for the DNS lookup change. Usually 4 A records are returned.

NTP clients tend to ask for the NTP servers once. They often do not re-query the same DNS record for new servers; i.e. they do this one time and assume those servers will be up/functional/working until the next time the NTP client is restarted.

3. In my experience, most of the NTP servers returned by ntp.pool.org are sub-par quality. They're often stratum 3 or 4, and I can't tell you how many times I've seen them go down for whatever reason -- they just stop responding to NTP requests. They are not "stable" servers. The design/premise behind pool.ntp.org is "if you give people a bunch of NTP servers at once, who cares how reliable they are", which is a really awful model IMO. The servers are basically random people on the Internet who apply to have their server added to the list. Yes really.

4. pool.ntp.org is "worldwide" set of servers. https://www.ntppool.org/ has details of how it's all set up, but if you read their "Use" page, you'll find that there are "sub-domain" FQDNs that break down by geographic region. I think you're in Europe somewhere, so you'd want europe.pool.ntp.org.

5. Back to timezones: https://www.ntppool.org/en/use.html at the bottom of the page lists of the nuances of all of this. You'll see their mention of timezones there.

My general recommendation for Good NTP Housekeeping is that you go through this list and this list and pick some public servers. Specifically: pick a SINGLE stratum one server, and pick 3 or 4 stratrum 2 servers. Try to pick servers in different geographic locations, e.g. do not pick 2 or 3 that are all in, say, Hamburg. You want network and geographic redundancy if one goes down.


Top
 Profile  
 
PostPosted: Tue Jan 29, 2019 4:25 pm 
Offline

Joined: Sun Apr 13, 2008 11:12 am
Posts: 8136
Location: Seattle
koitsu wrote:
You should be wary using pool.ntp.org. It is a common go-to default for people who aren't familiar with NTP. The mindset is: "oh wow! A magic hostname that I can use with an NTP client and it'll just work!" While that's true, there are major problems with the overall model/implementation that nobody thinks about:
To be fair, most people's needs really would be adequately met with the largely-deprecated DAYTIME service. Most end users don't need greater absolute precision. (Not saying they don't exist. Just that it's not all that common, and giving stratum 4 precision by default sure sounds reasonable to me)

Most battery-powered devices don't even keep good enough relative precision for the initial error to even be able to matter.

Quote:
1. pool.ntp.org requires functional DNS to resolve.
That's an argument in its favor, not against it. The laziest and problematic alternative is to hard-code an IP address... the entire point of the wikipedia article I linked is illustrating how that can (perhaps even will) backfire.


Top
 Profile  
 
PostPosted: Tue Jan 29, 2019 5:09 pm 
Online
User avatar

Joined: Sun Sep 19, 2004 9:28 pm
Posts: 3861
Location: A world gone mad
I don't agree with hard-coding IP addresses of NTP servers in embedded devices -- it's bad, for all the reasons demonstrated. I'm not saying do that. I'm saying DNS is fine to use but pool.ntp.org is flaky. Let me try to reiterate:

pool.ntp.org requiring DNS is not necessarily a strength when you consider all my other points, re: the NTP servers in the A records returned are not necessarily reliable, nor is the list consistent (see: RR DNS being used, with a TTL of 150 seconds). NTP daemons (servers), one having performed the record lookup, generally *do not* resolve the FQDN again at a later date -- they do it once at start-up and only once. Thus, reliable NTP servers (read: stable availability) is important. You can't get that with pool.ntp.org. What you *can* get with pool.ntp.org is "a best-effort list of NTP servers" -- these would generally work well for one-time synchronisation, such as if using rdate/ntpdate at boot.

Proper NTP setups consist of an NTP server running on a LAN with a static IP -- i.e. an IP address that the DHCP server can delegate to clients via option 42 -- which should always be available and doesn't necessarily require DNS. That NTP server should have a stable clock -- it can be a small PC, though consumer embedded devices are usually a very bad choice because they all tend to suffer from bad clock drift and high clock frequency variance -- and combines that with several servers on the Internet (diverse geographically and network-wise). By providing an NTP server on a LAN, you no longer have N clients on your LAN having to worry about the state of the Internet or DNS or anything else because 192.168.1.123 is always up. Adding more LAN NTP servers (to create a pool, where they even talk to one another for synchronisation) is a good idea too, but obviously overkill for a home network.

When NTP goes down or stops working properly, the effects can be dire. A common one that people rarely think of is SSL CA validation, where a proper system clock on any TLS/SSL client is required else validation can/will fail. Quite often this drives users bonkers because absolutely nothing SSL-stack-wise gives good details as to that being the root cause; the error is just some nebulous failure. Even the most senior of SAs often lose their minds over this. Another example where time accuracy is important is 2FA (2-factor authentication), where a skewed clock -- even by a few full seconds (varies per 2FA provider) -- can result in validation failure.

The reason I'm not big on strat 4 is because in my experience, NTP servers that deep in the hop list tend to be "best-effort". I've had a lot more success in general with strat 2 and 3 servers, and I always recommend one strat 1 if you can get access to it. I've worked at places that tried relying on a redundant set of NTP servers across several datacenters, whose sources of truth were pool.ntp.org and each other, and it was quite common to see 2 or more of the pool.ntp.org servers have horribly skewed clocks (enough that ntpd would mark them x) or situations where upon ntpd starting, multiple servers were stuck in .INIT., indicating they could never be contacted (no response to NTP packets -- ever). The situation was rectified by purchasing a couple GPS time servers (thus stratum 1) and removing pool.ntp.org from the configuration and instead picking a good/static list of strat 2 and 3 NTP servers from the links I provided. What I've used for years (over 10) are the same 5 public NTP servers: 1 strat 1, 3 strat 2, and 1 strat 3 -- all in different geographic and network locations.

In short: I really don't like pool.ntp.org. DNS for NTP server lookup is fine. My DHCP server on my LAN delegates option 42 with an IP address of an NTP server I run on my LAN (because there's no guarantee DNS resolution is necessarily working -- DHCP server and DNS server are two different boxes on my network), and that server is configured as described in the above paragraph. Never failed me.


Top
 Profile  
 
PostPosted: Tue Jan 29, 2019 7:52 pm 
Offline

Joined: Fri Feb 24, 2012 12:09 pm
Posts: 812
lidnariq wrote:
There are many stories of people getting that wrong, e.g. https://en.wikipedia.org/wiki/NTP_serve ... _and_abuse

Good point, I wasn't aware that traffic caused by improper NTP use could seriously overload universities or individuals whom are running NTP servers. Hah, in fact that might be a good reason for some such servers getting shutdown in future, especially those that aren't run by governments or big companies (which might get moved or shutdown for other reasons).

It doesn't really matter if the server name, or the server IP address are getting overloaded with mass-requests from mass-produced stuff - either one is getting overloaded either way.
The recommended "should-have-been" workarounds for the dlink issue were providing automatic updates to fix possible problems, and/or passing the requests to your own NTP server/domain instead of risking to render someone else's server useless (though, thinking about it, the latter might be preferred in some situations).
The indirection via ntp pool might help to 'spread' the problem (instead of focusing on overloading a specific server), but it won't help too much if the overload is getting too huge (like someday soon when there are more electronic gadgets than mammals on this planet).

koitsu wrote:
You added (presumably to the DHCP client) the ability to request the following DHCP options: 2... 4... 42... 100... 101.

Yes, those. I had originally tried 42, and then tried 2 and 4 to see if I could get at least get anything time-related. The options 100 and 101 for timezones would have been potentially useful (if they were working, and easy to interprete, and contain the correct local timezone). Ideally, I would have hoped that the local network/telecom provider would hand out all info about time and timezone (alongsides when handing out a global IP address to the router). But that seems to be science fiction.

Thanks for the rundown on other possible issues! Accuracy won't matter for NDS/DSi consoles. My console has been stuck at year 2000 for the past some years because I didn't bother to enter the correct time after battery removal. And my PC's clock is usually off by 5-15 minutes (I don't care so much, but yeah, it can be a problem when buying essentials like food/tobacco before shops are closing at night). Anyways, I would probably be fine with stratum XXIV. A shutdown or defunct NTP with wrong century might be a problem though (not that I'd care that much about which month or century I am in... a rewind to 1990's wouldn't be bad though).

Timezones with DST daylight saving time can be also difficult. In my experience DST is getting changed every once and then, so hardcoded databases won't help for that purpose. Best I could think of would be user-configurable DST settings in form of three parameters:
Code:
1st/2nd/3rd/4th/Last ... Mon/Tue/Wed/Thu/Fri/Sat/Sun/AnyDay ... of Jan/Feb/Mar/Apr/May/Jun/Jul/Aug/Sep/Oct/Nov/Dec
That should cover most DST variants, excepts those like "Upon pale dead moon after hell got frozen".
Btw. one old dlink router here (now used as mere switch/access point) doesn't remotely support such things, it can support only "1st..31st Day" for DST (nothing like "Last Sunday" of month or the like). And for the manual date setting, it supports a maximum range of years 2002...2007. It can also do NTP (there moving me to December 2005). It's a DI-524, and seems to be even older than those ones that had caused the NTP overload troubles (or maybe it can do that, too).

DNS won't be too much of a problem in this case. Except, I tried yesterday, and one minor problem was that I had too less memory allocated (my dswifi-based code has some DNS cache for last some 256-byte server names; taking up some kilobytes - but of course I could allocate more memory, or completely remove the caching feature if needed).

My plan was to support RTC updating so that other NDS/DSi users (if any) could use that feature, too. Telling them to buy better routers, or to request new router firmwares with DHCP-related updates won't work too well. So DHCP would work only as optional feature.
NTP pool (and/or a manually entered NTP server name/IP) should work... and might be fun to try for curiosity. But it won't resolve timezone/dst, and it's kinda ridiculous to try to make it robust & flexible enough to survive the next some decades... the current network protocols are probably getting discontinued before hell freezes and before the NDS/DSi retro-scene finally kicks off.

The PC's RTC is really the best choice here. I had planned to retrieve the network time only on wifiboot uploads anyways (not on firmware power up), so I can as well upload the PC time, matched to DST/timezone, and the OS time can be easily matched to the BCD time format of the NDS/DSi (without needing to care about converting seconds to leap years). And most modern PCs are having their RTC synced with NTP anyways.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 

All times are UTC - 7 hours


Who is online

Users browsing this forum: Dwedit, Garth and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group