Thursday, October 19, 2017

My WWVB Radio Clock

Obelisk, a.k.a. O-3, is the third (and probably final) digital clock I've designed and built that is ultimately synchronized to the atomic clocks that serve as our standard for time in the U.S. It synchronizes to the sixty kilohertz transmission from radio station WWVB in Fort Collins Colorado. WWVB is operated by the National Institute for Standards and Technology (NIST). NIST has one of their big facilities in Boulder Colorado, where the F2 master atomic clock, the source of ultimate time and frequency reference for the United States, is located. Fort Collins is just a hour or so drive north of where I am located near Denver Colorado. But the WWVB time signal reaches most of the continental U.S. and beyond, typically at night when the sun doesn't interfere with the long-wave transmission.

Untitled

The hardware of Obelisk is very similar to my prior two clocks. It is based on a Raspberry Pi 3 Model B running Raspbian, a Linux distro based on Debian. It uses an Adafruit LCD display. And it includes a real-time clock board with a battery backup, based on the DS1307 chip, to maintain the time when the system isn't running.

What sets this clock apart is that it uses a tiny SYM-RFT-60 AM radio receiver configured out-of-the-box to pick up the WWVB sixty kilohertz transmission. The receiver provides just six pins: one for power, one for ground, two for the antenna, one for reset, and the last for the digitized and inverted data derived from the modulated radio signal. The reset and data pins are connected to GPIO pins on the Pi.

The WWVB radio signal is amplitude modulated: the reference state is 0 dBr and the asserted state is -17 dBr.  The receiver inverts this so that the reference state is read by the GPIO as a zero, and the asserted state is read as a one. The transmitter emits one pulse per second, with the leading edge of each pulse aligned to the beginning of a second as determined by an atomic clock located at the transmitter. The duration of the pulse is used to encode three symbols: 200 milliseconds is a binary zero, 500 milliseconds is a binary one, and 800 milliseconds is a special marker used to identify the boundaries in the data frame.

A frame consists of exactly sixty pulses, so there is always one frame per minute. The leading edge of the first pulse in the frame indicates the beginning of the first second, :00, in the current minute, and the leading edge of the last pulse of the frame indicates last second, :59.

Values in fields are encoded as binary coded decimal (BCD) digits. So, for example, the minute of the hour is encoded as two BCD fields, the tens portion, and the ones portion. The UTC minute and hour, the day of the year (Julian day), and the last two digits of the year, are encoded in fields in the frame, along with other useful stuff like whether the current year is a leap year, whether a leap second is going to be added at the end of the current month, whether Daylight Saving Time is in effect, and the difference between the UTC time and UT1 time. (UTC is kept within +/- 0.9 seconds of UT1, a time reference based on celestial observations that is relative to the Earth's rotation, through the introduction of leap seconds.)

Obelisk merely samples the GPIO pin in user-space using a periodic timer, and measures the pulse width to determine the three symbols. Most everything else is just the bookkeeping involved with decoding the fields in the frame and turning them into a useful timestamp. Not that that was completely trivial.

The hardest part of this process was designing the state machine to parse the language whose grammar is implied by the layout of the symbols in the frame. While the AM radio signal can be received throughout all or most of the continental United States, it is extremely susceptible to interference. (This is what Steely Dan was alluding to when they sang "no static at all" in their 1978 song FM.) The WWVB signal can be corrupted by the placement of the radio antenna, by other electronic devices (particularly LCD displays), and even by strong sunlight. So the Obelisk parser tends towards paranoia, with a lot of stuff for error detection and recovery.

One technique I have found useful to measure the quality of my clocks is to run the Network Time Protocol daemon on them and use my implementation as an NTP reference clock. Obelisk emulates a GPS time source by emitting standard National Marine Electronics Association (NMEA) RMC sentences containing a UTC time stamp every second. It also generates a Pulse Per Second (PPS) on another GPIO pin; the PPS is synchronized (subject to some software delay) with the leading edge of the WWVB pulses that are timed to begin at the start of every second.

NTP continuously compares the reference clock with clocks from other NTP servers on the internet. Querying the NTP daemon on Obelisk for its statistics often reveals that my implementation is pretty usable. It will never be as good as a stratum-0 reference clock (one that is disciplined by a cesium atomic clock), but it's not bad. Below you can see the output from the ntpq command that includes my radio clock masquerading as PPS and GPS reference clocks.

Untitled

WWVB radio clocks are inexpensive and common; there are several such off-the-shelf devices scattered around the Palatial Overclock Estate. And WWVB radio clocks can be quite small. One of the wristwatches in my collection is a Casio Wave Ceptor with a built in WWVB radio receiver which it uses to set itself. (It's also solar powered.)

Untitled

My first clock, Hourglass (O-1), synchronizes to the signals from the satellites in the Global Positioning System. Like other radio navigation systems I have studied - for example, LORAN, for Long Range Navigation - GPS is based on time: the accuracy of the GPS position fix depends on the precision of the timestamps generated by the GPS system. Each GPS satellite has multiple atomic clocks, each of which is synchronized to the U.S. Air Force master atomic clock at the GPS ground station near Colorado Springs Colorado. I described Hourglass in My Stratum-1 Desk Clock. It was a relatively straightforward project, using off the shelf hardware and open source software.

Hourglass on a Stand

My second clock, Astrolabe (O-2), similarly synchronized to GPS, but uses a miniature cesium atomic clock to maintain time if and when the GPS signal was lost or compromised. I described Astrolabe in My Stratum-0 Atomic Clock. The software for Astrolabe was almost identical to that of Hourglass, although I found the hardware design much more challenging; there was a fair amount of hardware hacking (for a software guy like me anyway) to integrate the board containing the chip-scale atomic clock with the rest of the system. The biggest problem was that the Raspberry Pi is a 3.3 volt device and the CSAC board is a 5 volt device with standard RS-232 12 volt serial output. The design was complex enough that I felt the need to design in test points through which I could examine both the cesium atomic clock and its uBlox GPS receiver independently of the rest of the system.

Untitled

Obelisk differs from these two clocks in that I had to write a substantial amount of software to decode the WWVB signal and put it in a form usable by the open source software in the clock. But that's what learning experiences are all about. Obelisk provides a user-space application I call wwvbtool. The lower level details of decoding the WWVB transmission is handled by functions in the Obelisk library, which can be used for other WWVB-based applications. wwvbtool also makes use of my Hazer (GPS sentence generation and parsing) and Diminuto (Linux systems programming) libraries. wwvbtool, Obelisk, Hazer, and Diminuto are all written in C. All of the code is open source - licensed under either the GPL or the LGPL - and can be found on GitHub.
https://github.com/coverclock/com-diag-obelisk 
https://github.com/coverclock/com-diag-hazer  
https://github.com/coverclock/com-diag-diminuto 
Although wwvbtool can be used from the command line, Obelisk runs it as a background daemon where it communicates with the GPS daemon gpsd, which in turn communicates with the NTP daemon ntpd. Obelisk also uses the same hourglass.py Python script I wrote for Hourglass and Astrolabe to maintain the LCD time display.

When Mrs. Overclock (a.k.a. Dr. Overclock, Medicine Woman) and I first moved to Colorado in 1989, I didn't appreciate that I was going to be living at a nexus of super precise time keeping, what with NIST's F2 atomic clock in Boulder, the WWVB NIST transmitter near Fort Collins, and the GPS ground station near Colorado Springs, all a relatively short drive west, north, and south respectively.

But precision timing and frequency sources keep cropping up again and again in my professional life, from the supercomputers and their associated peripherals and networks that I worked with at the National Center for Atmospheric Research in Boulder, to the traffic shaping and traffic policing firmware I wrote for ATM OC-3 optical networking products at Bell Labs north of Denver, to the geolocation and navigation systems in products I helped develop for the business aviation marketplace for a client near Denver.

The other day I was talking about Hourglass, Astrolabe, and Obelisk with a friend and colleague of mine. He repeated that aphorism "A man with more than one clock never knows what time it is." I had to remind him that my three clocks all display exactly the same time.

Thursday, September 07, 2017

John Sloan on Time and Space

Once again my friends and colleagues at Gogo Business Aviation in Broomfield Colorado indulged me, in the form of my alter ego John Sloan, by letting me come and give a talk entitled Time & Space on the historical connection between precision timekeeping and navigation. They taped it and put it on the 'tube. I had a great time, and hopefully I wasn't the only one. (I did, in my enthusiasm, misspeak a couple of times, and have made come corrective comments on the YouTube page.) Among lots of other things, I describe how both GPS and atomic clocks work.



A big thank you to the folks at Gogo for sponsoring this.

Wednesday, July 12, 2017

Mo My MIMO

I have a tough time thinking of the Internet of Things as a new technology arena, considering I've worked on all kinds of embedded systems that were connected wirelessly, directly or indirectly, to the Internet. These ranged from eight-bit micro controllers to in-flight entertainment systems for business jets.

Recently I've been futzing around with a couple of IoT devices:

Untitled

the Amazon Web Services IoT Button and

Untitled

the TP-Link Smart Plug Mini. I was inspired by the clever work of my friend, former colleague, and one time college flat mate Paul Moorman who used the IoT Button with a Smart Plug to implement a wireless light switch. Based on prior work by other folks who reverse engineered the Smart Plug (and which he referenced in his article), Paul controlled the Smart Plug by punching a hole in his firewall for the port on the Smart Plug from which it receives commands, 9999, and then writing an AWS Lambda script in Python to talk to it. Pressing the IoT button stimulates the Lambda script in the AWS cloud to send the appropriate message to the Smart Plug to turn the power to any connected device on or off. Pretty cool.

I'd be lying if I didn't admit I was a little envious of Paul's ingenuity.

Ultimately I came up with my own practical application - significantly, as it turned out, one that required no development or firewall changes on my part - of just the Smart Plug: to turn our "cat cam" - an existing web cam in our kitchen near the cats' food bowls - on and off from my iPhone. It was pretty trivial to do. On a recent trip from Denver to Phoenix I was able to turn the cat cam on remotely using the TP-Link Kasa app and then use the webcam's app to peek at our beloved feline overlords (long may they rule).

Here's the thing: I had to punch a hole in my firewall for a port forwarding rule to connect to the cat cam from the webcam app on my phone, resolving the address using dynamic DNS. But I didn't have to punch a hole for the Smart Plug, or use dynamic or any other kind of DNS identifying my home network. What had to be happening is that the Kasa app was connecting to a server in the inter webs, and the Smart Plug was making its own outgoing bidirectional TCP connection through my firewall to a server (perhaps the same one) also on the inter webs.

Untitled

My first stab at reverse engineering this was to dig out my AirPcap NX tool, a USB 2.0 device from Riverbed Technologies that allows you to look at WiFi traffic and, providing you have the WPA pass phrase and catch the wireless end point when it first connects to the WiFI network, decrypt and decode IP packets as they fly through the air. The AirPcap has drivers for Windows and it is recognized by Wireshark.

This pretty much went according to plan until I realized that I was only seeing the traffic from the server on the inter webs to the Smart Plug, not in the other direction. Peering closely at the TCP connection setup responses from the server, I could tell definitively that the Smart Plug was the connection instigator. The Smart Plug was establishing a TCP stream socket to port 50443 on server dev.tplinkcloud.com. That domain name resolved to a server that was hosted in the AWS cloud (and unrelated to the AWS IoT button implementation). I could see packets from the server to the Smart Plug, but not in the other direction.

This put me in a paroxysm of web searching.

What I think was happening (see disclaimer above) is that my home WiFi access point supports 3 x 3 MIMO (multiple input multiple output) WiFi channels: it exploits spatial multiplexing to divide the packet stream into three transmit and three receive WiFi streams, all on the same radio channel, and which are all in the air simultaneously, each being handled by a separate antenna. My AirPcap NX supports at most 2 x 2 MIMO; it simply could not see all the traffic in the air.

I was using the AirPcap on my company laptop, a Lenovo ThinkPad running Windows 7. As part of my research, almost by accident, I came across an article describing the WiFi radios used by Apple, and discovered that my personal MacBook Pro came equipped with a radio capable of 3 x 3 MIMO with the same WiFi channels and technologies as my access point. Even better, the MacOS WiFi drivers natively support "monitor" mode, equivalent to promiscuous mode for Ethernet wired interfaces. And Wireshark for MacOS understood all of this already.

It didn't take very long for me to verify all of this using my MacBook Pro. I had Wireshark on my personal laptop peeking at all of the WiFi traffic between the Smart Plug and the TP-Link server in the Amazon cloud.

Spur WLAM pcapng

I was pretty proud of myself until Paul asked me why I didn't just insert a SharkTap into the Ethernet connection between my cable modem and my access point and Wireshark the traffic from there. A little embaressed, I did that too.

SharkTap Between Cable Modem and Router

For sure that was the simpler solution, but it had the side effect of NATting all the IP addresses on my home network to the one address assigned via DHCP to my access point by my internet service provider. It made the Wireshark traces a little harder to grok, but really no big deal.

Never the less, I'm glad I wasn't as smart of Paul and so didn't think of that solution right off the bat. I now know I have another useful tool in my troubleshooting arsenal, one that came ready made right out of the box.