After a long period of our SatNOGS station being well-behaved, we’ve recently had a sequence of interesting problems. This first came to light when our station had a number of “Bad” observations.

Bad observations are marked with red color, good observations with green.

The waterfall diagrams were empty, except for terrestrial noise.

This turned out to have a rather simple cause: The rotor controller was not responding. 

$ rotctl -m 603 -r /dev/ttyUSB0 p
get_pos: error = Communication timed out

However, before we were able to debug it, it mysteriously started working again. We were not absolutely completely 100% sure what was up with that, but at least we got good satellite observations again. This problem will probably crop up again sometime in the future.

Nice and green observations.

Waterfall diagrams with satellite signals:

This could have all been well and dandy, and this blog post could have had a happy ending. But after a day of observational bliss, we were back to observations with no satellite signals.

Observations are marked with “Failed”/black this time in order to not perturb the satellite statistics with our malfunctioning ground station.

Empty waterfall diagrams:

It was a bit confusing since it happened straight after the rotor failure, and we were therefore on the wrong debugging track for quite some time. It turned out to actually be something else entirely.

A lot of exciting things happen here, but you can’t see any of them since the receiver is broken.

Firing up gqrx against the SatNOGS SDR (USRP N210), there was no measurable difference in the radio output when we 1) turned on and off the LNA, 2) switched the antenna switch, 3) switched USRP antenna input ports, 4) attempted to receive LA2VHF, and 5) attempted to TX with a hand-held radio close to the USRP. We checked all these on a different radio, and we got nice responses on all five points. Conclusion: The SatNOGS radio is dead! Earlier rotor controller failures were entirely unrelated.

Maybe it’ll start working again if you stare at it long enough.

It turns out that the waterfall diagrams were also hinting at what kind of failures we were coping with. We live in a very noisy environment, and strange, additional signals are expected in all satellite observations.

  • Malfunctioning LNA.
  • Malfunctioning rotor.
  • Malfunctioning radio.

These noise sources are present as curved lines. The noise has the same frequency the entire time, but the frequency of the radio is adjusted against the expected Doppler shift of the satellite, and yields a curved appearance to something which should have been a straight line.

With rotor, but no LNA, additional broad-band noise sources appear at the lower end of the diagram due to the rotor pointing in different directions. With no rotor, the noise sources don’t change. With malfunctioning radio, there are no terrestrial noise sources at all, and no discernible difference to the waterfall despite the Doppler shift frequency adjustment.

We had similar symptoms on the same USRP a couple of years ago, where one of the internal LNAs and a voltage regulator were malfunctioning. Similar failures were suspected now. Luckily, these issues were fixable the last time, and we had high hopes for the current case.

LB6RH and LA3WUA went on to reproduce the final debugging steps taken the last time.

LA3WUA prepares the multimeter.

The USRP was probed at the 3V3 receive rail while the USRP was put into receive mode.

Joint collaborative multimeter probing.

Ultimately, this revealed that the 3V3 was sagging yet again. LA3WUA went on to attempt replacement of the LNA.

LA3WUA at the soldering station, doing battle with a daughterboard.

This did not immediately fix the problems. LA3WUA has some Ideas, however, and we’ll get back to a proper fix in a later blog post. In the meantime, the SatNOGS station can be considered QRT until we fix the SDR or find a sacrificable replacement.