Akademisk Radioklubb


Month: January 2018 (page 1 of 3)

Plotting Norwegian ham radio contacts on a map using Pandas, Cartopy and Geopy

In the post about the Norwegian telephony contest, we plotted the contacts on a map of Norway. Plotting contacts during short time periods like this can give a good visualization on the propagation conditions. In this particular case, it could yield some pointers on how we might do better the next time we run a similar contest. Due to sparse settlement patterns in Norway, however, this mostly demonstrates that Norway is a rather empty country. Regardless, they are nice illustrations :-).

Here, we outline how we did this using Pandas in Python. The more specific details apply only to Norwegian ham radio contacts, as we have used Norway’s database over the address information for Norwegian licensees. It could be applicable to other countries as well, given similar databases and some small tweaking to the scripts.

We will go through it step by step. First, we import all necessary libraries. Pandas for convenient data wrangling:

import pandas as pd

The Nomantim submodule of Geopy for conversion of addresses to coordinates using OpenStreetMap:

from geopy.geocoders import Nominatim

Matplotlib and cartopy for plotting of the data points on a nice map:

import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy.io.shapereader as shpreader

We could also have used Basemap, but in my short experience, cartopy seems to produce maps that have a lower threshold for looking nice by accident, with a bit more convenient API. This is probably extremely subjective :-).

First, we use the N1MM database reader introduced in a previous post to read the logs into a Pandas dataframe. Since we don’t need all the fields, we remove everything except for the callsign, timestamp and band fields.

qsos = analyze_logs.get_n1mm_logs_in_path('path_to_logs/')[['Call', 'TS', 'Band']]

We then download the official Norwegian ham database, which exists as an Excel sheet and a CSV file. This can conveniently be read using Pandas.read_excel(...), but in order to automatize the joining of this information and the QSO log, we’ll also rename the Norwegian ‘Kallesignal’-column to have the same name as the corresponding column in the QSO log, ‘Call’.

lahamsin = pd.read_excel('Liste over norske radioamatører (Excel) 03.10.2017.xlsx').rename(columns={'Kallesignal': 'Call'})

From geopy, we can get a fine granularity of the coordinates down to address level, but we’ll limit ourselves to the postal code in order to heighten the probability of obtaining an actual coordinate match. The address field can have some weird stuff like a post box number. We add the postal code information to the QSO table by running

qsos = pd.merge(qsos, lahamsin[['Call', 'Postnr']], on='Call')

Each contact in the log will now have its corresponding postal code. For conversion to coordinates, we’ll be using geopy.Nominatim(...), which can look up coordinates from addresses using OpenStreetmap. Geopy also has support for various other map APIs, but these typically require API keys or user authentication. We bias the look-up to Norway in order to probably play a bit more nice with the server.

geocoder = Nominatim(format_string='%s, Norway', country_bias='Norway')

Next, we obtain all unique postal numbers from the QSO table in order to limit the number of look-ups we have to do.

addresses = qsos[['Postnr']].drop_duplicates()

We then look up each postal number against the OpenStreetmap API using Pandas.DataFrame.apply(), which can apply a function row by row to the dataframe. In this case,
geocoder.geocode(...) is used to look up coordinates from the postal code in each row, assuming Norway as the country.

addresses["geocoder_res"] = addresses.apply(lambda x: geocoder.geocode({'postalcode': x.Postnr, 'country': 'Norway'}), axis=1)

Latitudes and longitudes are extracted in a similar way.

addresses['longitude'] = addresses.apply(lambda x: x.geocoder_res.longitude if x.geocoder_res is not None else None, axis=1)
addresses['latitude'] = addresses.apply(lambda x: x.geocoder_res.latitude if x.geocoder_res is not None else None, axis=1)

Merging back the addresses by joining on the postal code, we finally get a complete QSO table including the longitude/latitude coordinates of each contact.

qsos = pd.merge(qsos, addresses.drop('geocoder_res', axis=1), on='Postnr')
qsos = addresses.drop(['Postnr'], axis=1)

Now that we have wrangled all the data, we can finally turn to plotting :-). It can probably be a good idea to save the data to a file in order to avoid having to look up the coordinates against OpenStreetMap every time the script is run.

Cartopy has in-built support for automatically downloading Natural Earth shapefiles when a specific shapefile is requested (and then avoid having to download it again by caching the file). The shapefiles contains various information like country borders, roads, railroads, …, and can be used to plot the borders of a specific country like e.g. Norway down to various resolutions.

We use the admin_0_countries shapefile from the cultural database at 10 meters resolution.

shapefilename = shpreader.natural_earth(resolution='10m',
category='cultural', name='admin_0_countries')

I have not yet found a nice way to find the geometry shape corresponding to Norway except for looping through all records and stopping at “Norway”. There are probably better ways than this :-).

reader = shpreader.Reader(shapefilename)
countries = list(reader.records())
for country in countries:
   if country.attributes['NAME_LONG'] == 'Norway':
      norway = country.geometry

Now that we have the country borders of Norway, we can finally set the borders of Norway and prepare a country map.

projection = ccrs.Mercator()
ax = plt.axes(projection=projection)
ax.add_geometries([norway], projection, facecolor=(0.8, 0.8, 0.8))

Plotting all contacts made on the 80 meters band can then be done using e.g.

band_80m = qsos[qsos.Band == 3.5]
ax.plot(band_80m.longitude.dropna().as_matrix(), band_80m.latitude.dropna().as_matrix(), 'o')

This image is from one of the periods during the Norwegian ham radio contest “Telefonitesten”, and contains the 40m contacts and 80m contacts with different marker colors and sizes.

This should serve as a nice example on how data from three different sources can be combined using Pandas.

LA2SHF Beacon completed and temporarily active from JP53EJ

For a long time we have wanted to have a beacon for propagation studies and prototyping/debugging in the 23 cm band. Due to the recent activity on the 1 to 10 GHz project, the need for a beacon has resurged. After a period of intense work, we are excited to announce that we have gotten the LA2SHF beacon active in test operation!

LA2SHF operates on 1296.963 MHz with a A1A CW sequence consisting of “LA2SHF JP53EJ” followed by a 10 second carrier. The output power is 30 W onto a sleeve dipole. LA2SHF is currently situated at LA1BFAs house in JP53EJ, but we will move the beacon to our beacon-site at Vassfjellet once the snow thaws (expect this is May/June). The current QTH is quite low altitude, so we expect that it might be hard to hear the beacon during test operation. If you do hear the beacon we would love to hear a report on la2shf@la1k.no, or on your favorite DX cluster.

JP53EJ region (red square) on a map overlay of the southern part of Trøndelag.

ARK has held the LA2SHF license since 1979, and several attempts at getting the beacon running have been made since then. Most notable is a working beacon in the 1980s that had to be taken down  due to interference to a air traffic control radar at Gråkallen.

The 1980s edition of LA2SHF.

The new beacon builds upon the LA1K CW beacon PCB and firmware created by LA3JPA (with help from LA1BFA and LA7VRA) in 2012. The project is available open source on github. In addition to the CW beacon, amplifiers are needed in order to bring the output power up from approximately 15 dBm to 45 dBm (30 W). Bert Modderman, PE1RKI, provided us with a great custom tuned driver/power amplifier pair that allows us to do just this. We also purchased an interdigital bandpass filter and a circulator from him.

Block diagram of the beacon.

The purpose of the bandpass filter is to reject harmonics and nearby transmitters. Previously we have had an issue with one of our beacons, where nearby transmitters would modulate the exciter, causing unintentional spurious behavior. The bandpass filter should make sure that this does not happen.

A circulator adds additional protection to the power amplifier. If the antenna is disconnected or damaged, the reflected power from the antenna will dissipate in the termination load of the circulator, ensuring that no harm is done to the power amplifier.

The build was started by assembling the CW beacon PCB. Parts were ordered from Digikey using the supplied Bill of Materials (BoM). PCBs were ordered from DirtyPCBs using the gerber files (PCB production files) supplied on Github.

The PCBs are programmed using the MPLAB-X IDE for Linux (Windows and Mac versions also exist). In order to adapt to the previously developed code, some minor modifications are needed.  Changing the frequency and CW sequence was done by altering line #56 and #57 in main.c. Additionally all references to plib.h were replaced with xc.h.

A thermal imaging camera allowed us to check the thermals of the CW beacon PCB. As expected the voltage regulators and RF ICs are the parts that get hot. All seems to be in order, but the temperature got all the way to 59 degrees after a couple of minutes. As a security measure, a fan is placed next to the PCB inside the final chassis.

Once the parts from PE1RKI arrived, we were ready to start the build. After unboxing we eagerly connected the filter and circulator to our network analyzer to check that everything was in order.

The most time-consuming stage was integrating all the components inside a chassis. At Vassfjellet, the other beacons are mounted in a 19″ rack, so we needed something that would fit in there. Luckily we had a surplus computer cabinet that was up to the task. Most of the work here consisted of drilling and tapping holes, and hooking up wires to all the parts.

Once the internals were completed it was time to ensure that the beacon would work as intended on the air. First order of business is a test of the amplifiers. To conduct this test a signal generator is used as the exciter for the amplifiers. The output level of the generator was carefully stepped up until the desired amplifier output power was achieved. The corresponding input drive was then measured with a spectrum analyzer.

The output power of the CW beacon PCB is adjustable through the ADL5330 Variable Gain Attenuator (VGA) circuit. By changing line #58 and #64 in main.c this power may be adjusted. We connected the beacon output to a spectrum analyzer and started tuning the power until it was at the level obtained in the previous paragraph. Once these levels were found it was time to investigate that no splatter occured after amplification.

With a relatively clean output signal, and a successful test reception between LA1K at JP53EK and LA1BFAs site at JP53EJ we now have a working beacon!

LA2SHF as received from LA1Ks 3m dish.

It has been very exciting to work on this soon to be 40 year old project. We are on the finish line, all that remains now is to mount it at Vassfjellet. It will be exciting to see how it performs in test operation, and if there are any bugs that need to be ironed out. As a stretch goal we want to make the beacon available on the PI4 digital mode, and are working on the code to do so. For now CW is the main operational mode.

If you are able to hear the beacon, please send us a signal report at la2shf@la1k.no.

Older posts