CQ Monitor Data Source

28 posts / 0 new
Last post
hi8smx
CQ Monitor Data Source

Hi, is there a way that I can program where the CQ Monitor goes to get data for stations?, I've noticed that CQ Monitor is displaying certain stations as located in a particular state in the US and when the qso is made I see that they are in a different location. As soon as I begin the qso, in the New Qso Window the correct information is displayed and often differs from what is shown in Cq Monitor.

Can I direct CqMonitor to look into QRZ.com for example?

73

Santiago
HI8SMX

oh1kh
CQ Monitor Data Source

Hi!

CQ-monitor uses same ctyfiles as cqrlog itself. So you see the same problem with manual enter of New Qso.
us_states.tab in ~/.config/carlog/ctyfiles holds the information, but it fails as US stations need not to change their call any more when they move.
Same here in Finland. Previously country was clearly divided to counties OH0...OH9 and you had to ask for new call from telecommunication author if you moved. Not any more.

That makes precise county definition from callsign prefix impossible.
Only way, in question of US states, would be to make database holding all calls and counties from FCC databases and update it on regular basis.

--
Saku
OH1KH

hi8smx
Thanks

Thanks Saku for the clarification. I now understand. Really appreciate it.

73
Hi8SMX, Santiago

ok1rr
ok1rr's picture
Prefix alocation in OH

Saku,

if I understand correctly, now a station from Helsinki can use OH9 prefix. Hmmm...
What about OH0? Must be they on Aland? And Market Reef? Still OJ0?
Can you tell me the date when your authorities gave up the prefix alocation?
It would help me to keep the country files fresh and useful.

Many thanks, 73
Martin, OK1RR

oh1kh
Prefix alocation in OH

Hi Martin.

I'm not sure how long this has been on. But quite long. One source says that selling callsigns started in mid of 1990's ( http://oh2mp.ham.fi/kutsut.html , in Finnish :use google translate)
The provinces were abolished on 1 January 2010 and I think that at least that time also prefixes by provinces finally lost their original status.
How ever authorities still try to give a new call following the old province formula. How ever yo can buy a callsign that is not in use.

I could buy OH9SAKU and use it here (in old province formula of OH1).

So you can not specify area by prefix any more for sure.

Except OJ0, and I think OH0 is also quite sure (but I do not know if it is just a gentlemen agreement).
Mainland prefixes may be mixed as in U.S.A.

--
Saku
OH1KH

oh1kh
CQ Monitor Data Source and US counties

Hi!

Still thinking about this US county finding.
If we forget callsign prefix as unreliable county recognize source, what else we have?

At least with wsjt modes it could be easy to use locator grid as approximate county indicator.
With 4 char grid it is not very accurate but gives good direction.
With 6 char, or more, the it would be great.

If just someone would do a list of counties with grids they have. At level of 6 char would be very usable even when we could search from it with only 4char grids in most cases.
And this list would be rather stable, I think so :-)
No need to upgrade very often...

I tried to make few quick google search with this subject, but did not find any ready made list:
So if someone is wiling to do it, or knows a ready made list or database somewhere, it could maybe be used as US county recognize source for cqrlog.

--
Saku
OH1KH

oh1kh
CQ Monitor Data Source and US counties

Or should we use word states instead of counties when US is in question?

--
Saku
OH1KH

ok1rr
ok1rr's picture
Re: CQ Monitor Data Source and US counties

The only reliable data source is the FCC Database. FCC made it available to the public (they have no stupid Data Protection measures like here in EU). Simply hit F11 (in Preferences MUST be the Callbook support pointed to HamQTH.com) and the State and County columns are automatically filled with the data from the FCC Database.

The State and County can be determined also from the ZIP code, CQRlog has a VERY RELIABLE algorithm for ZIP code extraction from the address and a database allowing county determination by finding the ZIP there. The ONLY problem is the reliable callbook - but, again, the FCC Database can be used. Actually, no need to use the ZIP_code/County database (although this option still exists and works) because the direct County/State reading from the FCC Database works.

NO NEED OF EXTRA PROGRAMS - CQRLOG HAS IT ALL!

oh1kh
Re: CQ Monitor Data Source and US counties

HI!
We have discussed this before.
US counties, or states if you wish to use that word, are not reliable as in US (as in Finland) callsign change is not any more required when you move. So states by perfixes do not work.

Routine that is in cqrlog and used in CQ-monitor as well as in NewQSO gives very often wrong state.
When callsign data fetch from HamQTH or QRZ you find out that state is different than cqrlog offers.

So CQmonitor gives (may give) different state when just monitoring. If call is selected for qso and transferred to newQSO callsign field and HamQTH/QRZ fetch is done you notice that it is not the state you are looking for!

In WSJT modes the locator (4chr) is sent with CQ and if we just could have database that list locators by states (or states by locators) we get the real state (or near by with 4chr) already at CQmonitoring phase.

And if station calling CQ is not in his normal qth (portable or something) the locator send with CQ is the only way to try to dig out what state he is in. FCC databases or HamQTH/QRZ does not right work then.

This works only with wsjt modes as "preview of state", not with other modes as locator is normally not included into CQ.

That is what I did to explain.

--
Saku
OH1KH

KB6IBB
Re: CQ Monitor Data Source and US counties

I have to make mention of some experiments that I have done. I too suffer from the same issue of the States not being displayed correctly. On 6, 10, and 12 meters this is more than annoying. I have had to shutdown CQRLog operations and push the UDP traffic over to a Mac for logging.

I am using SuSE Leap 15.1 and all of the latest versions. Using samplicator, I am able to send the UDP packets to multiple IP's. This allows me to make side by side comparisons between CQRLog's monitor, JT-Bridge on the Mac, and JTAlert on Windows. The source of my frustration is that JT-Bridge and JTAlert are getting the States correct with every single decode. Whereas CQRLog for example will show either Alabama or Arkansas for any 5 call. My neighbor who I defiantly know is in Texas with a 5 call shows up in Alabama with CQRLog. He shows up correctly in Texas with JT-Bridge and JTAlert.

The conceptual solution that I have come up with should work. I just don't have the time on hand to sit down and write the code on a unfamiliar code set. Yes, we would want to use the grid locator sent during the CQ for all callsigns world wide to achieve the highest degree of accuracy. The only time this would be ineffective is with a non-standard callsign where a grid locator may not be sent (IAW Chapter 7.5 WSJT-X User Guide). Non-Standard callsigns are usually special events and operators would be looking those up manually anyway.

My CQ looks like this from my home in Texas:
CQ KB6IBB EM13

However, I also have a home in Honolulu Hawaii, so my callsign can also look like this:
CQ KB6IBB BL01

If the operator chooses to use callbook data, the grid will need to be overwritten with the actual grid that was sent if they don't match. This needs to happen no matter what. CQRLog will look up the callsign in the database (HamQTH or QRZ) and fill in the field data. Then, a comparison needs to be done before the record is written to the database. If qrz grid <> grid sent then overwrite the grid in the log entry with what was sent. Save entry. Update the QSO List.

If the operator chooses to not use a online callsign database, the the log entry gets written with what WSJT-X sent to the log. Which would be callsign, grid, signal reports.

This gives you both an accurate location in the CQ Monitor for world wide CQ's, and an accurate log entry for the contact. Easy to tell if the operator was at home or portable. In fact, you could add some artificial intelligence by appending a "/P" to the callsign indicating portable/mobile for the contact. Which then gives CQRLog additioninal search criteria and better chances of earning awards.

This bug is of such ultra high importance CQRLog is virtually unusable for work with WSJT-X until fixed. I would be more than happy to beta test for you in what spare time I do have.

Thanks for reading
DE KB6IBB BL01

oh1kh
Re: CQ Monitor Data Source and US counties

Hi!

If I understood ok there are 2 things you are complaining.

1st: States are not correct in CQ monitor (that is discussed before, but they will be ok at logging phase if user has valid HamQTH/Qrz record and internet is in use)

2nd: Locator is not correct if callsign has more qth's than one (that he has set to HamQTH/Qrz data)

For the 1st one we should need valid FCC database based record for all US callsigns. And it must be updated at least once month.
Possible to fix, perhaps if someone does the coding.

2nd:
"If the operator chooses to use callbook data, the grid will need to be overwritten with the actual grid that was sent if they don't match. This needs to happen no matter what. CQRLog will look up the callsign in the database (HamQTH or QRZ) and fill in the field data. Then, a comparison needs to be done before the record is written to the database. If qrz grid <> grid sent then overwrite the grid in the log entry with what was sent. Save entry. Update the QSO List."

That is just how wsjt remote works.
When you have worked a wsjt qso you MUST log it by pressing OK at wsjt-x log window. You MAY NOT use cqrlog's "save qso " button.
Using OK at wsjt-x log window causes wsjt-x produce UDP message #5. That is decoded in cqrlog and that includes:

loc:= trim(StFBuf(index));
if dmData.DebugLevel>=1 then Writeln('Grid :', loc);
if dmUtils.IsLocOK(loc) then
if pos(loc,edtGrid.Text)=0 then //if qso loc does not fit to QRZ loc , or qrz loc is empty
edtGrid.Text := loc; //replace qrz loc, otherwise keep it

That means the locator shown during qso at NewQSO window is replaced with locator that is received from wsjt-x UDP frame if they do not match. As wsjt-x sends 4chr locator then if those 4 chrs are same locator from qrz then qrz locator is accepted.

This may lead to wrong locator only if user has several qths in the same 4chr grid area: KP01tn, KP01rq and that is a "bug", but not very bad.

In other cases locator from wstj-x is selected and should be the right one. Only cases with special callsign, when locator is not sent with qso data may lead to wrong locator (if it is set wrong in qrz/Hamqth data) .

NOTE that the locator you see from NewQSO window during qso MAY NOT be the one that is seen from wsjt-x CQ monitor.
Only check from "Qso list" after qso will tell what locator was finally logged.

--
Saku
OH1KH

oh1kh
Re: CQ Monitor Data Source and US counties

Hi!
Just tested locator to make sure it still works as should. It was done like this:
1) I made a audio file with ft8sim "CQ OH1KH AB12" (ft8sim comes with wsjt-x source and can produce test wav files for given text)
2) started cqrlog and set wsjt remote on
3) started wsjt-x
4) from wsjt-x File/open I choosed generated wav file that produced a decode with my call and AB12 locator
5)At report giving phase of qso cqrlog fetched my data from HamQTH with my true locator
6) clicked wsjt-x main window's "log" button. Locator AB12 was visible there.
7) clicked wsjt-x log window's OK button. (This is told in cqrlog/help/operation/digital modes:wsjt-x/Logging qso to cqrlog). Qso was logged.
8) opened "qso list" and qso is there with AB12 locator like it should.

Picture attached is manipulated. Qso list part is dropped over screen copy afterwards to get it visible at same time while newQSO and wsjt-x log window still holds qso data (before pressing OK button).

File: 

--
Saku
OH1KH

KB6IBB
Re: CQ Monitor Data Source and US counties

Here are the results of the test. N5ITO was calling CQ on 6 meters. The call is: CQ N5ITO EM23

The CQ Monitor placed him in Arkansas instead of Mt. Vernon, Texas. I confirmed that EM23 is indeed Mt. Vernon, Texas. Confirmed he did send the correct grid locator. Looked him up manually on QRZ and all checked correctly. Also reviewed data from JT-Bridge on the Mac and it was also correctly showing him in Texas. CQRLog internet files are all up to date.

File: 

KB6IBB
Re: CQ Monitor Data Source and US counties

Here is a comparison between monitor programs.

File: 

KB6IBB
Re: CQ Monitor Data Source and US counties

Here is a comparison between monitor programs.

File: 

oh1kh
Re: CQ Monitor Data Source and US counties

Yep!

That is just the problem #1.
Cqrlog gets state from ctyfiles that are DXCC updates that you receive time to time. It is not just CQ-monitor's problem.
If you enter manually a call to newQSO, lets say N5ITO (and you disable qrz/HamQTh from preferences) you will see "USA - AR,LA,MS,NM,OK,TX" at DXCC info. It fixes to TX a second later only if you have internet and enabled qrz/HamQTH search.

That is where also CQ-monitor gets it's information. From local DXCC file.
As space for country text in CQ monitor is limited you will only see USA - AR. If there would be more space would it be any better to see all: AR,LA,MS,NM,OK,TX on the line? (It could be quite easily fixed) But it is still a guessing game.

CQ-monitor does not use qrz/HamQTH for fetching information to ALL HEARD stations. It has no mind.
You may have slow internet, or no internet at all. It still works with this way.
If you get 10 CQ lines for one 15sec period it needs 40 info fetches per minute. Then if there are 1000 users with same program it makes 40000 fetches per minute for server. Lot of unnecessary traffic.

That is why the qrz/HamQTH info is fetch at report giving phase of real qso, just for that worked call.

It would be interesting to know how JT-Bridge does that. Does it have local database for all calls and states or does it rely only on internet search? Does it work at all if you switch off internet? (If it still works it has a local database for all US calls)

--
Saku
OH1KH

ok2cqr
ok2cqr's picture
Re: CQ Monitor Data Source and US counties

Hi Saku,

as far as I know, JT-Alert uses HamQTH to fetch correct data. There are many request every second but HamQTH can handle it without any problem. Looking for state detail could be possible but there should be some kind of cache of already fetched information. That could work nice, too.

73 Petr

oh1kh
Re: CQ Monitor Data Source and US counties

Hi!

Ok.
My original idea was not to stress qrz or HamQTH too much.
As you know the popularity of FT8 and if every decode are fetched that means lot of traffic. How ever short one request may be.
Usually server farm companies charge by transferred Gigabytes per month. So the problem is service keeper's, not individual user's.

And second notice is that there is only couple of seconds time between periods to do the decoding, search qso database for Wkd b4f info, and in case of fetching all CQ decodes from qrz/HamQTH to do those web requests.
It needs fast pc and fast internet..

And it does not work in places without internet (well, mobile net is almost everywhere but may have long delays depending of how many users share base station's channel)

So some kind of cache would be good. I have understood that there is also possible to fetch US callsign information directly from FCC database.
If that is true it would be rather easy to fetch data and create states data file to ctyfiles and load it to mysql like other dxcc information.
Then it would be there locally and we could continue fetching just worked qso's data from qrz/HamQTH.

We just need the US state data per callsign, so the database size would not be problem. And I think monthly update for data would be enough.

--
Saku
OH1KH

oh1kh
Re: CQ Monitor Data Source and US counties

I was looking at https://www.fcc.gov/uls/transactions/daily-weekly#fcc-uls-transaction-fi... but it seems not to have state info we need (or I have to understand how to read the file)

Then just came into my mind:

You have the HamQTH data. You could easily generate UScall/USstate file and deliver it with DXCC updates to cqrlog local database.
It is not complete, but has quite lot of that info.

--
Saku
OH1KH

KB6IBB
Re: CQ Monitor Data Source and US counties

Yes, exactly. I don't know anything about JT Alert as I have not used MS Windows in over 17 years. I do know about JT-Bridge on MacOS and of course CQRLog on Linux. JT-Bridge does download 4 files at start up which are used for looking things up quickly.

You do have to adjust thinking in terms of callsigns and databases. Those are irrelevant. It's only looking up the location of the grid square the station sends. So.... CQ KB6IBB BL01 shows me in Hawaii,or, CQ KB6IBB EM13 shows me in Texas. All other information for the CQ Monitor is irrelevant. Remember that callsigns move around, so I can be in three different grid squares in one afternoon operating portable on 6 meters. Callbook data isn't really all that important, other than to know where to send a QSL.

You also want to take into consideration the FT4 mode, as FT8 is slow and obsolete with FT4's introduction last month. It's now a 6 second turn around time. In terms of looking up a group of callsigns, 6 seconds is more than enough time for either a SQL lookup or even a sequential text file look up.

Either way, the CQ Monitor is not usable until it can accurately reflect the location of the station.

oh1mrr
Just a notice, FT4 is not a

Just a notice, FT4 is not a MODE, it is SUBMODE for MFsk, so calculations
for FT4 is not relevant. FT4, what I've read, is meant FOR contesting. Suppose
not for daily use. At the moment there is lots of FT4 contesting going on every
day :).
How about forgetting STATE from CQMONITOR? Just show locator....We can
fetch state from QRZ or HAMQTH or where ever..

<p>Jarmo</p>

oh1kh
Re: CQ Monitor Data Source and US counties

Hi!
I made CQ monitor just for my own needs in the beginning. Later shared it also to others.
Main goal was to see have I had qso before with call who calls CQ. Is it a new country, band country or mode country for me. Colors will tell that.
Also locator has interest as I'm collecting main squares, now later extended also to sub squares. Colors will again tell me have I worked that one or not.

Sometimes detecting a country from callsign can be problem. I can not remember all prefixes. As cqrlog had a subroutine for that I decided to take also that to monitor. Unfortunately at US case it tries to offer also state. That is one that I'm not collecting. At one time I almost started, but then noticed that this does not work any more as US prefixes do not follow states any more. Same also here with OH prefixes. They used to show county, but not any more.
The easiest way to get rid of this problem is to cut country information (drop out states part) if string "USA" is detected.

FT4 works just same way as FT8, there is no difference in UDP messaging, so all features will work same way as with FT8. Turnaround is enough for SQL or text file seek as you say, but if we start to seek information from qrz/HamQTH individually for every CQ line decoded the time is quite limited.
Ok, we could think that there is time until next period ends, but it is not so. From 6 seconds we have to leave about 2 seconds for operator to decide what he wants to work from CQ monitor list before next period begins.
So we have 4 seconds left. That is maybe enough if we have fast enough internet and pc and there are not so many lines to decode. But normally there are. And most of them are not CQ lines. CQ lines do not drop at the beginning of 4 second gap, they drop in decoding order and might be that most CQs come as last lines and so last UDP frames.

CQ monitor is very useful in several ways, but US state info is bogus (also with all other modes than digital if internet and qrz/HamQTH is not in use).

--
Saku
OH1KH

oh1kh
Re: CQ Monitor Data Source and US counties

Hi!
I made CQ monitor just for my own needs in the beginning. Later shared it also to others.
Main goal was to see have I had qso before with call who calls CQ. Is it a new country, band country or mode country for me. Colors will tell that.
Also locator has interest as I'm collecting main squares, now later extended also to sub squares. Colors will again tell me have I worked that one or not.

Sometimes detecting a country from callsign can be problem. I can not remember all prefixes. As cqrlog had a subroutine for that I decided to take also that to monitor. Unfortunately at US case it tries to offer also state. That is one that I'm not collecting. At one time I almost started, but then noticed that this does not work any more as US prefixes do not follow states any more. Same also here with OH prefixes. They used to show county, but not any more.
The easiest way to get rid of this problem is to cut country information (drop out states part) if string "USA" is detected.

FT4 works just same way as FT8, there is no difference in UDP messaging, so all features will work same way as with FT8. Turnaround is enough for SQL or text file seek as you say, but if we start to seek information from qrz/HamQTH individually for every CQ line decoded the time is quite limited.
Ok, we could think that there is time until next period ends, but it is not so. From 6 seconds we have to leave about 2 seconds for operator to decide what he wants to work from CQ monitor list before next period begins.
So we have 4 seconds left. That is maybe enough if we have fast enough internet and pc and there are not so many lines to decode. But normally there are. And most of them are not CQ lines. CQ lines do not drop at the beginning of 4 second gap, they drop in decoding order and might be that most CQs come as last lines and so last UDP frames.

CQ monitor is very useful in several ways, but US state info is bogus (also with all other modes than digital if internet and qrz/HamQTH is not in use).

--
Saku
OH1KH

oh1kh
Re: CQ Monitor Data Source and US counties

Oops....

I was fooled again!
I must be more careful with reading.

You say there is 6 seconds time for SQL /text file seek. NO. That is not true with FT4, nor FT8.

With FT4 we have just only 2,5 seconds time to find all the information for all decoded lines of last period before next period begins and we need to catch the caller we want. Not 6 seconds.
And if we count operators reaction/decision time we have something around one second to use for all decoding and information fetch.

--
Saku
OH1KH

KX4KU
Re: CQ Monitor Data Source and US counties

I know this is an old thread, came on this looking see if it had been fixed. Being a long time user of JTAlert, the author of the software has a SQLite database that is used that is built from FCC, LOTW and eQSL data that he uses to get accurate information. The DB is about 38 MB in size. It is updated pretty regularly. I don't know if you want to do it that way or not.
73,
Kevin, KX4KU

73,
Kevin
W4RKU

ok1rr
ok1rr's picture
Re: CQ Monitor Data Source and US counties

Many thanks, Kevin. Of course, the more accurate (more detailed) info source, the better. Actually only FCC database is used because it has an official support. eQSL and LoTW data can contain an user input which can be messed with some "test" entries etc.

oh1kh
Re: CQ Monitor Data Source and US counties

HI Kevin !

In my test binary I have US counties check fro wsjt-remote. https://github.com/OH1KH/cqrlog/tree/loc_testing/compiled
It downloads and processes data from FCC and picks up call=state pairs.

I has no database table, instead it uses a loaded text file, that's why I have not pull requested to to official source. How ever I have has reports that it works ok.
It should be converted to mysql table for official release.

--
Saku
OH1KH

KX4KU
Re: CQ Monitor Data Source and US counties

Thanks Saku!
I really like your logger, even more that my current windows logger. I am thinking of running it in the Windows Linux subsystem on that PC.
Yes I this that is somewhat the way JTAlert does it, He builds his own table from data extracted, probably with just state, country data.
How I noticed it was our club president who lives here Virginia with was calling CQ and he showed up as Arizona with his "7" call, so I knew something was up.
Thank you and love your program.
73,Kevin

73,
Kevin
W4RKU