Preparing for the next formal release of Thetis

[w-u-2-o]

It's totally up to you, of course, but after uncountable database transitions during the last ten months of continuous development, I'm preparing to rebuild my database from bare metal when the next formal release of Thetis comes out. The database is so complex that there's significant risk of skeletons, ghosts and all manner of weird stuff that might be lurking in there. Indeed, my band stacks no longer work correctly and haven't for some time now.

If you are also interested in doing that, here is my checklist:

1. Make screen shots of all the setup tabs where you've made changes. For me that was 38 out of 49 tabs. I use Microsoft Snipping Tool and stored them, six to a page, in a Word document.

2. Export your MIDI configurations: Setup > CAT Control > CAT > Configure MIDI > Manage Mappings > Export Mappings.

3. Export your transmit profiles. In Setup > Transmit, select each profile you want to save in turn and the use the Export Current Profile button.

4. Make note somehow of any changes you've made to the filter presets (if you've ever right-clicked on a filter preset). There's no way to export these, they have to be redone manually.

5. I don't use memories, but you can save the DXMemory.xml and memory.xml files and replace the new ones with the old ones. They are found in the settings folder along with the normal database files: C:\Users\your-windows-username\AppData\Roaming\OpenHPSDR.

Unfortunately there is no way to save bandstacks without saving the entire database, which is what I'm trying to avoid. The good news for me is that I have very simple bandstacks and it won't take me long to rebuild them. I'll just do it every time I work a new band and let them build back up naturally.

It's quite a bit of work to do this, and then to manually redo most of your settings except for MIDI and transmit profiles, but IMHO it's worth it every once in a while, and when the new formal release comes out that's a natural time to do so.

[ramdor]

So to try and document this somewhat, and remove some of the confusion regarding updating the database I figured I might as well cover this. Chris W2PA performed this extensive work to simplify the importing of old databases which may/may not be partially corrupt, to bring them into a new database structure. So let us look at what happens....

When a new version is released you will get the message 'Your database file is from a different version'... etc. The version number change makes this happen. As you know, I have not changed the version number during my modifications so I needed another way to forces this, hence holding CTRL at start up. So, what happens next?

1) it makes a copy of your existing database.xml file in DB_Archive folder
2) makes a copy of your existing database.xml to a new file called databaseToMerge.xml in the same folder as database.xml
3) deletes database.xml
4) After a prompt explaining what is happening, Thetis then fully initialises and shuts down, causing defaults to be written to a fresh new database.xml on exit
5) You then restart, and the databaseToMerge.xml file is detected, and merge is started

What does the merge do ?

1) basic validation on the databaseToMerge.xml to make sure it is not corrupt/blank/etc
2) any table that exist in databaseToMerge.xml but not in database.xml is merged into database.xml (BandStack2Form, XVTR, SpotForm, DiversityForm, EQForm, MemoryForm, WaveOptions, AmpView, PureSignal, WideBand)
3) the data from the following tables is now merged into the new database.xml from the old databaseToMerge.xml
[bandstack2, old bandstack data if available, bandtext, grouplists, memory, TXprofile, state (main UI state/buttons), options (all of setup form), eqform, memoryform, diversityform, ampview, puresignal, notch data]
4) if merge success then...
5) deletes the databaseToMerge.xml
6) updates database.xml with the new merged settings
7) displays the msg box 'Your db from a different version was imported successfully into a new one.'
8) if merge failure then...
9) Inform user of failure, and continue to use fresh database.xml

Note: using setup->import db, will attempt to run through everything after 'What does the merge do?', but using the file supplied files instead of databaseToMerge.xml. The file in this setup->import can be a partial xml, for example just containing the TX profile table.

In short, you should not really need to do anything other than let Thetis do its thing when a new version starts up. It refreshes the database as part of the process and unless you have some settings out of whack (or somehow duplicated perhaps by erroneous direct xml editing) then it should all be ok. It would be worth testing this in the short term so we make sure that it is covering everything before Doug releases.

Richie.

[ramdor]

Indeed, my band stacks no longer work correctly and haven't for some time now.

please send me your database.xml

Richie.

[w-u-2-o]

Sent as requested

[JJ4SDR]

Hi,
I have never installed any software upgrades to my ANAN 7000 DLE MKII and am still using P1 (basic version of Thetis 2.8.11of 10/20/20 that was pre-installed on the radio when I received it in August 2021) nor any patches. TX profiles abound on my radio (I have this many due to 2 different microphones and at least 3 different bandwidths for each). I only use CFC/LEVELER inside the ANAN and have VST plug-ins mold my sound inside a DAW for the rest. I haven't modified anything else on the different forms except calibrating the watt meter and PA Gain on the "PA settings" form and did do the frequency calibration. Some slight tailoring of the colors of the GUI, that's it, but if I lose that isn't a big deal.
Scott suggested a way to do it and perhaps I go about it the same way as I haven't really messed with the myriad parameters all that much.
Questions:
1) I have a stable and well behaving P1 operating environment right now. Which folder(s) do I need to make a back-up of exactly to enable me to revert to my currently working system should my upgrade to 2.9.0 fail?
2) Should I take pictures of the the many CFC settings form to have a fall-back?
3) If I understand from what Scott was writing, is that the easiest way to start from scratch, like I just bought the radio and want to get and up running on the latest "official" P2 version of Thetis?
Apologies if I am also not quite using the terminology and nomenclature that I should ! That just goes to show how much I have yet to learn.
Awaiting some down-to-earth suggestions that even I can understand !
Regards,
JJ

[JJ4SDR]

Richie,
I think what you were writing partially answered my questions and addressed my concerns, but I am still a newbie in all of this and just want to make sure that I will absolutely have a "safety" if all fails.
JJ

[w-u-2-o]

1) I have a stable and well behaving P1 operating environment right now. Which folder(s) do I need to make a back-up of exactly to enable me to revert to my currently working system should my upgrade to 2.9.0 fail?

Assuming the 64 bit version, make copies of:

- C:\Program Files\OpenHPSDR\Thetis
- C:\Users\<your windows username goes here>\AppData\Roaming\OpenHPSDR\Thetis-x64

If everything goes to hell you can just put those back where they were with their original names and all will be as if nothing ever happened.

2) Should I take pictures of the the many CFC settings form to have a fall-back?

Yes. Do this for all your setup screens just as I recommended above.

3) If I understand from what Scott was writing, is that the easiest way to start from scratch, like I just bought the radio and want to get and up running on the latest "official" P2 version of Thetis?

Not necessarily. It's probably easier to uninstall the old version and install the new version and trust that the database update will be perfect. But it is very important to be properly prepared in case something goes wrong with the update.

[ramdor]

Yes. Do this for all your setup screens just as I recommended above.

If this is needed, then the software needs changing so you don't have to do this.

We could do wit a definitive list of things that don't get re-imported, so they can be included. I have not spent any time looking at this tbh.

[W2PA]

If this is needed, then the software needs changing so you don't have to do this.

We could do wit a definitive list of things that don't get re-imported, so they can be included. I have not spent any time looking at this tbh.

I agree. The merge process you outlined above after "What does the merge do ?" should account for everything. If anyone discovers something not being imported, please report it - that kind of thing should be easily remedied.

[K9RX]

Chris: As mentioned by me in a WANTWISH a while back and by Scott at the top of this thread - what about the filter settings? I edit all of my filters for USB/LSB/CW which involves considerable entry work (each one of the 10 filters requires 3 entries)... I apologize if I missed it in Richie's list but I didn't see it there (or recognize it if it is called something different)... will this be merged as well and if not can it be, please?

Gary
K9RX

[w-u-2-o]

Yes. Do this for all your setup screens just as I recommended above.

If this is needed, then the software needs changing so you don't have to do this.

We could do wit a definitive list of things that don't get re-imported, so they can be included. I have not spent any time looking at this tbh.

While there are definitely things that don't get imported, like the filter settings, my concerns and my caution are not associated with that issue.

Historically the Thetis (and PowerSDR before it) database structure has proven to be very fragile. As you know there are all kinds of duplicate paths through the code. I may be wrong, but I doubt anyone could 100% predict all possible actions on the database.

While database operations, particularly those associated with version upgrades/updates, have become substantially more reliable with all of the various bugfixes that have gone in over the past year or so, my fear factor remains high. And not without justification. You remember my problem with the band stacks that you could not replicate? 100% completely fixed when I rebuilt the database from a default state using my document full of screen shots and my separately saved TX profiles.

So things are much better, yes, absolutely. But like Ronnie Reagan said, "Trust, but verify." With 40 something setup pages, if one were faced with a hopelessly corrupted database (like I was), there's no way to remember every single, little change that I did over the years. But with 40 screenshots I can do that if I have to. And it's even more than 40 now with the new PA profiles. In fact, like TX profiles, it would be good if a PA profile could be saved out separately. That'll save a LOT of typing from screen shots if a database becomes hopelessly corrupt for some, admittedly increasingly rare, reason.

73!

Scott

[W2PA]

I may be wrong, but I doubt anyone could 100% predict all possible actions on the database.

You're not wrong, Scott. The DB handling code (import and export) is fairly robust against broken records of all sorts. But it won't find and correct more complicated things like incompatible settings that somehow got set.

If I recall the code correctly (and it's been a while) there is an in-memory DB that is active while Thetis runs, and a storage (i.e. disk) based DB which gets written to by Thetis based on its in-mem DB, and gets read by Thetis at startup to initialize the in-mem DB. There are various parts of the code that have access to both.

Correcting for unsaved items in Setup should be straightforward. Consistency or compatibility of settings before storing is another more challenging thing (but should be looked into as well).

[K9RX]

notwithstanding your concerns and cautions Scott, just to 'ask again' (as it might have been missed by Chris or whoever might work on it) ... the filters are indeed something that I'd hope are straightforward to IMPORT/EXPORT.

I had suggested that there be, on a right click of a filter for a mode, the options: 'configure' (already there), "Use for VFOB" (when active, grayed out if MULTI not selected), EXPORT ALL, IMPORT ALL, Reset to default (already there) .... and they do take a lot of work to rebuild as mentioned. I also took pictures in the past and will continue to do so - like now - when there are cautions about the DB.

Now that all stated, I do appreciate the gravity of your concerns re the DB and would agree empathetically to others to do the same - take screen shots.

And I'll add as a very strong THANK YOU to Richie and Doug and Chris and Warren (and any others involved) on the work that has been done. Previously when using DIG modes on 3.4.9 I'd get these burbles in TX - I'd have to reboot the PC and most of the time it would clean them up for a while - days - but would then need to be redone. Thetis JUST WORKS in FT8 and Q65 (EME on 6M). Incredibly stable in that regard. Congratulations.

Gary
K9RX