Goldenseal runs OK in Mac OS 10.12 (Sierra)

We just tested Goldenseal with the public beta version of the next Mac OS (10.12 Sierra).  Everything still runs as it should.

The only problem we found is that the blue-highlighted OK buttons in dialogs are hard to read. The ‘flat’ buttons in OS 10.10 and newer have a dark background that is designed for white text, but our code still draws the text in black.

Goldenseal Pro won’t have this problem, and it is far enough along that we probably won’t fix this appearance issue in the current Goldenseal app.

HINT: if you click on the colored button and then move the mouse cursor off the button temporarily, the blue turns to white so you can read the text.

Update: Gears Shifted

After some discussion with our contractor, he has decided to stop work on Goldenseal Pro.  He did a good job on the early parts of the project (opening files, reading resources) but did not have enough Cocoa experience to get anywhere with the GUI (graphic user interface) for Mac.

This is the second contractor who stalled out after a few months. There were two others who gave up after a few weeks.  I think it is just too large a project for most programmers to handle.

The good news is that we’ve finished the prep work and programming that we needed to do.  So, we will start work on the GUI with our own staff.  We are already very familiar with the existing code, so there won’t be any “learning curve” there.

The plan is to get the basic GUI working, and then subcontract a few remaining details (printing, networking, Custom Layouts).  Those will easier for folks to bid on and complete, and they can also be done in parallel.

One wrinkle is that the Swift programming language is currently getting a complete rewrite, with a new Swift 3.0 planned for completion in late 2016.  It is not backward-compatible, so anything written in Swift 2.0 will need to have some parts rewritten.  However, it will not take much Swift code to create the GUI, so that change will not affect us by much.

Dennis Kolva
Programming Director
Turtlesoft.com

Goldenseal Pro Progress Report #9 (Aug 17)

The contractor who is working on Goldenseal Pro got “stuck”.  He wrote new GUIs (graphic user interfaces) for Windows and Mac, but has not been able to connect them with the database, or our business logic.

We are reviewing his code now, to see how to fix that.  We are also discussing ways to make things go faster.  We are eager to see it finished this year.

Most likely our staff will take on a larger portion of the development work, and we will contract smaller parts of the project.  Our experience over the past two years seems to be that many programmers think they can successfully complete a large job like this, but very few actually can do it.

Dennis Kolva
Programming Director
Turtle Creek Software

Goldenseal Pro Progress Report #8 (July 29)

We have shifted gears, and are reading up on all the stuff we’ll need to know for writing our own GUI (graphic user interface) code. That applies to both Goldenseal Pro, and the small phone apps we will link to it.

On Windows, that means developing with Android libraries, using the Java programming language.  On the Mac, it means using Cocoa libraries, and a combination of the new Swift language, and Objective-C.  On desktop it also requires Objective-C++ to link to our existing C++ code.

The number of libraries and new languages is daunting, but fortunately it is not as bad as it might seem.  All of the libraries use concepts that are very similar to what we already use. And all those new languages are pretty much derived from C++.  The syntax may be different (some have semi-colons and some don’t) but the coding is the same basic stuff.

Meanwhile, our contractor is currently on vacation, but he is going too slowly and we’ll have a discussion with him soon.

With both Swift and Java, it’s possible to develop a GUI with very little actual coding.  The hard part is linking it to our existing code.  If we write the Pro GUI in Swift, it’s going to involve four different programming languages!

C++ is a great language, and getting better all the time. Unfortunately, it seems like Apple, Google and Microsoft all want to lock developers into writing code that can’t be used on other platforms.  Nice for them, sucks for us.

Dennis Kolva
Programming Director
Turtle Creek Software

Goldenseal database code (July 19)

We are still fine-tuning our basic database code. It is a gradual process since it requires testing, and making design decisions.  This is pretty much the last programming we need to do, outside of the GUI (graphic user interface).  Our contractor is making slow progress on that.

We use a B-tree to store the disk location of every record.  It is a popular design that is similar to the older NeoAccess code.  However, we make each node much wider, thanks to the generous amounts of RAM available on modern hardware.  It’s also faster that way: these days, a 64K read takes much less time than 16 4K reads.

Neo had a complicated system to keep track of empty spaces in the database (necessary to keep the file from becoming huge).  It was buggy, so in 2003 we replaced it with our own File Manager.  The new system performed well but took up a big chunk of memory, which sometimes caused quits or file corruption when there was not enough RAM to load it. In theory we could have rebuilt the FileManager from the NeoAccess records, so corrupted files could be fixed.  Unfortunately, the Neo code was so complicated that we never could make that happen.

In Goldenseal Pro we now use multiple File Managers, so each keeps track of records in just one sector of the file (32,000 records). They have a relatively small memory footprint, so they’ll be quicker and safer.

In the new database, the location of every record is stored in two different places, and now it uses code that is easy to understand. That means we can cross-validate them, and fix any errors.  Our own company file actually has a few corrupted records from the early 2000s, so it will be the perfect test platform for developing a self-repairing database.

The classic B-tree requires ‘branch balancing’ code to keep it in shape, and part of the complexity in NeoAccess was that it kept moving things around to balance the tree.  We add ID numbers sequentially, and most users don’t delete many records.  Our initial design takes advantage of those conditions, but we are still researching other ways we can make it better.

Dennis Kolva
Programming Director
Turtle Creek Software

 

The 90-90 Rule

There is a saying in software engineering that the first 90% of the project takes the first 90% of the time, then the last 10% takes the other 90% of the time.  For brand-new coding, I think it may even be a 90-90-90 rule, where the final 1% takes the really-last 90% of the time.

Anyhow, our new database code is working pretty well now, though we still need to test it on really large data sets, and stress-test it over the summer.  In other words, the second 90% is now completed.  Everything is much simpler and safer than the old NeoAccess code, so it will be much easier to maintain.  It’s designed for modern systems and uses modern C++.

We also finished the code that reads existing Goldenseal files, and writes them to the new file format.  That means that people’s existing Goldenseal data will be very easy to move into Goldenseal Pro.  We will be testing that code as well for the rest of the summer.  We’ll run Turtle Creek on the new version for a while, before opening it to beta testers.

Dennis Kolva
Programming Director
Turtle Creek Software

Website outage July 6/7

Our ISP had a server melt-down, and turtlesoft.com was out of commission July 6 & 7, and possibly partial days on either side.  It’s back in action now (Fri, July 8).

Emails sent to us during that period were temporarily delayed.  We have a big pile of old messages now, but it is possible some were permanently lost during the outage.  If you sent us an email recently and don’t get a reply today, please resend.

Our apologies for any problems this may have caused.

Dennis Kolva
Programming Director
Turtle Creek Software

Goldenseal Pro Progress Report #6 (June 24)

The new database code in our Goldenseal Pro software now works.  A few details are still not finished, but it looks up records from the company file, and saves them.  That’s 95% of what a database needs to do.  The indexing design is much simpler than the old NeoAccess code, so it will be easier to debug and maintain, and probably will run faster.  There were a few serious design flaws in the old code, and they are now gone.

Our contractor is still working on the GUI (graphic user interface), but the Mac code now shows a data entry window, and the Windows code is making progress.  It requires a lot of prep work to get that first window showing, and we hope that progress will be faster after the basics are finished.

The next thing on deck for our staff is to write a converter that will read existing Goldenseal files, and convert them to the new file format.  We will still use the older NeoAccess code to read files created in version 4.93 and earlier.  However, we are eager to get rid of NeoAccess entirely, so we probably will only support the conversion process for a year or so.  Latecomers will need to do a two-step conversion.

Dennis Kolva
Programming Director
Turtle Creek Software

Farewell to NeoAccess

When we started work on the Goldenseal software back in 1993 or so, we licensed an object database library called NeoAccess to manage how records are stored on the hard drive.  It performed well, but as we neared completion we started to discover serious bugs.  Unfortunately the Neo developers disappeared, and never fixed them.  It was too late to start over, and there was no good alternative to replace it.

So, we rewrote their index class in 1999, and fixed an awful bug that corrupted the entire file if you deleted more than 16 records.  In 2002 we did another rewrite, and added more changes to make it more reliable.

Since then we have gradually improved or replaced many parts of NeoAccess.  Unfortunately, the main part of the code was just too complicated to change.  It worked, but was extremely complicated, and almost impossible to debug or improve.

A few weeks ago we started on a major refactoring of all the NeoAccess code.  We stripped out about half of it, but when we got to the hard parts we decided it would be easier to just start over, rather than fix the rest.

We looked into SQL and other relational database libraries, but soon decided it will be better to stick with an object database.  We checked a few open-source object database libraries, but none are very solid.

So, last week we started a complete redesign and rewrite, based on the parts of NeoAccess that worked well.  It only took a week to create the new database classes, and we now have a brand-new, working database.  It still needs a few weeks to polish and debug, but it will end up being much more reliable than what we had before.

The new database allows 64-bit addresses, so the max file size increases from 4 gigabytes to 9 exabytes.  Big number names jump by 1000x at a time, and go kilo/mega/giga/tera/peta/exa.  That means it will support yuuuge files.  A few users have pasted in many pictures and come close to the 4-gig limit, but now that won’t be a problem.

We’ve redesigned the indexes that keep track of where every record is located. That info is now stored in two different places, so we can recover more easily if data is corrupted.  NeoAccess scattered the indexes all over the file, but we will keep them near the beginning of the file so they are less likely to be lost or corrupted.

The new code also uses much larger memory blocks. Modern systems can move one 64K chunk much faster than 64 1Ks. That means the new code will be quicker, and also easier to debug.

Generally, the new code is simpler, more compact, and much easier to understand.  C++ has come a long ways, and we were often able to replace 20 lines of code with just 1 or 2.

One thing that will help us in the redesign is to see some basic file data from people who have been using Goldenseal for a long time.  If you have a few minutes, please do the following:

1   Open your company file.
2   Choose Security from the View menu, and choose Database Report from the submenu.
3  Click the Save as Text button, and save a file.
4   Attach the file to an email and send it to support@turtlesoft.com
5   The data is anonymous. It only tells us the number of records in each class, which will help us design the best index sizes for typical users.

Thanks,

Dennis Kolva
Programming Director
Turtle Creek Software

New Goldenseal File Format

While working on the code that saves changes to disk, we ended up rewriting some old, complicated NeoAccess code which kept track of “dirty” records that needed to be saved.  It was 2 files and a few hundred lines of code, that we replaced with about 10 lines.  Saves will now run a bit faster, but more importantly, will be more reliable, and much easier to maintain.

We ran into the same thing while rewriting lists (arrays) in PowerPlant. The new code is about 10% as big, and 10x easier to understand.  That translates to easier maintenance and fewer bugs.

The C++ programming language has improved enormously since we started using it, which is the main reason we could shrink things so much. Since 1998, the “standard library” has added many new classes to handle programming tasks that used to require hand-rolled code.  Our old compiler did not support them, but now we can be more up-to-date.

Since the initial work has gone so well, we are ready to start a major rewrite of the basic database code.  We can throw out a lot of hard-to-maintain code from the 1990’s, and replace it with clean, modern, simpler language.  It’s also time to modernize the file format.

Back in version 2.0 we did a similar overhaul to the database format.  In that one, users had to export their data to a text file, and then import it back into a new file.

However, this time around we can make the process much easier.  When you open your old company file, it will automatically read all your records using the existing format, and write them to a new file with the new format.  Everything will look the same in the new file, but the basic database format underneath it will be different, and better.  Slightly better performance, and much greater reliability.

We will be testing the new code on our own Turtlesoft company file this summer.  Some time in the Fall we will also be ready for beta testers, to try out the new format.  Please let us know if you are interested.

Dennis Kolva
Programming Director
Turtle Creek Software