Part 1: Damaged Files and Hidden Corruption in FileMaker Pro ( A Guide to "Recover" )

Part 2: TechInfo Article 102516

Excerpt from:

What Happens to FileMaker Pro Files During Recovery ?

FileMaker 2.0

As posted in Tech Info Article claris.com/support/techinfo/articles/102516.html
No longer in the TechInfo Knowledge Base

Editor's Note: While quite old, the steps are mostly the same. It is quite possible that Recover was originally intended to be a validation utility

Recover step by step ...

The first step in Recover is to open the damaged database and create a new empty database to hold data blocks that are about to be copied from the damaged file

Next, FileMaker resets the logical End of File (EOF) in the damaged database to be equal to the physical EOF and starts copying each block of the damaged database over to the new target file. As each block is copied, it is examined to validate its internal structure. If any problem is found, the block is repaired

The name of this step is a bit of a misnomer. Rather than rebuilding the status information, FileMaker actually reverts the database to a default state similar to a brand new database. Once again, this is consistent with the philosophy of removing what can easily be recreated in order to concentrate on capturing as much of the old content as possible.
The items that are removed or reverted include:
Record Count, Summaries, Sub-summary sort order, Sorted Order, Custom Sort Order, List of Found Records, Import Order, Export Order, Calculation trigger table. Find Patterns, Sort Specification

Next, FileMaker validates each record in the database. This process involves fetching each of the records and checking to make sure that it has valid header information. Also, each field within the record is checked for a valid key and valid (non-zero) length. If any invalid fields or records are discovered in this process, they are removed from the database

As each field is validated, it is checked against a master list of existing fields. If a field is encountered that does not already exist in the master list, the key is added to the list so that a temporary field can be created automatically later. Again, this is consistent with FileMaker's philosophy of preserving as much of the information as possible. It is easy to delete unwanted fields after the database is recovered; much easier than attempting to recreate lost information

Furthermore, in FileMaker Pro 2.0, any data in repetitions of repeating fields that are beyond the maximum value specified in field definitions are deleted. Deleting these unseen repetitions will cause any summaries or calculations that include the repeating fields to be calculated correctly because they no longer involve the "hidden" data. When FileMaker finishes checking the records, it writes the maximum record count and the maximum record key back into the database

In this phase of recovery, FileMaker validates each layout in the database. First, the current defaults (i.e., current font, size, style, etc.) are replaced with defaults that match those of a newly created database. In addition, lists used by FileMaker to quickly access the layout names and information about custom layout order are deleted.

Each layout is then examined for consistency. Every layout must have a name (or it is given a default name), parts (at least one) with valid options and valid size, and a valid object list. The object list is a structure that contains a reference to each object (i.e., field, rectangle, line, text, etc.) on the layout

Each object referenced in the object list is fetched to make sure it is present and that it is a valid object type. Any objects that cannot be examined or are not of valid type are deleted from the layout. If any invalid parts are encountered, they are also deleted. If no parts are left after this process, a default part is added to the layout to make it accessible

Next, FileMaker validates all field definitions and scripts. First, lists used by FileMaker to quickly access the field names and information about the custom field order are deleted. While validating field definitions, any fields without matching definitions (discovered while checking the records) will have new field definitions automatically created.

Each automatically generated field is named "Recovered Field n", where "n" is a number starting at 1 and incrementing with each field ("n" is automatically generated during recovery). FileMaker also fetches each field definition from the database and verifies that it has a valid field name and valid field type. If the field type is missing or invalid FileMaker makes the field a text field (the most flexible field type), so any data can be retrieved from the field.

In the case of calculations and summaries, the formula is fetched and validated. If any of the formulas are invalid or cannot be found a default (empty) formula is created for the field. Finally FileMaker writes the count of the number of fields into the file

When it has finished processing field definitions, FileMaker validates the scripts, in the database. As was done for layouts and field definitions, FileMaker starts by deleting the custom order information for scripts. Every script is then checked to make sure it has a valid name, valid status information and valid options. If a problem is detected with any of these important components of a script, the script is deleted

After checking each record and rebuilding the field definitions, FileMaker is finally ready to reconstruct the inverted list or index. The index is used to assist FileMaker in locating data quickly. After completely deleting the existing index, FileMaker fetches each record; and, based on the data in the record, reconstructs the index. This step is necessary to maintain consistency in the database. If the index were not rebuilt, it would be possible for FileMaker Find requests to return records that did not match the specified request

Finally, FileMaker frees unused space in the file. All of the disk blocks that may be unused after recovery are coalesced to the end of the file and removed

Recovery Results
When recovery is completed, a status dialog is displayed showing what was done to the database during recovery. The dialog shows:
Number of bytes copied from the damaged database into the new database
Number of records that had to be skipped (deleted) from the recovered database
Number of field values that had to be skipped (deleted) from the database
Number of field definitions that had to be recreated

If any of the values shown, other than the number of bytes copied, is greater than zero, you should carefully check the file for consistent content.
For example, if field definitions are recreated by Recover, you can create a new layout with the "recovered" fields and search for records where the field values are not empty. Based on the contents of each field, you may decide to delete the field again

In many cases, a successfully recovered database may be larger than the original database.
This is normal and is caused by new disk blocks being allocated as the database is recovered. For example, rebuilding the index field by field and record by record can cause data distribution that is different (and possibly larger) than the original file

A newly recovered database will also take longer to open than a database that was closed properly the last time it was used.
This only happens the first time a recovered database is opened and is the result of rebuilding various internal structures that were deleted during recovery

----------------------------------------------------------------------------
This article originally appeared in issue 47 (1992) of The FileMaker Report: A Journal for FileMaker Users, published ten times a year by Elk Horn. The first issue was published in 1987