How to increase computing speed (additions)?

[drschnagels]

Hi there!

Is there any way to increase the speed of database math-additions?

I have a (small?) database, let's say 70.000 rows and 8 columns.

When entering our data into a form and hitting "sum up" the program works through the mentioned database and adds all values in column "test1" and writes sums and found entries into an seperate "test1-sums-by-entries" file.

In Virtualbox with WinXP this takes less than 10 seconds. In vDos 10 Minutes.

Still.... vDos is sooooooooooo much more convenient and better to use. I'd just love if i could make it faster without trimming the database (deleting rows).

How can i make sure the new FPU is used? Can I find out if there is a memory problem going on?

Our old DOS/Win7-32bit/Virtualbox+WinXP used these special settings:

config.nt:

dos=high, umb
device=%SystemRoot%\system32\himem.sys
files=100

autoexec.nt

SET CLIPPER=F71
SET DBFPFAD=V:\xxx\ddddd
SET CLSYSPATH=V:\xxx\ddddd

Thanks a lot in advance!

Michael

[Jos]

No way to speed up vDos operations by a setting in vDos.

70,000 rows and 8 columns (indexes?) is indeed a relatively small database, even in the DOS days.

vDos being some 60 times slower than NTVDM (Windows 32-bit, most XP versions) is extreme.
I guess it’s not actually a problem of processing power of the simulated CPU (some 100 MIPS), but vDos simply going idle while the computations are performed. The idle detection of the current (and previous) vDos versions has a mishap: It goes idle to soon. You can check this with the Windows Task Manager: The vDos CPU usage should be maxed out during the recalc operation. I reckon it isn’t, due to a delay between file operations.

Nothing to do about that, though it’s fixed in the upcoming release of the next vDos version.

Jos

[drschnagels]

Hi Jos!

I have taken the attached data.

Is this what you mean?

Furthermore i deleted entries in the database (now ranges about 1 year of entries = 400 rows). The program calculates then extremely fast (2 seconds?).

(But of course this was a test, we need those entries).

What can you analyse in my screenshots? Can I do another test?

Thanks a lot!

Edit: Yes, it maxes out my 4/8 CPU at 12,5% during calculation.

Attachments:

[Jos]

Your screenshots didn’t clarify much (anything).
You added the environment variable settings also to autoexec.txt?
The V: (mapped) drive is also set/available in vDos, it’s actually some local drive?
The mean performance issue of your programs "sum up" operation would be disk i/o, not adding numbers with, or even w/o FPU.
And when disk i/o is the bottleneck, vDos performance should come close to NTVDM…

Jos

[drschnagels]

Jan 18, 2019 0:25:56 GMT 1 Jos said:

Your screenshots didn’t clarify much (anything).
You added the environment variable settings also to autoexec.txt?
The V: (mapped) drive is also set/available in vDos, it’s actually some local drive?
The mean performance issue of your programs "sum up" operation would be disk i/o, not adding numbers with, or even w/o FPU.
And when disk i/o is the bottleneck, vDos performance should come close to NTVDM…

Jos

Sorry, I thought this would help.

config now has only printer settings.

autoexec has this, like before:

@echo OFF
SET CLIPPER=F71
SET DBFPFAD=C:\BDADaten
SET CLSYSPATH=C:\BDADaten
call bda2008.exe
exit

I copy the whole program and database directory for tests on my local drive. We usually use it on a mapped drive.

The whole database is 4 files with about 4,2 MB. Some other .idx files etc. In the end its about 7 MB.

From my understanding (screenshot, lower part) it is not disk I/O problem at all.

[Jos]

Well, the i/o graph mostly only shows the number of bytes read/written. Not how much time that took in relation to the CPU usage.
What if you compare your program running in vDos versus NTVDM, local versus networked?

Jos

[drschnagels]

Jan 18, 2019 9:51:48 GMT 1 Jos said:

Well, the i/o graph mostly only shows the number of bytes read/written. Not how much time that took in relation to the CPU usage.
What if you compare your program running in vDos versus NTVDM, local versus networked?

Jos

There is zero difference between local and networked.

Using a WinXP Virtual Machine our program performs totally normal. Caculations are done in less than 20 seconds?!

vDos takes for the same calculations 7-10 minutes.

Im very sure that its the calculations: Order ID numbers which are occurring once (f.e. 19444 as order number) calculate super fast. They have only a few hours saved to the database.

"Order numbers" like ".WA" occur thousands of times in the database. This is the entry were we sum up work hours for factory cleaning f.e..

I found the part in the source code which takes up so much time. Maybe that would help to find the problem? Unfortunately im not a programmer.

[Jos]

20 seconds is 21-30 faster than 7-10 minutes.
At the FAQ’s page of vdos.info: “In raw processing power the emulated CPU can be up to 40 times slower than the real thing!”.
Typically a DOS program runs much faster because disk i/o is very time consuming. However, the program being equally fast local and networked, indicates there’s real little disk i/o. So also little compensation of vDos CPU slowness.
You could send me the program (eventually scramble the data) so I can do a profile what’s holding back vDos. Preferably use wetransfer.com (because of the size and executables that are blocked otherwise).
My mail address is: [e-mail] (change the “a” to an “o”).

Jos

[Herman]

It seems suspiciously like my earlier observations, that vDos is much slower than if I run this in NTVDM in Windows 7 with dBase V.
The only difference is that far greater differences occur when I press a key in NTVDM during a run. Then it is much faster (up to at least 10x) in NTVDM in a DO WHIL where record for record (total +/- 60,000) is read from a database to variables and then further generated in an output run as a report, then as I do that in VDOS.
Just as an input record page, VDOS is a small improvement in the accuracy.
I am not a LOW level programmer, but I would really like to know why that is. For the record, I work local on the c-drive.

There must be something that has not been arranged in VDOS, I would think about the effort towards 100% CPU processing.

Herman

[Jos]

dBase V running faster in NTVDM when pressing a key can hardly be considered something would be missing in vDos, it’s just a NTVDM mishap (or do you use TameDOS?).
Since vDos runs in one single core, it can only utilize 100% of that core, not 100% of a CPU with more than one cores. Simulating a CPU in software is of course always slower than the real hardware.

Jos

[Herman]

Jos,

I think that there is really a miscommunication about speeds and where and especially how and why.
This is probably also partly the case.
My excuse for this.
I am not a programmer, but always know how to detect and estimate the right pain points. This explanation is difficult and because I probably do not always use the right terms.

What happens in my situation in Windows 7 (32 bit) Virtual running NTVDM on Windows 10 with Hyper V:
When I press and hold a key, the data record processing and variable content throughput are much faster and the total application control speed increases.
The same things happen on an older Windows 7 (32 bit) computer.

Not that it feels sluggish without a key press, but the speed is at least 10 times faster (read 60,000 records (20 Mb) in about 1 minutes).

If I do not press the button, it will take about 10 minutes.

If I run this run in VDOS, this takes at least 30 minutes and a key press has no effect.

10 minutes in NTVDM (Windows 7) and 30 minutes in VDOS I find a very important difference to investigate.

I do not define anything specifically separate for the HMA or extended memory in dBase.

The dBase write buffer is standard with me; because this (also in the past with other users) causes unforeseen I / O problems.
NTVDM disk buffering is simply regulated in Windows.

The typed speeds from my XP time were about the same with Virtual PC as in this message by drschnagels.

There was also a considerable speed gain under the circumstances.

Note: I do not use any other third-party software to influence anything or whatever, for these actions.
So only default Windows, so no specific settings made.

My question is: What is going on here?

Herman

[Jos]

In NTVDM dBase runs natively on the CPU, while in vDos the CPU is emulated in software. Just consider fetching an instruction to execute (simplified, real mode):

Real CPU: Establish the memory address (CS:IP) where the instruction is located, get it, increment the instruction pointer, perform the instruction. Only one memory access, the rest is internal to the CPU.

Simulated CPU: Get the code segment register and instruction pointer, get the instruction, increment the instruction pointer, call (the real CPU jumping) the function that emulates the instruction. Not one, but five (minimal) memory accesses since the emulated CPU internals come from there. Only then the instruction itself is emulated, even something simple like "ADD AX,BX" results in memory accesses, and more jumping to continue the CPU operation.

Don’t know why dBase would run faster in NTVDM with key presses; a mishap of NTVDM, or else a timing issue in dBase…

Jos