FoxPro query running hundreds of times slower in vDos

[Jorge Zaldívar]

I have the following piece of FoxPro code:

create cursor qDigits (digit C(1))
for i = 0 to 9
    insert into qDigits value (str(i, 1))
endfor

select;
    T1.digit + T2.digit + T3.digit + T4.digit + T5.digit as sequence;
    from qDigits T1, qDigits T2, qDigits T3, qDigits T4, qDigits T5;
    order by 1;
    into cursor qAll

The query takes 80-100 seconds to run under vDos.

On an old Windows XP computer (Pentium 4, 2.8 GHz, 2 GB RAM), it takes 0.5 seconds. Same as on an emulated (VMWare+Win10) WinXP machine.

By the way, under vDosPlus the query also runs very slowly (it takes 10 seconds), but not as extremely slow as under vDos.

Is this expected behavior?

[Jos]

I ran the query in version 2018.05.01, it took 7 seconds to finish. Somewhat faster in the next version, this conforms to the 10 seconds in vDosPlus (based on an older version of vDos).

Your query doesn’t do disk access. I guess you ran it on a slow system, the query already took longer and vDos went into idle mode before the query finished, not detecting any ‘real’ activity. You can check this by the Taskmanager.

Jos

[Jorge Zaldívar]

I don't have a top-of-the-line machine but I don't consider my system particularly slow, but perhaps I'm comparing it to the old WinXP we have lying around (Pentium 4 w/2 GB RAM)  . It's an Intel Core i5 with 8 GB of RAM and an SSD.

Here's an screen capture of the query running.



I ran it twice and here are the results:



They took slightly over two minutes to run each.

The vDos executable is dated 30/05/2018. 

[Jorge Zaldívar]

I ran it again with logging activated. I'm attaching the log file. There're a lot of FindFirst failed entries, don't know if that's OK.

Attachments:

vDos.log (3.92 KB)

[Jos]

Hard to tell if vDos is idle or not during your query, there’s a lot more going on. Better select the Processes tab and watch the CPU usage at starting the query. I wouldn’t be surprised if you closed all other applications, the query would also run in 7 seconds. Although Windows 10 is multi-tasking, the more applications are running, the slower they will run individually. If you’re using FoxProX, you could also disable the mouse. That has a real impact on its performance under the current vDos version (and NTVDM).

The “FindFirst failed …” entries are of no concern, mostly FoxPro testing for utility programs of the extended/developers version. At 109.01 seconds after vDos started, it went idle for 79.31 seconds, executing FoxPro at a lower rate. Mostly due to vDos deciding FoxPro doing nothing useful since some time. Questionable is still your query, how is this related to an actual program (no disk access etc.)?

Jos

[Jorge Zaldívar]

I am using FoxProX but setting MOUSE = OFF didn't have a significant effect. Neither did closing all the other (foreground) applications (background tasks kept running).

I observed the Processes tab as you suggested and noted that CPU use never spikes, it did fall to 0% during the execution of the query (I installed FlashBack recorder and recorded, if you're interested). Several times, CPU use increases and then goes back to 0%. It never increases by too much, I think it rarely surpasses 10% and never 30%. In fact, when vDos is actually idle, CPU use is higher!

That (CPU use being higher when idle) gave me an idea. I ran again the query, but this time, while the query was running, I pressed the shift key several times: execution time dropped to 15 seconds. WT?

I also tested FoxPro (not FoxProX) and the query ran in just under 9 seconds. For the real apps I need FoxProX.

I'll tell you later why I'm doing this type of query in a real-life application.

Interested in the recordings (1.4 MB, FBR format, FoxProX with and without a key being repeatedly pressed during execution of query)?

[Jos]

When vDos is idle it releases processing time to Windows. So vDos being idle and a higher CPU usage contradict. The CPU usage somewhat increasing periodically is just vDos still attending the DOS program. Taskmanager’s timings are not real-time nor that accurate to show in detail what’s going on.

Pressing a key is a (modest and temporary speedup) signal to vDos; something is still actually going on. If the test ran fine in FoxPro: FoxProX always lacked in speed due to its virtual paging memory addressing. The next vDos version has three distinct emulated CPU’s, one dedicated to virtual paging mode. FoxProX will then be even faster than FoxPro.

Jos

[Jorge Zaldívar]

I though you'd understand but, apparently, I didn't make myself clear. I'll try and explain myself.

When FoxPro is just waiting for keystrokes, vDos uses more CPU than when FoxPro is running the query. So, vDos, incorrectly, identifies the intensive process of performing a full join on five tables (albeit very small ones, but still producing 100,000 records) as FoxPro doing nothing and so, goes idle (releases processing time to the OS). I would expect vDos to identify that task (the query) as requiring more processing power than just waiting for a keystroke. It doesn't. As you mention, vDos needs the user to signal it that "something is going on" by pressing some keys.

By the way, your answer (vDos getting "woken up" by keystrokes) surprised me. I've read several posts in which users mention vDos being sped up by pressing random keys while a certain process is running and your answer had always been either that you didn't understand what they were referring to by "pressing some keys" or that it had nothing to do with the topic being discussed.

Regarding Task Manager's lack of real-time precision, I only used it because you suggested opening the Processes tab and watching CPU usage.

The next version of vDos being "smarter" in detecting computing needs (with its emulated CPU dedicated to virtual paging) is, indeed, very good news!

[Jos]

FoxPro constantly probes the keyboard, so that can’t be used to determine it is idle or not. Only actually calling BIOS - Wait for key (eventually mids DOS) can be used, for the rest mostly disk (in)activity.
A key press could mean activity, the DOS program waiting for a key actually puts vDos in idle mode. I only remember a post about adding waiting for a key, then pressing one speeding up NTVDM operations. I suspect this is caused by TAMEDOS (lowering CPU usage), in vDos that method would only slow it down, while it was running at full speed (due to disk activity).

Task Manager is the first instrument to get an idea how busy vDos is, though fluctuations of low CPU usage doesn’t mean vDos is more or less idle/busy.

The dedicated page mode only makes FoxProX run faster. The next version also has a new idle detection scheme. Whereas the current version by default runs at 100% and determines when to go idle, the next version will be idle, and eventually switch to 100% for sometime time. Main difference will be that the current version miscalculates when to go idle (too soon) with disk activity. If your query then runs in let’s say under 7 seconds, it’s mostly due to improved speed, the query finishes before vDos goes idle.

Jos

[Jorge Zaldívar]

As for the reason for my query.

It's part of a routine for automatic assignment of UPC bar codes. Currently, the database has non-consecutive codes because they were trying to assign meaning to the diferent digits of the code (family, size, color, etc.), contrary to UPC guidelines. So there are a lot of gaps in the sequence.

So, to assign codes, I generate a cursor with all of the possible codes (numbers from 0 up to 99,999, i.e., the query in question), then I select the already assigned codes, and finally I select from the "possibles" cursor those that are not already assigned into the "available" cursor.

The first and the third queries return tens of thousands of records, from which I would only need a few. But since, when making the proof-of-concept procedure and testing it, they ran in about 0.5 seconds each, bringing the total time for the routine to about one second, I decided to use them as they were (I wouldn't spend several hours to save a few seconds).

But, remember I develop on a dinosauric, Pentium, WinXP, 2GB-RAM computer, so that vDos would not corrupt my source code files. And then, the acceptable 0.5 seconds run time was on the dinosaur. On a (more) modern computer, with vDos, the run time scalated to around 2 minutes. Even the 10 seconds it took to run on vDosPlus are prohibitive.

No problem. Being a programmer, I'm used to code around limitations (in this case, of vDos), so I divided the cursor of all possible alternatives in "pages" of 1,000 rows. If there are not enough codes remaining in the first page, the procedure will take another one, and so on. I can foresee problems with this approach as the first page with available codes moves towards the middle of the sequence (easily solvable recording the "current" page on a table), but for now, the whole process runs in fractions of a second... even on vDos. 

And that's is the reason for that query with no disk access.

Regards,

[Jos]

If the 10 seconds delay is excessive, the 7 seconds of the next version also is. While there’s no way to speed that up further, the emulated CPU is just slower than the real thing. So your software solution is indeed the sensible solution.

vDos corrupting source code files is still strange, no other reports to this happening. FoxPro(X) could well be the most used DOS program in vDos. Though still those (only recent resolved) page fault errors in vDos/FoxProX. Can just be symptoms, trapped by more extensive operational checking, while the direct cause isn’t directly related. The FoxProX program already ‘misbehaving’ long time before, even not trapped at all.

Jos

[iobagg]

Same here. For Foxpro 3.6. In windows xp it loads in 3 seconds, while on i7 with ssd it takes 20 seconds. Other operations are very long waiting. For a operation of 5 seconds in Windows XP I wait 3 minutes on Windows 10



Is it any solution for this?

Windows 10

CPU: Intel® Core™ i7-3610QM Processor

8 GB RAM

SSD Sata 3.0

[Jos]

FoxPro 3.6 is a Windows program, so that won’t work in vDos.

If you however mean FoxPro(X) 2.6, the IDE loads instantly. You could have a look at www.vdos.info/faqs.html - Miscellaneous - My application (sometimes) runs slower than before.

Though for an operation to take 3 minutes while 5 seconds in XP, you would have to setup some artificial logic, and perhaps use an older vDos version.

Jos