|
|
Our Hot Pick: Rising Antivirus 2006 - Certified by TUV & Checkmark! Get 10% discount by entering this coupon code: ONDISCOUNT10
I'm running a certain search engine on a Windows 2003 server, with
1.5GB of RAM (which is enough for the processes). The search engine
uses a 15GB index. The index is updated about once an hour, using
this
method: an updated index is copied to a temporary directory from
another server; then the new and the old indexes are renamed so the
new becomes active; and finally the old is deleted.
The problem we're facing is that the search performance decreases
significantly after the index update, and it takes some time till it
gets better. My bet is this: between the index updates, the file
system cache stores the important file parts, and so it performs much
better. The file copy, however, overloads the cache, and forces it to
flush the important content. Or, if the file copy doesn't actually
use
the file system cache, it is the deletion of the old index that
invalidates all the important cache content.
Given this update scenario, how can I reduce the performance hit
after
each index copy? Will increasing the file system cache or so help?
|
|
What database is used to store the index? I'd be more inclined to think it
was an issue with application level database cach filling. When you load a
new inex, you're essentially at cold start until cache populates.
John
"kalevet" <kalevet[ at ]gmail.com> wrote in message
news:1176971503.765129.212580[ at ]p77g2000hsh.googlegroups.com...
[Quoted Text] > I'm running a certain search engine on a Windows 2003 server, with
> 1.5GB of RAM (which is enough for the processes). The search engine
> uses a 15GB index. The index is updated about once an hour, using
> this
> method: an updated index is copied to a temporary directory from
> another server; then the new and the old indexes are renamed so the
> new becomes active; and finally the old is deleted.
>
> The problem we're facing is that the search performance decreases
> significantly after the index update, and it takes some time till it
> gets better. My bet is this: between the index updates, the file
> system cache stores the important file parts, and so it performs much
> better. The file copy, however, overloads the cache, and forces it to
> flush the important content. Or, if the file copy doesn't actually
> use
> the file system cache, it is the deletion of the old index that
> invalidates all the important cache content.
>
> Given this update scenario, how can I reduce the performance hit
> after
> each index copy? Will increasing the file system cache or so help?
>
|
|
Ok, I doubt it has anything to do with anything else except the file cache.
If you can find someone who has Exchange 2003, try using the ESEUTIL.exe
tool from it to copy the file. I've been doing testing and find a huge
difference in resource utilization and speed. Eseutil bypasses the windows
disk cache and therefore won't flush the existing contents of the cache.
I've been testing with a 200 GB file. Without eseutil I see a max copy
speed of about 25 MB/sec. Using eseutil I see about 60 MB/sec. I've also
tried it with multiple smaller files with a similar result. The biggest
issue I have with using ESEUTIL is that it doesn't allow wildcards, you have
to specify each file individually.
Syntax:
eseutil /y <sourcefile> /d <destinationfile>
eseutil /y d:\largetestfile.txt /d \\fileserver\testshare\largetestfile.txt
You probably have issues as well with the cache growing so large that it
starts to swap your apps out to make room for it. I've seen 64 bit system
with 8 GB of RAM use 7.3 GB for the disk cache even though it already had
1.5 GB committed. Use perfmon to check "Memory:Cache Bytes" and
"Memory:Cache Bytes Peak" to see the current and max size of the disk cache.
If it is too large, then you might want to look at sysinternal's cacheset
utility, which can be used to clear the cache or manage/set it's size. When
I have a box I need to use it on, then I usually need to run it every 5
minutes.
Hope it helps. If it helps, then please let others know about it. The more
people who know about it and complain to MS, the better the chance that
they'll do something about it.
Thanks,
Stephen
"kalevet" <kalevet[ at ]gmail.com> wrote in message
news:1176971503.765129.212580[ at ]p77g2000hsh.googlegroups.com...
I'm running a certain search engine on a Windows 2003 server, with
1.5GB of RAM (which is enough for the processes). The search engine
uses a 15GB index. The index is updated about once an hour, using
this
method: an updated index is copied to a temporary directory from
another server; then the new and the old indexes are renamed so the
new becomes active; and finally the old is deleted.
The problem we're facing is that the search performance decreases
significantly after the index update, and it takes some time till it
gets better. My bet is this: between the index updates, the file
system cache stores the important file parts, and so it performs much
better. The file copy, however, overloads the cache, and forces it to
flush the important content. Or, if the file copy doesn't actually
use
the file system cache, it is the deletion of the old index that
invalidates all the important cache content.
Given this update scenario, how can I reduce the performance hit
after
each index copy? Will increasing the file system cache or so help?
|
|
|