Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Synchronize Album - Extremely slow since upgrading to GSP 2.6.0
Petercloet
#1 Posted : Monday, 16 January 2012 3:05:02 AM(UTC)
Rank: Advanced Member

Joined: 10/12/2007(UTC)
Posts: 34
Location: Belgium

Hello,

I upgraded to GSP 2.6.0 in November.
Up to now, I haven't had to add any new files to my gallery so I didn't use the synchronizer.

This weekend, I added numerous files to the gallery and fired of the "synchronize albums" job.
Was I in for a surprise. The synchronization of new files was extremely slow. In the past, I used to achieve an average sync rate of 12-13 objects/sec. Now, I'm down to 0.1 objects/sec.

I was under the impression that the move SQL CE would improve performance.
Something must be wrong...

Any pointers on where I should start looking to track down the cause of this bad performance?

Peter
Roger Martin
#2 Posted : Monday, 16 January 2012 8:05:17 AM(UTC)
Roger Martin

Rank: Administration

Joined: 3/08/2007(UTC)
Posts: 3,300
Location: Fort Atkinson, WI

SQL CE is somewhat slower than SQLite, but I have not observed the 120x slower performance you are seeing. I am not sure what can explain that. You can always try a compact and repair on the DB (Site Settings - General page), but I doubt that will be a cure-all.

For the best performance, switch to SQL Server. Even the free SQL Server Express has very fast performance. The Admin Guide has instructions for doing a conversion.

However, it is possible the bottleneck is caused by something other than the DB, in which case switching to SQL Server may not have a noticeable effect. In my experience, the vast majority of server resources during a sync are involved in loading images into memory and creating the thumbnail and low-res versions. If you are syncing images that are larger than before, you will notice a slowdown.
Roger Martin
Creator and Lead Developer of Gallery Server Pro
dp4610yz
#4 Posted : Tuesday, 17 January 2012 8:38:43 PM(UTC)
Rank: Member

Joined: 17/01/2012(UTC)
Posts: 5
Location: Singapore

I am also experiencing the same issue with slow synchronizing of the albums....i have added like 40gb of photos in various folders into the gallery... but currently it is taking about 1/2 a min to synch one photo... Anyone able to help to speed up the progress? i have unchecked all but include child directories as i have quite few child directories..
Roger Martin
#5 Posted : Tuesday, 17 January 2012 9:20:47 PM(UTC)
Roger Martin

Rank: Administration

Joined: 3/08/2007(UTC)
Posts: 3,300
Location: Fort Atkinson, WI

I suspect most of the slowness comes from ASP.NET loading each image into memory and building the thumbnail and low-res versions. If your image files are large (say, more than 2 MB each), that will be a pretty slow process.

If you don't care about preserving the originals, make smaller versions with an image editor and then sync those.
Roger Martin
Creator and Lead Developer of Gallery Server Pro
Ernie
#6 Posted : Friday, 20 January 2012 10:40:39 AM(UTC)
Rank: Member

Joined: 20/01/2012(UTC)
Posts: 4
Location: texas

I under stand a lot of this is being done thru the web interface.

Do you have a cli program that will sync, I can't see a user waiting 3 or more hours waiting with a web page open
Roger Martin
#7 Posted : Friday, 20 January 2012 12:24:02 PM(UTC)
Roger Martin

Rank: Administration

Joined: 3/08/2007(UTC)
Posts: 3,300
Location: Fort Atkinson, WI

A client program won't be any faster and will still require an open window. Plus, it has all the disadvantages associated with fat clients.

Remember, you can use auto-sync to sync things up without any user intervention or waiting for the UI.
Roger Martin
Creator and Lead Developer of Gallery Server Pro
Ernie
#8 Posted : Friday, 20 January 2012 12:26:15 PM(UTC)
Rank: Member

Joined: 20/01/2012(UTC)
Posts: 4
Location: texas

Duh, my bad !
Petercloet
#3 Posted : Saturday, 21 January 2012 7:39:16 AM(UTC)
Rank: Advanced Member

Joined: 10/12/2007(UTC)
Posts: 34
Location: Belgium

Roger Martin;8195 wrote:
SQL CE is somewhat slower than SQLite, but I have not observed the 120x slower performance you are seeing. I am not sure what can explain that. You can always try a compact and repair on the DB (Site Settings - General page), but I doubt that will be a cure-all.


Indeed. I tried the compact and repair. It didn't improve performance.
It came back with message that the database was healthy and no inconsistencies were found.
The size went from 52884Kb to 50196Kb. Is this size within normal parameters for a SQL CE database referencing 16.152 files with a total size of 40,5Gb?

Roger Martin;8195 wrote:
However, it is possible the bottleneck is caused by something other than the DB, in which case switching to SQL Server may not have a noticeable effect. In my experience, the vast majority of server resources during a sync are involved in loading images into memory and creating the thumbnail and low-res versions. If you are syncing images that are larger than before, you will notice a slowdown.


The average size of the images I'm sync'ing hasn't changed since the upgrade, neither has the amount of files. So - although my images are on average 3-4 Mb - this can't be the trigger for it running slower after the upgrade.
The platform the Gallery is running on hasn't changed either: a VM running Windows Server 2008R2 on a physical box - also running the same OS. I haven't changed any of the VM parameters (vCPU, memory, disk).
I was thinking it might be related to the video-conversion running in the background and consuming resources but I removed all the conversion entries and checked that it remained "idle" during sync'ing. It didn't improve the performance a single bit.

Was the sync'ing logic updated/changed as part of the upgrade?
The IIS Works Service is maxing out to 97-99% during the sync, but I suppose that is to be expected.
Another thing I noticed was that - although Galleryserverpro is running under .Net 4.0.30319 - the overall .Net Framework version is still referring to 2.0. Could this have something to do with performance? I'm not big a IIS specialist... :-)

I'd like to hold of installing and running SQLServer for the time being because - as you already indicated - I'm not sure this will actually solve the problem.

Regards,

Peter
Roger Martin
#9 Posted : Saturday, 21 January 2012 10:11:30 PM(UTC)
Roger Martin

Rank: Administration

Joined: 3/08/2007(UTC)
Posts: 3,300
Location: Fort Atkinson, WI

The logic didn't change in 2.5 or 2.6 but the DB technology did (SQLite => SQL CE) as well as the technology used to access it (ADO.NET => EF 4.1 Code First). I have done extensive profiling of the code and can confirm that most of the time spent during a sync is loading the files into memory and building the thumbnail and optimized versions. That algorithm has not changed in recent versions.

I suggest logging in to the demo site, creating a test album somewhere in the Public Playground area, and uploading several images. Then run a sync with the overwrite and extract metadata flags enabled. That will give you close to the same performance as a raw sync. If you notice a significant speed difference, then you know it is worth it to migrate to SQL Server.

I don't understand your statement about .NET 2/4. .NET 4 has no dependence on .NET 2, so nothing about the website or app pool should be referencing .NET 2. Everything should be .NET 4. You can even remove .NET 2 and GSP should be unaffected.
Roger Martin
Creator and Lead Developer of Gallery Server Pro
Hallupa
#10 Posted : Saturday, 28 January 2012 4:31:25 AM(UTC)
Rank: Member

Joined: 28/01/2012(UTC)
Posts: 4
Location: UK

I seem to be having the same problem. Synchronising is taking about 100 seconds per image.

I downloaded the source code and added logging around all of the synchronisation code to try to figure out what was taking so long. I traced it back to:

Factory.GetDataProvider().MediaObject_Save(this._galleryObject)
->
GalleryServerPro.Data.SqlCe.SqlCeGalleryServerProProvider.MediaObject_Save and was logged at taking 90267ms
Hallupa
#11 Posted : Saturday, 28 January 2012 5:04:10 AM(UTC)
Rank: Member

Joined: 28/01/2012(UTC)
Posts: 4
Location: UK

Looking in more detail it seems InsertMetadataItems in SqlCeGalleryServerProProvider class is slow.

For some pictures there are 30 meta data items, which this method adds to the database. Each insert for each meta data item takes around 3 seconds to complete which is making up nearly all of the 90 seconds for each picture.

It seems adding to the MediaObjectMetadata table is the bottleneck. Not sure why this would be.

I ended up changing the provider to SQL Server as I couldn't figure out why this was so slow, with SQL Server it's now running at full speed.
Roger Martin
#13 Posted : Saturday, 28 January 2012 2:37:40 PM(UTC)
Roger Martin

Rank: Administration

Joined: 3/08/2007(UTC)
Posts: 3,300
Location: Fort Atkinson, WI

Interesting. 3 seconds for each insert to the MediaObjectMetadata table? Not cool. I wonder if it correlates to the number of records in the table. Can you tell me about how many you have in there?
Roger Martin
Creator and Lead Developer of Gallery Server Pro
Hallupa
#14 Posted : Saturday, 28 January 2012 3:17:58 PM(UTC)
Rank: Member

Joined: 28/01/2012(UTC)
Posts: 4
Location: UK

Just checked and there is 167126 rows in that table and 6937 in the gs_MediaObject table

Oliver
Roger Martin
#15 Posted : Saturday, 28 January 2012 4:51:22 PM(UTC)
Roger Martin

Rank: Administration

Joined: 3/08/2007(UTC)
Posts: 3,300
Location: Fort Atkinson, WI

Thanks. When I get a chance I'll try to repro the situation. One of the first things I'll look into is whether the indexes are causing INSERT slowdowns.
Roger Martin
Creator and Lead Developer of Gallery Server Pro
jambalaya
#12 Posted : Wednesday, 1 February 2012 10:48:52 AM(UTC)
Rank: Newbie

Joined: 15/01/2012(UTC)
Posts: 3

Hallupa;8277 wrote:
Looking in more detail it seems InsertMetadataItems in SqlCeGalleryServerProProvider class is slow.

For some pictures there are 30 meta data items, which this method adds to the database. Each insert for each meta data item takes around 3 seconds to complete which is making up nearly all of the 90 seconds for each picture.

It seems adding to the MediaObjectMetadata table is the bottleneck. Not sure why this would be.

I ended up changing the provider to SQL Server as I couldn't figure out why this was so slow, with SQL Server it's now running at full speed.


I have 47,000 photos and had given up trying to sync with SQL CE (it ground to a halt about halfway after days of processing). After seeing Hallupa's post I changed to SQL Server and succeeded in syncing in about 12 hours, so thanks to Hallupa for pinpointing the issue.

I also have some traps-for-young-players related to the long initial auto-sync processing (potentially for adding to the admin manual) -

1) Turn IIS Application Pool recycling off if you expect initial sync to run longer than the default 29 hours.

2) Change IIS Application Pool idle time-out from the default 20 minutes to 0 if you want auto-sync to last longer than 20 minutes (I am supposing the no-GUI aspect of auto-sync causes the process to look idle to IIS even when it is cranking).

3) When downloading SQL Express from Microsoft, choose the SQL Server database w/tools version (the 2nd option). I started with the standard SQL Express install, only to discover I couldn't create a database. Even though the admin manual mentions SQL Server Management Studio Express for this, it was trial-and-error to discover the SQL Express 1st download option was insufficient.

4) Roger, if you can't get to the bottom of the SQL CE bottleneck, perhaps add some advice in the admin manual about not using SQl CE if you have more than a few thousand photos (probably most folks nowdays..)

Now that I am over these hurdles Gallery Server Pro is looking pretty good!
Roger Martin
#16 Posted : Wednesday, 1 February 2012 10:54:42 AM(UTC)
Roger Martin

Rank: Administration

Joined: 3/08/2007(UTC)
Posts: 3,300
Location: Fort Atkinson, WI

Those are all good points, jambalaya. I intend to dig into this issue as soon as time permits, and there will probably be one of two outcomes: 1) SQL CE performance is significantly improved 2) The Admin Guide will prominently mention the performance difference between the two databases.

Hopefully it's the first one...
Roger Martin
Creator and Lead Developer of Gallery Server Pro
Rss Feed  Atom Feed
Users browsing this topic
Guest
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.