Blog Bot

Somewhere below the regolith in the Linden Lab secret lunar base, some of the Lab’s top minds have been tackling performance issues in Second Life. The areas of focus range from infrastructure improvements to server-side texture compositing to object caching.

This year, Linden Lab is making the single largest capital investment in new server hardware upgrades in the history of the company. This new hardware will give residents better performance and more reliability. Additionally, we are converting from three co-locations to two co-locations. This will significantly reduce our inter-co-location latency and further enhance simulator performance.

The Shining project is the name given to a recent Lab effort to identify and measure delays and other problems with streaming avatars and objects and to propose and implement solutions. During the Shining project, several small improvements have been identified and deployed. The next small improvement from Shining to be deployed will be pre-rendering the starter Avatars to improve the new resident experience.

As a result of the Shining investigation, the project has been split into three larger performance projects: Project Sunshine, Object Caching, and HTTP Library.

All together, the hardware upgrades and the Shining projects should improve avatar and object streaming speeds significantly. Depending on your unique situation, your mileage will vary.




Project Sunshine (Server-side Baked Texture Generation & Storage)


Currently, all client viewers are responsible for compositing their own Avatar textures, then sending the results back to the Sim for other viewers to access. This method can lead to slowdowns and errors. The actual calculations for compositing textures are straightforward and not particularly time-consuming. However, in order for the viewer to do the calculations it first has to download a lot of individual assets from the Sim, and must then upload large results back to the Sim. This pushes a lot of bits through the Sim / Viewer connection, which can be slow and unreliable.

Depending on client hardware to do the compositing and uploading of the resulting baked textures can introduce erroneous results (like way too much pink). In order to handle these errors, a number of retry and fallback mechanisms have been put in place. This adds further load and overhead to the whole system.

Project Sunshine stands up a Texture Compositing server that is separate from the Sims servers. When a Viewer needs to render an Avatar, it sends a message to the Sim, which in turn sends a message to the Texture Compositing Server. The Texture Server then performs the texture compositing and sends the results back to the Viewer.

The Texture Server has access to a database of previously generated Avatar textures. If the Sim is asking for the baked textures for an Avatar that has been previously calculated, the Texture Server can pull the results out of the database instead of recalculating. Even if it’s not the same Avatar, but the set of clothes is the same, the Texture Server can use the previously calculated results.

An additional advantage of having a Texture Generation server is that the heavy HTTP communication about the individual assets that make up an Avatar will happen on reliable, fast internal network connections instead of external connections.

Object Caching & Interest Lists

Caching objects on a resident’s hard drive is intended to provide faster drawing of objects that are most likely to be redrawn later in a resident’s session or in the next session.

The current object cache implementation has room for improvement. Currently, the viewer only caches objects at the point the resident disconnects from a region and only objects currently in view are cached. The current caching communication between Sim and Viewer when a scene is initially being loaded is inefficient and resource intensive.

The new system is designed to make sure that a viewer never has to download object information that it already has and is still valid. This will reduce the load on the Sim and on the Sim / Viewer connection.

With the new system, when the Viewer connects to a region, the Sim first says to the Viewer, “Here’s a list of all my object groups by UUID.” The Viewer checks its cache and replies, “Here are the timestamps for the object groups that I found in my cache.” (If an object group is not found in the cache, the timestamp is 0.) The Sim checks the Viewer’s timestamps against its own data and responds, “Here’s a list of object groups in your cache that are still valid.” While the Viewer starts drawing valid objects from its cache, the Sim starts streaming the new object data that the Viewer needs, based on the particular Viewer’s object priorities.

HTTP Library

Top performance for the Shining initiatives depend on the speed and reliability of HTTP messages initiated by both the Viewers and the Sims. The third part of the Shining project, the new HTTP Library, implements modern best practices in messaging, connection management, and error recovery.



More...

pancake

Dear Blog Bot,
This all sounds very interesting and exciting, but I just wanted to say I love the intro here!

Great to see this humour from the Lab, keep it coming.

__________________

Eunoli Rain

The object caching list sounds extremely promising.

Kick Frenzy

I'm wondering if that's why they've been quiet lately.
Too busy gettin shit right!

Sounds tasty... can't wait to bite into it.

Lordfly Digeridoo

I'll believe it when I see it. Any of it.

__________________
http://www.lordfly.com/

Evola

If their investment and other measures actually improve things as promised, it will more than make up for my disappointment in their not taking part in SL9B this year(though I do wish they had at least thrown up a sim for Bear Island this year, after so many residents came together with sims and builds on such short notice. Maybe SL10B!)

Darien Caldwell

Seems a bit redundant for the viewer to send it's object groups first, then have the viewer say which it has and how old, and then for the sim to return which are valid back.

It would be more efficient for the sim to send what object groups it has and their age from the start, and then the viewer can immediately determine which cached objects are old and which are not.

The big advantage to their method I guess, is that the server does all the heavy decision making. I can only guess they wanted to keep the client as unloaded as possible. Considering how old some people's computers are, I guess wise.

As for the HTTP library, I really hope the fix the viewer. Ever since they put in the nwe CURL library, mesh uploading fails constantly for many people.

__________________
[H]arsh Styles

Evola

At least they're throwing money at it. I realize it takes a lot more than that, but seems to be a change from the cutbacks of previous years.

Lord

I'm hoping that all of these server side changes and improvements are in preparation for LL's own version of a Distributed Scene Graph so they can get more people on a region. That, in combination with all the mesh stuff, the two deformer methods, pathfinding, and advanced experience tools, would be a good way to compete with the Cloud Party's of the world.

hibit

LL has made A LOT of small performance improvements over the last year. At the SL9B event I was actually able to move around and load textures unlike last year. It is good to hear that LL is going to continue to persue those upgrades even further.

Jopsy Pendragon

Outstanding! Thanks for the forecast/update!

Hitomi Tiponi

This is excellent news, especially on object caching. I think LL have been looking at usage stats and have seen that most users just hang around in a few regions so object caching would be great for them. There also seems to be a major strategy of 'let our servers handle all the heavy stuff' which will put them in a better position for handling cloud-streaming and access via mobile devices.

What surprises me is that there are also lots of other improvements that are currently in development (such as improved sim crossings), that will impact most users, that they don't even bother mentioning. News from LL is always welcome but hiring better PR guys would also be more welcome - it's no good doing good things if no-one realises. Burying a technical explanation in a non-headline blog post means that many of the users will still think that LL is not doing anything and just letting the place die.

Naiki

LL are making the single biggest investment in its history?

OMG!!!!! SL IS FAILING! BAIL NOW WHILE YOU CAN STILL CASH OUT LINDENS!!!!!!! I bet they are going bust! This is so they can sell the hardware after they go bust!!!!! Bloody LL'ers!!!!! I am going to go and check a reliable source I know on the matter....

*Opens Hemlets blog*

Naiki

More seriously... This mean we are getting a better technical SL right?

Kick Frenzy

Doesn't the Viewer have to tell the Sim it's there in the first place?
If so, wouldn't it make sense for the Viewer to say, "I'm here! And here's what I got in my bags!"
Then the server is all, "OHAI! We got this shit already, as a matter of fact, I'm showing it to you right now! So, you only need to show us what other shit you brought."
Then the Viewer is like, "Sweet dude. Imma just start sending you this shit while you're showing me the shit you already got."

Instead of the Sim being like, "I'm going to continuously shout about what I got so anybody who comes around knows"
And then a Viewer coming in and saying, "Shit dude! Ok, so you got that... lemme show you what I got that you don't."
Then the Sim/Server responding, "Cool, I'm gonna start showing you this shit, while you start sending me copies of what I don't have."
Then the Viewer being like, "Cool here's what I have that you don't."

Ailsa Muliaina

I read "Project Shining" and wondered if Rod had hired Jack Nicholson to help motivate his staff... Guess they'll be worried if he suggests one of those motivational team holidays in a remote hotel!

Darien Caldwell

The viewer connecting to the sim is one of the first things that happens when you log in. That's all a sim needs to "know" a viewer is there.

To put in your terms, "hi, viewer here!", then "ok viewer, here's what I got". is what I was talking about. A terse "Viewer here" message would reduce the amount of data being sent by 1/3, but like I pointed out, it requires the viewer to do the heavy decision making.

But as I mentioned, I see the wisdom in what LL is doing. I was really just talking to myself in my post, I know most don't have any idea how things actually work.

__________________
[H]arsh Styles

Hitomi Tiponi

I know, I know.

Log in....magic happens....the world appears!

Naiki

Are you suggesting theres more to it than that?

Penny Patton

Although I do have substantially more faith in present day LL following through with promises like these, I have to admit when I read "Linden Lab's top minds", this was the first thing to pop into my head;


So THAT'S where the Nimble volumetric clouds feature went!

__________________
The Digital Pasture - SL from a cow's point of view. When the cow is also a designer.

infiniview Merit

This is Huge!

Back in '08 there was a thread in the old forums about a new company called OTOY OTOY
That caused some speculation about whether SL would ever go to server side rendering.
Back then it was still a far off dream. Now it is almost compulsory to the future of SL in my opinion. Multiple device access will be just one of the possibilities. This should be one of the biggest performance improvements in it's history in terms of the realm of the possible. The client side rendering paradigm at one time a neccessity is quickly becoming a relic of old technology. Problems making your graphics card work with SL? Not any more, things like this will not be SL's concern anymore.

As the cost of processing and memory and storage has decreased the possibilities have exponentialized. Hence the existence of the clouds and people playing crysis on their ipads.

Cutting down on all of the variables included with needing to be able to send so much data that needs to be processed on each different type of local machines should free LL up to make vast improvements on performance issues. Difficulties arising from logins to sim crossings to lag from dense objects should eventually become a thing of the past once they get the systems up and running to their potential. This could even finally allow for the upgrade of the SL standard avatar. Which I think is another thing that is crucial to the number of people willing to use SL.

__________________
Texture Arts Headquarters

http://slurl.com/secondlife/puea/126/10/60/

Blueman Steele

To have me make an SL Universe account and pop my head back into SL.

Of course all I can think of right now is...

Lloyd Newman

I see they've started hacking on the server code already...

__________________

Username

Hmm ok, so instead of the viewer requesting your avatars textures, baking them and sending ONE set of textures back, and often failing to do so for various reasons, we are now going to have a constant back and forth level of chatter that attempts to keep the viewer and servers object list in sync, and involves more load on both the server and viewer end to attempt to do so.

Yes. That'll work. :|

__________________

Cerise

The server already has to track all that, and simhosts all over the grid already serve the baked textures to the wearer and everybody else. The only big change is where the baking is done, and as a side effect the fragile temporary asset setup can be eliminated.