Miles By Motorcycle
established 11 years ago
You must be logged in and belong to this group to post to this forum.
Performance #bug I strated looking more deeply into the page performance and notice a couple of things. One is that FormVista is taking a ton of time to generate the pages, but it is also using a huge amount of memory to do it considering the page content. For example this was for the main "Forum" page:
formVista generated this page in:
6.5392
seconds
using
'12406132'
bytes of memory
formVista generated this page in:
6.5392
seconds
using
'12406132'
bytes of memory
6 seconds is extreme. 
I'm seeing 2.3 here and I think that's way too long.
I do understand the need to have something that makes a good first impression. This is why I waited so bloody long to release something, but even this I know is not nearly good enough. So there's this balance between getting something to be "right enough" and getting something out there. It took a lot for me to let this alpha quality release escape. A project of this size and complexity may simply be too much for one person to build effectively.
As far as the performance problems go, they are not due to server configuration. The performance problems are, however, solvable, at least for the small site case. What I've done is to create an XML based intermediate language I call FVML, a.k.a. the formVista markup language. It's a super-set of HTML that has some control structures and other features that allows me to encapsulate complicate functionality including Javascript features into single tags. For instance, the Like links you see are literally <like> tags in FVML. The same is true for all the dialogs.
This is /great/ for prototyping as I can experiment with layouts and functionality easily and quickly. In the early stages of a project, it allows me to figure out what works and what doesn't without incurring a huge development cost. In the early days, I also thought it might be a good way for getting relatively low cost HTML designers into programming. (I mean, seriously, what other system out there can let designers field dialogs, like buttons, tagging systems, etc without knowing programming?)
Frankly, it's the reason I've been able to get all this work done. However, it comes at a serious runtime cost. In previous builds, I wasn't doing nearly as much work as I am in this latest release. I've finally run into the performance wall that I always knew FVML would have.
I have a few options. The one I've been planning on for quite some time is to create an FVML to PHP compiler. Basically, to take the XML based FVML source files and turn them into straight PHP which can then be cached (instead of, as is the case now, interpreting the XML files at run time incurring the huge cost that represents.)
The other option is to rewrite the main forum components by hand.
There are a number of other optimizations I can make which will each provide incremental gains.
Ideally I would like to keep the flexibility of FVML while increasing the performance.
The other area that is causing a huge amount of overhead is the "social index" manager. This is the core part of the system that implements that tagging, embedding, mentioning, etc. There's this concept of "privacy" that I"ve built but largely turned off. If you attempt to view a thread, the system will determine whether or not you are allowed to based on the privacy setting. Most object in the system have a privacy setting of "inherit" which means the privacy manager will ask the object containing the object in question. In the case of a forum post (each and every one), it'll ask the containing thread, which is also at a privacy setting of "inherit" which causes it to ask the containing forum which then finally passes the buck up to the group. So there are four requests per object just to detemine whether or not you're allowed to see it. Now I do some caching, but it's clearly not fast enough.
I may scrap the privacy manager in it's entirety or more likely drastically simplify it.
The rest of the performance issues have to do with the general nature of PHP. For that, my plan is to eventually move over to the JIT (just in time) PHP compiler virtual machine Facebook has released called HipHopVM. This platform should result in a 3X to 5X performance improvement but represents some work.
And I should point out the server the software is currently running on is 2004 vintage.
I'm seeing 2.3 here and I think that's way too long. I do understand the need to have something that makes a good first impression. This is why I waited so bloody long to release something, but even this I know is not nearly good enough. So there's this balance between getting something to be "right enough" and getting something out there. It took a lot for me to let this alpha quality release escape. A project of this size and complexity may simply be too much for one person to build effectively.
As far as the performance problems go, they are not due to server configuration. The performance problems are, however, solvable, at least for the small site case. What I've done is to create an XML based intermediate language I call FVML, a.k.a. the formVista markup language. It's a super-set of HTML that has some control structures and other features that allows me to encapsulate complicate functionality including Javascript features into single tags. For instance, the Like links you see are literally <like> tags in FVML. The same is true for all the dialogs.
This is /great/ for prototyping as I can experiment with layouts and functionality easily and quickly. In the early stages of a project, it allows me to figure out what works and what doesn't without incurring a huge development cost. In the early days, I also thought it might be a good way for getting relatively low cost HTML designers into programming. (I mean, seriously, what other system out there can let designers field dialogs, like buttons, tagging systems, etc without knowing programming?)
Frankly, it's the reason I've been able to get all this work done. However, it comes at a serious runtime cost. In previous builds, I wasn't doing nearly as much work as I am in this latest release. I've finally run into the performance wall that I always knew FVML would have.
I have a few options. The one I've been planning on for quite some time is to create an FVML to PHP compiler. Basically, to take the XML based FVML source files and turn them into straight PHP which can then be cached (instead of, as is the case now, interpreting the XML files at run time incurring the huge cost that represents.)
The other option is to rewrite the main forum components by hand.
There are a number of other optimizations I can make which will each provide incremental gains.
Ideally I would like to keep the flexibility of FVML while increasing the performance.
The other area that is causing a huge amount of overhead is the "social index" manager. This is the core part of the system that implements that tagging, embedding, mentioning, etc. There's this concept of "privacy" that I"ve built but largely turned off. If you attempt to view a thread, the system will determine whether or not you are allowed to based on the privacy setting. Most object in the system have a privacy setting of "inherit" which means the privacy manager will ask the object containing the object in question. In the case of a forum post (each and every one), it'll ask the containing thread, which is also at a privacy setting of "inherit" which causes it to ask the containing forum which then finally passes the buck up to the group. So there are four requests per object just to detemine whether or not you're allowed to see it. Now I do some caching, but it's clearly not fast enough.
I may scrap the privacy manager in it's entirety or more likely drastically simplify it.
The rest of the performance issues have to do with the general nature of PHP. For that, my plan is to eventually move over to the JIT (just in time) PHP compiler virtual machine Facebook has released called HipHopVM. This platform should result in a 3X to 5X performance improvement but represents some work.
And I should point out the server the software is currently running on is 2004 vintage.
This morning, I'm seeing 4-8 second load times on links, which is about what I've been experiencing since the new build, with the exception of yesterday morning, where I was seeing load times as long as 15-20 seconds. 4-8 is not horrible, but maybe the easiest, quickest fix for now is to "throw some HW at it", to buy some time and breathing room, and to allow for making improvements on a somewhat more flexible schedule.
Yermo I have a couple of multi-cpu (Xeon) 1U Dell PowerEdge 860's here that aren't being used. Would be happy to donate them to the cause. They aren't state of the current art but they are very good performing dedicated servers.Yermo actually....let me know what OS you run on and I can get the server loaded ahead of time and bring it up tomorrow. I have on hand:Win Server 2008
Win Server 2012
CentOS 6.3
Centos 5
Ubuntu
Pretty much anything else from the last few years LoL
So I did a bunch of profiling of the code and I know where the majority of slowdown is coming from. To do it "right", where right means in a way that I"m not going to have to do over again in the near future, will take me about a week.
So what I'd like to propose is that I do that work in addition to fixing a few bugs and then let's see how the performance improves. I think I can get a bit better than a 50% reduction in load times without too much effort.
After that first 50%, additional performance gains will be more challenging. The biggest gain I can get after that would be to move over to what's called the HipHopVM platform released by Facebook. Most of Facebook is written in PHP as well. They have done a ground-up re-implementation of the language as a just-in-time compiler/virtual machine. They claim a 3X to 5X improvement in performance without many code changes.
Unfortunately, it only runs on 64 bit machines.
rshaugit's an incredibly generous offer my friend. If one of those machines is 64 bit, it would solve a big problem and enable us to have a much faster site sooner.
So what I'd like to propose is that I do that work in addition to fixing a few bugs and then let's see how the performance improves. I think I can get a bit better than a 50% reduction in load times without too much effort.
After that first 50%, additional performance gains will be more challenging. The biggest gain I can get after that would be to move over to what's called the HipHopVM platform released by Facebook. Most of Facebook is written in PHP as well. They have done a ground-up re-implementation of the language as a just-in-time compiler/virtual machine. They claim a 3X to 5X improvement in performance without many code changes.
Unfortunately, it only runs on 64 bit machines.
rshaugit's an incredibly generous offer my friend. If one of those machines is 64 bit, it would solve a big problem and enable us to have a much faster site sooner.Dude, you rock! Thanks.
Latest CentOS 64 bit, but of course I can install it as well.
This will become the dedicated M-BY-MC box.
What are the specs of the box? RAM? Drives?
I should mention that the conditions that it would live in aren't the best. I moved all the servers to the basement to reduce the noise in the house. Conditions down there aren't the best. I don't have a 19" rack anymore so it would end up free standing on some shelving.
Latest CentOS 64 bit, but of course I can install it as well.
This will become the dedicated M-BY-MC box.
What are the specs of the box? RAM? Drives?
I should mention that the conditions that it would live in aren't the best. I moved all the servers to the basement to reduce the noise in the house. Conditions down there aren't the best. I don't have a 19" rack anymore so it would end up free standing on some shelving.
Will load CentOS 6.3 64bit and make sure it's good.
Server Spec:
Dell PowerEdge 860
Multicore 3.0 GHz Pentium-D CPU
2 Gig ECC DDR2250G SATA Primary HDD
Plenty of room for additional drives. May have one more HDD kicking around.
Server Spec:
Dell PowerEdge 860
Multicore 3.0 GHz Pentium-D CPU
2 Gig ECC DDR2250G SATA Primary HDD
Plenty of room for additional drives. May have one more HDD kicking around.
I found the specs online. Looks like RAM has gotten wicked cheap. I can upgrade the thing to 8gigs pretty easily.
rshaugThat box sounds perfect. Thank you so much! That'll enable me to come up to speed on HipHopVM much sooner and since it'll be a dedicated box running a later Linux kernel, we should notice a performance boost just from hardware alone.
rshaugThat box sounds perfect. Thank you so much! That'll enable me to come up to speed on HipHopVM much sooner and since it'll be a dedicated box running a later Linux kernel, we should notice a performance boost just from hardware alone.