time to bleed by Joe Damato - Latest Comments in 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Anon — Tue, 20 Oct 2009 05:52:30 -0000

This is way over my head. :(

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

darekjones — Fri, 26 Jun 2009 14:18:08 -0000

Interesting post. I have stumbled and twittered this for my friends. Hope others find it as interesting as I did.

Darek from classifieds posting service

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Glenn Gillen — Thu, 14 May 2009 13:20:32 -0000

I'm always very impressed when people go to these lengths to track a problem. Very very nice work, and thanks!

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

radarek — Fri, 01 May 2009 10:15:10 -0000

Awesome post! I really like such a write-ups!

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

raggi — Thu, 30 Apr 2009 10:50:24 -0000

Just for extreme clarity:

This patch affects ruby code running on EventMachine with the epoll reactor.

require 'eventmachine'; EM.epoll; EM.run {} # 800k stack on linux on old EM

There may be a similar patch coming for the kqueue reactor, but that is less affected (40k vs. 800k).

require 'eventmachine'; EM.kqueue; EM.run {} # 40k stack on OSX

The issue does not exist in any extreme with the select reactor.

require 'eventmachine'; EM.run {}

You can see the effects of the issue most drastically when using EventMachines defer method, because it spawns up 20 threads, each of which will carry a heavy stack.

I have recently done some research on performance metrics of ruby's green thread implementation. Whilst there is a much more significant impact to ruby's performance when the stack size is large, the specific impact of the number of threads does not become noticeable until high numbers of threads (two orders of magnitude and above (100+ threads)). The choice of 20 threads seems sensible, given this research.

Each spawned thread carries a copy of the stack, though, so if you spawn threads 'later' in your execution stack (like Thin does), then you will see extra performance degradation from these issues. As such the real highlight comes from an app like this:

require 'eventmachine'; EM.epoll
require 'sinatra'; def (Sinatra::Application).deferred?(env); true; end
get '/' do escape_html Thread.current.inspect end

Some of the performance impact can be slightly subdued by doing the following:

EM.defer {}

Prior to thin loading, as the 20 threads for doing work in will be sitting on a significantly smaller stack. The main eventmachine stack would still be very large though, so this 'quick fix' isn't really a fix at all.

Some of the general impacts of stack size on ruby performance are demonstrated here:
http://pastie.textmate.org/...

And can be generated like so:
http://gist.github.com/104213

Maybe I'll collaborate or separately post some ruby programmer oriented explanations of these posts soon. :)

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Kate — Thu, 30 Apr 2009 02:11:01 -0000

Cool story, bro.

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Duncan — Thu, 30 Apr 2009 02:05:50 -0000

I spend most of my time programming in higher level language, but have always wanted to learn more about lower level tools like gdb, nm, objdump, etc. What resources would you recommend for learning these tools? I have tried reading guides online, but I have always found them lacking, particularly on how one goes about creating a mental model of the running system from all the data that can be seen with the tools.

Also, how does one go about honing their assembly skills? Other than the architecture class in college where I learned MIPS, I never had the opportunity to write any assembly, not in MIPS and not in x86. Do you have any recommendations on that too?

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Kraln — Thu, 30 Apr 2009 01:41:11 -0000

Another ruby superhero saves the day. Why not use something where you don't have to debug language internals? Sounds awful productive to me.

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Joe Damato (ice799) — Wed, 29 Apr 2009 18:33:53 -0000

Thanks for reading guys!

Don't forget to subscribe, lots of cool posts in the queue =]

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Joe Damato (ice799) — Wed, 29 Apr 2009 18:19:28 -0000

Thanks! If you like slightly-larger changes with big results you should check out my other post about a fibers implementation for Ruby 1.8.7: http://timetobleed.com/fibe...

Thanks for reading and don't forget to subscribe =]

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Brian Morearty — Wed, 29 Apr 2009 17:30:51 -0000

Nice work. I love small changes with big results. And thanks for the quick intro to gdb.

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Joe Damato (ice799) — Wed, 29 Apr 2009 17:15:44 -0000

Why doesn't it seem likely? I've fixed a couple bugs in the Ruby threading implementation and implemented Fibers for Ruby 1.8.7. So I've messed with the threading implementation quite a bit =]

Anyway, please see:

http://github.com/ice799/ma...

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

oldmoe — Wed, 29 Apr 2009 17:06:30 -0000

Great post and a really nice job. I suffered from this and I was clueless but now I know better :)

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Jon — Wed, 29 Apr 2009 16:59:24 -0000

I would check to make sure that thread-switch will cause a full memcpy of the whole stack. It doesn't seem likely.

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

James — Wed, 29 Apr 2009 16:17:50 -0000

Sounds great. Hit me up. lastname.firstname@gmail.com

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Joe Damato (ice799) — Wed, 29 Apr 2009 16:09:47 -0000

We can chat about this via e-mail if you like - this is difficult via blog comments.

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

James — Wed, 29 Apr 2009 16:03:38 -0000

> However, stack allocating stuff is useful and *can* be done safely, even for large objects.

I'm not really concerned about memory use here. (Although, haha, I think we can both agree that you should attempt to conserve memory.) I'm concerned about virtual address space use.

Do you know of a threading implementation that does do this safely? As far as I know (and again, I'm no expert), pthreads does not solve this problem. If a function that uses a lot of memory in a frame can be called from any thread, all stacks have to be spaced at a distance wide enough in virtual memory to accomodate that frame, even if it's almost never called and never simultaneously.

I'm thinking of a large thread pool executing a variety of periodic tasks, among which is one that uses the large function or functions in question. In such a pool pthread_attr_getstacksize for each thread in the pool must be at least as large as the largest possible stack generated by a call from that pool, even if the average stack is very small. And that stack size affects the overall usage of virtual space by pthread stacks as a whole.

You can make it better by guaranteeing that the big function is only called from one thread, then setting the stack size of that thread much larger and setting all the rest small. But in my inexperienced knowledge there is no thread implementation that will handle this detail automatically. Please let me know if I'm wrong; I'd love to find that this is something I don't have to worry about.

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

terry — Wed, 29 Apr 2009 15:44:05 -0000

I read the explanation and it makes perfect sense now. I haven't coded much in C++ since college, mostly .NET apps nowadays, so my analysis was admittedly going slower than I would've liked. Anyway thanks for the explanation.

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Joe Damato (ice799) — Wed, 29 Apr 2009 15:43:05 -0000

This speeds up Ruby's GC -if- you are using EventMachine. The patch fixed a bug in EventMachine which was slowing down Ruby's GC.

Does that clear it up?

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

mark — Wed, 29 Apr 2009 15:35:48 -0000

I am not sure if i understood this. Does this mean that this patches speeds up ruby's GC collection?

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Ken Pratt — Wed, 29 Apr 2009 15:32:59 -0000

Ah, yes, that makes sense, thanks ice.

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Joe Damato (ice799) — Wed, 29 Apr 2009 15:22:59 -0000

Let me know if the explanation I gave Ken clears it up. I'll add a little more to the article itself in the "patch" section.

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Joe Damato (ice799) — Wed, 29 Apr 2009 15:21:44 -0000

AH, Ok.

Moving it into the class definition causes the object to be allocated on the heap when an object of the class is created with new.

The answer to your question is that it doesn't decrease the memory usage. It decreases the size of the stack because it moves the object over to the heap.

Ruby scans the process stack, not the process heap (it scans its own internal heap which is not the same as the process heap).

Does that make more sense? I'll add a little bit of info to the article in the "Patch" section above.

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

terry — Wed, 29 Apr 2009 15:20:00 -0000

@ice799 You are correct and my mistake, I missed the patch link when I originally looked over the article. However, like Ken I'm curious about how the patch works but now that I'm aware of the link I'll do some detective work of my own. :-)

Re: 6 Line EventMachine Bugfix = 2x faster GC, +1300% requests/sec

Joe Damato (ice799) — Wed, 29 Apr 2009 15:19:17 -0000

OK, so I'll agree that it isn't as simple as I said it was. I should be more careful; I didn't think anyone would actually read this in the first place.

It's only a performance issue on certain threading implementations. It could be a portability issue, though.

It depends on what your invariant is - if you just need to stack allocate a huge amount to use and toss out once your function returns, I don't see that necessarily being problematic. If your invariant is that that stack frame doesn't return for a while (for some definition of "a while") it could be a problem.

If your argument is simply "using a lot of memory is bad" then sure I agree with that argument.

However, stack allocating stuff is useful and *can* be done safely, even for large objects.