| 00:00:05 | *** njero has joined #openmrs |
| 00:00:10 | *** ChanServ sets mode: +v njero |
| 00:18:59 | *** njero has quit IRC |
| 00:19:07 | *** njero has joined #openmrs |
| 00:19:07 | *** ChanServ sets mode: +v njero |
| 00:22:11 | *** njero has quit IRC |
| 00:22:47 | *** njero has joined #openmrs |
| 00:22:47 | *** ChanServ sets mode: +v njero |
| 01:08:23 | *** Keelhaul has quit IRC |
| 05:48:47 | *** atomicturtle has joined #openmrs |
| 07:00:44 | *** bwolfe has joined #openmrs |
| 07:00:44 | *** ChanServ sets mode: +o bwolfe |
| 07:42:25 | <bwolfe> atomicturtle: Error: The license key is invalid. In order to use the control panel, please, obtain and install a new fully featured license key. |
| 07:42:26 | <bwolfe> Your license key has expired. To continue using your control panel, purchase a non-expiring commercial license. |
| 07:42:27 | <bwolfe> :-( |
| 07:42:47 | <atomicturtle> "whats openmrs doing trying to see how un memory effeicent they can make that application?" from our hoster |
| 07:44:36 | <atomicturtle> he was saying openmrs uses more memory than all the other VPS's combined :P |
| 07:46:43 | <bwolfe> ha |
| 07:46:53 | *** bmckown has joined #openmrs |
| 07:46:53 | *** ChanServ sets mode: +o bmckown |
| 07:46:54 | <bwolfe> looks like we're using 1.5gb |
| 07:47:12 | <bwolfe> cruisecontrol is given spiked memory up to 256 |
| 07:47:19 | <bwolfe> tomcat is given 192mb |
| 07:47:28 | <bwolfe> I don't know what else is using it |
| 07:47:42 | <atomicturtle> yeah 1.5 is crazy for a VPS |
| 07:48:17 | <bwolfe> supybot uses 20mb |
| 07:48:57 | <bwolfe> spamd uses 80+...and we aren't using mail facilities on the box |
| 07:49:15 | <bwolfe> the others are http threads |
| 07:49:38 | <bwolfe> I upped the number of threads apache was allowed because we get long pauses at page load time occasionally |
| 07:50:03 | <bwolfe> it seemed to help somewhat, but we still get them |
| 07:50:10 | <atomicturtle> I forgot to check if query caching is on in mysql |
| 07:50:23 | <bwolfe> svn, trac, wiki...all have some delays |
| 07:50:30 | <OpenMRSBot> Recent updates in the world of openmrs: OpenMRS Changesets: Changeset [3547]: in patientmatching module, added logging to analyzer classes and added XML ⦠<http://dev.openmrs.org/changeset/3547> || OpenMRS Changesets: Changeset [3546]: -- chica * renamed consumeCheckbox rule to more accurately reflect what ⦠<http://dev.openmrs.org/changeset/3546> |
| 07:50:40 | <bwolfe> I turned on the slow query log for mysql to see if that was it...there don't seem to be a lot of repeat offenders |
| 07:52:44 | <bwolfe> yeah, looks like they are httpd threads that are hanging around hogging memory |
| 07:53:16 | <bwolfe> feel free to take a look at the mods I made to the conf file and tweak them |
| 07:56:10 | <atomicturtle> oh yeah, this will make a huge difference |
| 07:56:30 | <atomicturtle> the query cache wasnt on |
| 07:56:52 | <atomicturtle> wow |
| 07:57:33 | <atomicturtle> yeah httpd is using a lot of memory |
| 07:58:08 | <atomicturtle> its 621m virtual and 177m reserved |
| 07:59:36 | <atomicturtle> I see httpd restarting on the box frequently, is that you doing it manually? or is dying on its own |
| 08:00:59 | <bwolfe> I was working on trac stuff there other day so I was reloading the dev apache |
| 08:01:06 | <atomicturtle> ok |
| 08:01:11 | <bwolfe> but I haven't restarted apache as a whole in a while |
| 08:01:24 | <atomicturtle> ah, we might have something with a memory leak then |
| 08:03:37 | <atomicturtle> yeah one apache thread just shot up to 129m |
| 08:04:16 | <atomicturtle> the good news is that apache will restart, so its somewhat self healing |
| 08:04:42 | <atomicturtle> do you know if there were any changes made to mod_rewrite rules recently? That would be the first culprit |
| 08:05:47 | <bwolfe> hmm |
| 08:06:02 | <bwolfe> I can't think of any...but I know we employ them |
| 08:10:50 | <bwolfe> atomicturtle: found one circular redirect match in dev.openmrs.org/ confs |
| 08:11:24 | <atomicturtle> yeah that is exactly the kind of thing that would do it |
| 08:12:40 | <bwolfe> ok, would reloading dev do it or should we restart apache altogether? |
| 08:13:01 | <atomicturtle> restart apache, I can see 2 threads grabing about 150-160m right now |
| 08:13:46 | <bwolfe> ok, sent the restart signal |
| 08:13:52 | <bwolfe> to apachectl |
| 08:14:15 | <atomicturtle> to keep an eye on it, watch the RES (reserved) column in top |
| 08:14:25 | <atomicturtle> sort by memory use, hit m |
| 08:14:39 | <atomicturtle> pardon M |
| 08:15:01 | <atomicturtle> you'll see they're at 54m right now |
| 08:15:22 | <bwolfe> right |
| 08:16:03 | <atomicturtle> I have to head to the office, bbiab |
| 08:16:21 | <bwolfe> still getting spikes in memory |
| 08:16:27 | <atomicturtle> yeah see one just hit 135 |
| 08:16:28 | <bwolfe> not there are some up to 130 again |
| 08:16:37 | *** atomicturtle has quit IRC |
| 09:02:04 | *** atomicturtle has joined #openmrs |
| 09:22:26 | <OpenMRSBot> Recent updates in the world of openmrs: OpenMRS Changesets: Changeset [3549]: report-api-refactoring branch: Changed global_property.property_value to ⦠<http://dev.openmrs.org/changeset/3549> || OpenMRS Changesets: Changeset [3548]: Adding jstl jar to versionedfileupload module to give it autonomy <http://dev.openmrs.org/changeset/3548> |
| 09:31:36 | <bwolfe> atomicturtle: I dropped the min/max number of httpd threads down to what they were before (5, 10) |
| 09:31:57 | <bwolfe> we're not using as much memory, obviously, but we still have delays in response |
| 09:32:16 | <bwolfe> is there a tool you know of that shows what each httpd is doing ? |
| 09:32:22 | <bwolfe> or what it has processed? |
| 09:32:30 | <atomicturtle> I think theres a debug mode |
| 09:32:55 | <atomicturtle> theres also an apache_top mod |
| 09:35:07 | <bwolfe> yeah, I found apachetop, but its not very useful |
| 09:35:14 | <bwolfe> partly because it can only act on one log file |
| 09:35:26 | <atomicturtle> there was some module I ran across |
| 09:35:36 | <atomicturtle> mod_top |
| 09:35:36 | <bwolfe> and ours are broken up in openmrs/dev.openmrs/amrs.iukenya |
| 09:35:48 | <bwolfe> an actual apache mod? |
| 09:35:57 | <atomicturtle> http://www.mod-top.org/ |
| 09:37:54 | <bwolfe> interesting |
| 09:45:12 | <atomicturtle> I had that on my todo list for rpm'ing up |
| 09:51:02 | *** Echidna has quit IRC |
| 09:51:10 | <bwolfe> heh, was wondering if my search skills were just bad or if it didn't exist yet |
| 09:51:19 | <bwolfe> I guess it doesn't exist as an rpm yet :-p |
| 09:52:17 | <atomicturtle> yeah, havent had the time |
| 09:52:23 | <atomicturtle> been working on ASL |
| 09:56:16 | *** Echidna has joined #openmrs |
| 09:57:48 | *** njero has quit IRC |
| 10:01:53 | *** Keelhaul has joined #openmrs |
| 10:27:33 | *** ChanServ sets mode: +v atomicturtle |
| 11:37:44 | *** pombreda has quit IRC |
| 11:38:50 | *** pombreda has joined #openmrs |
| 12:25:18 | <OpenMRSBot> Recent updates in the world of openmrs: OpenMRS Changesets: Changeset [3552]: -- chica * made changes to reflect changes made in dss 2.09 <http://dev.openmrs.org/changeset/3552> || OpenMRS Changesets: Changeset [3551]: -- atd * made changes to accommodate changes made in dss 2.09 <http://dev.openmrs.org/changeset/3551> || OpenMRS Changesets: Changeset [3550]: -- dss * added token_name column to dss_rule table * removed purging of ⦠<http://dev.openmrs.org/changeset/3550> |
| 13:33:00 | *** bwolfe has quit IRC |
| 13:38:47 | *** bwolfe has joined #openmrs |
| 13:38:47 | *** ChanServ sets mode: +o bwolfe |
| 13:41:20 | *** bmckown_ has joined #openmrs |
| 13:41:23 | *** bmckown has quit IRC |
| 14:05:06 | *** bmckown__ has joined #openmrs |
| 14:07:22 | *** bmckown_ has quit IRC |
| 14:43:51 | *** pombreda has quit IRC |
| 15:33:32 | *** Keelhaul has quit IRC |
| 15:40:47 | *** bwolfe has quit IRC |
| 15:42:42 | *** pombreda has joined #openmrs |
| 15:56:20 | *** bmckown__ has quit IRC |
| 17:41:29 | *** Keelhaul has joined #openmrs |
| 18:04:51 | *** atomicturtle has quit IRC |
| 19:02:47 | <OpenMRSBot> Recent updates in the world of openmrs: OpenMRS Changesets: Changeset [3553]: report_api_refactoring: Fixing some compile errors. Not particularly well. <http://dev.openmrs.org/changeset/3553> |
| 19:49:21 | *** bwolfe has joined #openmrs |
| 19:49:21 | *** ChanServ sets mode: +o bwolfe |
| 20:11:24 | *** bwolfe has quit IRC |
| 20:13:22 | *** bwolfe has joined #openmrs |
| 20:13:22 | *** ChanServ sets mode: +o bwolfe |
| 20:26:32 | *** atomicturtle has joined #openmrs |
| 21:08:58 | *** bwolfe has quit IRC |
| 21:57:09 | *** pombreda has quit IRC |
| 22:16:32 | *** pombreda has joined #openmrs |