Get the Drivers

Since my last installation, things have changed. a bit. Many different distros are now supported, so we don’t have to shoehorn the RPMs onto the system with Alien. Here are the packages I grabbed off their site:

iscan-data_1.0.1-1_all.deb
iscan_2.25.0-1.ltdl7_i386.deb
iscan-plugin-gt-f700_2.1.0-3_i386.deb


Once you have them, you can install them along with sane and sane utils:

sudo dpkg -i iscan_2.25.0-1.ltdl7_i386.deb iscan-data_1.0.1-1_all.deb iscan-plugin-gt-f700_2.1.0-3_i386.deb
sudo apt-get install sane sane-utils


Edit the Configs

Edit /etc/sane.d/dll.conf, adding epkowa to the list.

Create /etc/udev/rules.d/45-libsane.rules, adding the following:

# Epson Perfection v350
SYSFS{idVendor}=="04b8", SYSFS{idProduct}=="012f", MODE="664", GROUP="scanner"

I use the group “scanner” above; You could go the easy route and use a group that you’re already part of, such as “users” or “admin”, otherwise you’ll have to create the scanner group, add yourself to it, then log out and log back in.

Finishing Up

Afterwards, connect your scanner and run the following command:

scanimage -L


The scanner should fire up now, which means you can use your newly installed iscan app to scan pictures.

Disclaimer- your laptop may catch on fire and explode (or worse) if you attempt this… or something.

We’ll be presuming that you’ve already resized your windows partition and have both a working Windows and Linux partition.

In Windows

Log into XP, grab MergeIDE.zip from Virtualbox’s site, extract and run it. It should be a quick flash and be done. (Note: I am not 100% sure this step is needed)

Create a new hardware profile and name it virtualbox. Make sure to set it as a choice during boot. Try rebooting into native windows once to ensure that it does offer you profile options.

In Linux

You’ll need the following packages installed (May differ for non-ubuntu systems):
mbr, virtualbox-ose, virtualbox-ose-qt

Create a stand-alone mbr file to use for booting (yes, you need the force flag):

install-mbr ~/.VirtualBox/WindowsXP.mbr --force

We’re presuming that your windows partition is /dev/sda1. In the below command, we are defining

• a vmdk file (WindowsXP.vmdk)
• which raw disk to read (/dev/sda)
• which partition (1)
• the new MBR file we just created

VBoxManage internalcommands createrawvmdk -filename ~/.VirtualBox/WindowsXP.vmdk -rawdisk /dev/sda -partitions 1 -mbr ~/.VirtualBox/WindowsXP.mbr -relative -register

Note that you’ll need read/write access to that drive as your user, so you may want to figure out a cleaner/securer way to implement this, rather than adding your user to the disk group (which is very dumb and insecure). I would, but it’s working and I have more important things to do at the moment.

Another issue- apparently thinkpads report the drive heads and cylinders oddly (T410 for me and T60p in article), so we have to add some vmdk settings before virtualbox creates them incorrectly. Open ~/.VirtualBox/WindowsXP.vmdk and add the following at the bottom:

ddb.geometry.biosCylinders="1024"
ddb.geometry.biosHeads="240"
ddb.geometry.biosSectors="63"

The biosHeads appears to be the magic value- it seems to work if it’s set to 240, but the default is 255 (which fails).

Once you add those, start up virtualbox and check the virtual media manager, your new vmdk should be listed there. Once it’s confirmed, create a new virtual machine. Rather than creating a disk, select your vmdk as an existing disk.

After you finish, go the the VM settings->system and make sure the motherboard tab as io-apic  enabled (I also had PAE/NX enabled under processor and VT-x enabled under Acceleration).

Start the VM

There are several errors that could pop up. I’m sure there are plenty more that I stumbled across, but these were the two big ones:

• a disk read error occurred, press ctrl+alt+del to restart - Caused by incorrect biosHeads- check and make sure it’s set to 240 (this was the fix for me, results may vary).
• Complaint about kvm/vmx – Virtualbox does not like kvm. Uninstall qemu-kvm.

If things go well, it should flicker mbr in the corner, then go to the hardware profile selection screen. Select the virtualbox profile, and continue, then log in.

What follows is a half-hour of installing generic drivers and dealing with hardware specific auto start apps complaining that they won’t work on this installation. Windows will warn that the new drivers are not blessed, so be forewarned.

Once completed, at the top of the VM windows select Devices-> Install Guest Additions. This will download and mount an ISO, and windows will pop open a folder with the addition executables. Select the one best for you and run the installer. It will prompt you for video and mouse drivers (and trust me, you want them).

The final step is to shut down the windows VM, then reboot into the native windows partition to make sure it still works.  I did receive a few blue-screens before logging in at the beginning, but they appeared random and haven’t happened since.

And that’s all there is to it- simple, eh? Your windows partition should now run in native mode and vm mode.

Why you Monitor

Before you set up a single monitor, you have to ask yourself, “what is the goal?” After all, why are you even setting something up? Here a few common reasons for configuring monitors:

1. Notification: Warning of an issue that requires intervention. What most people think of when you say “Monitoring”.
2. Reactionary: Automatic actions are taken when certain criteria are met. If common countermeasures are automated, you’ll have less to handle manually.
3. Informational: System status and historical trending allows you to show business customers that production “isn’t always down.” In reality,  you may have 99% uptime, and often downtime is due to requested deployments. Statistical information can also be used for capacity planning.

Mentally dividing your monitors into groups will help you calculate which monitors require involvement. It’s not uncommon to have several thousand monitors at any given time, so it’s important not to assign critical importance to all of them. A wise man once said “When all alerts are critical, none of them are.”

When you should NOT Notify

Some monitors may have thresholds set which check for certain conditions; when those conditions are met, you may want to send some type of alert to an administrator. There are two types of notifications – Active and Passive:

• Active Notification: Immediate Action is Required: “Site is Down!” A phone call, page, or IM may be used to contact someone. Direct action expected.
• Passive Notification: Informational Purposes only:  “JVM Memory usage is high.” Information is logged, and perhaps an email is sent. No direct action is expected.

It’s easy to become addicted to passive notifications – but remember, data overload can mask important information. It becomes habit to ignore notifications if they are unimportant. The question then is not so much “when should you notify,” but “when shouldn’t you?” What it really boils down to is “Can/should I do anything about it right now?”

• Non-critical (disk space creeps above  90%  on /var on a dev server at 2am on a Saturday after several months of growth).
• Nothing Systemic is wrong (admins can’t fix “low sales”).
• 3rd party system, such as a geocoding webservice, is down.
• Will resolve shortly, such as a backup server pegging the CPU during midnight backups.

Some of these alerts can be avoided by setting a correct monitoring window (ignore CPU during the backup window). Others simply can’t be addressed by administrators, although you may want to send informational emails to other members of the company (those managing 3rd party SLAs or responsible for tracking online sales).  The next step after getting an alert is figure out what to do about it.

Reacting Properly

When a notification is sent out, there should be a definitive action that you can take. Think about why you were notified. There are a few rules to keep in mind when something goes wrong.

1. Don’t Panic. When 100 alarms go off, your first instinct is to panic. Before you act, take a breath. Spend a moment to get your bearings, and calm yourself. The worst possible thing you can do is flail. Randomly making changes without rhyme or reason and restarting services can do more harm than good and may make the situation worse.
2. Identify Obvious Patterns. What is the commonality? If a central system goes down, you may see many similar alerts. Dependencies can help immensely, masking redundant alerts. A single database failure could take down a dozen sites. Which is better: getting a single alert that the database is down, or 250 alerts that various sites are down and one database notification in the middle? While 250 alerts may impress the gravity of the situation upon you, it may instill panic and anxiety, which leads to flailing.
3. Get things up and running as quickly as possible. Root-cause analysis can be tedious, time consuming, and occasionally inconclusive. If you have a major system outage, don’t worry about doing root-cause analysis on the spot.  Do what you need to in order to get things up and running – you can search the logs later. If the problem is recurring, you’ll get another chance to investigate later.
4. Communicate with Stakeholders. The business units don’t need to know the details, but they do need to know that there is an outage and that it’s being addressed. If the situation is not quickly resolved, give them status reports. Be warned – any details you reveal will be warped and held against you. I’ve learned this one many times. People have a tendency to blame what they don’t understand. “Site is down? It must be a witch!” At my last job we had a “jump to conclusions” board which had our favorite scapegoats – load balancer, connection pool, Endeca, etc. Everyone is guilty of it – Business, devs, sysops, QA, etc. Even a one-time problem that has been resolved will be brought back up, even if it’s only tangentially related. Communicating too much information creates future scapegoats.
5. Contact Domain Experts. If your java site is crashing and you’re not a java developer, get a java developer involved. If your DNS server falls down and the fix isn’t obvious, contact your DNS administrator. Expert eyes on the problem may resolve it quicker. Group chat is crucial for sharing information and talking out theories. Someone familiar with the code will know what the error messages mean.
6. Fix the Problem. It should go without saying that if you find the problem, you should make every effort to resolve it. Workarounds are fine, just don’t let that band-aid become permanent. What often happens is a workaround is put in place; the alert clears and management no longer feels the pain, so they ignore the problem without putting forth the effort to fix the issue. When the next issue appears, a new fix is layered on the old. Band-aid is layered on band-aid. Eventually you’ll need to pull those band-aids off; and the more there are, the more painful it will be.

How Much is Too Much?

Most administrators prefer to be proactive rather than reactive, resolving issues before they become a problem. Proper monitoring can be a great asset, but if you’re not careful it can cause problems. For example, at a previous job we had a load balancer, apache instances and tomcat instances set up for each site. Each site had the following:

In (Sitescope) legacy monitoring system:

• Health check on load balancer URL

In Nagios:

• Health check on Apache instances
• Health check on Tomcat instances
• Health check on Load balancer URL

In Apache:

• Health check on tomcat instances

In Load balancer:

• Health check on Apache instances
• Health check on Tomcat instances

Individually, these don’t seem that bad. If an apache instance goes down *of course* the load balancer needs to know so it won’t send traffic to that instance. The same with Apache watching Tomcat. The problem was the frequency of the checks; the load balancer was checking each monitor every five seconds. When a poorly load-tested site update was released, certain pages took 7 seconds to load. Things quickly went downhill as threads and processes backed up, crashing the site.

Balancing responsiveness with common sense is essential. Having a monitor check every minute won’t change the fact that it will take an admin 20 minutes to get to a computer, boot up, log into the VPN, and identify the issue. Don’t add to the problem by DOS’ing your applications.

Making Contact

One mistake I’ve seen is using email as a reliable and immediate method of contact, often expecting a quick response. My favorite is when someone sends you and email, then walks down to your desk immediately after and asks “did you see my email?” You check and see it was sent literally less than two minutes ago. People don’t reliably check their email. Admins especially don’t due to the sheer volume we receive.

Email has it’s uses, but active contact in an emergency situation is not one of them. Personally, I only check my email when I think about it, which may mean large delays between when the message is sent and received. Couple that with spam filters, firewalls, solar flares and the 500 other unread messages and email becomes a less-than-reliable medium for emergency notifications (even during business hours).

Paging (or SMS)  is preferable if you expect a quick response, although it is far from perfect. Just like email, SMS messages can be lost in the ether, however recipients usually have their phone alert them when a message comes in since it happens far less often than an email drops into the inbox. That said, every alert should not be sent as a page, or apathy will quickly sink in. The escalation path should look something like this (although all steps are not needed):

• Front-end web interface alert: User would have to actively be browsing to see the status change. Usually the first clue something is wrong and shows the most recent status changes on a dashboard.
• Email Alert: User would have to be actively checking their email. Usually sent when something is first confirmed down.
• Instant Message: User would have to be at a computer and logged into IM to receive the alert. Rarely used, but an option during business hours.
• Page/SMS: Reserved for emergencies. This means there is trouble.
• Phonecall: Only used if Admin does not respond to the previous contact attempts. Usually performed by an irate manager.

If you’re lucky enough to have a 24×7 call center/ help desk, they can also be leveraged to resolve issues before a system administrator is needed. If recurring patterns start to emerge,  automation can be used to deal with the problem (or better yet you can fix the underlying issue). Sadly, many issues can’t be automated away or solved by a call-center staffer pressing a button. A real admin will eventually need to be contacted.

I don’t want to dig too deeply into on-call rotations, but an effort should be made to balance off-hours support with a personal life.  Being on-call means no theaters, fancy dinners, or quality time with the family. Without balance, burn out will ensue.

Afflictions

System monitoring often brings out odd behavior in even the most steadfast of administrators. Some behaviors are relatively benign, while others can cause severe problems down the road. Identifying these behaviors before they cause a problem is just as important as having good monitors.

• Data Addiction: Knowledge is power, but do not mistake information with knowledge. It’s possible to have 700 alerts, and not one of them identify the underlying issue. One of my least favorite phrases is “Can we put a monitor on that?” It’s often uttered right after a one-off failure; the type of thing that fails once, and once fixed will never cause a problem again. An example of this is a new server, where apache was not configured to restart after a reboot. When the server is restarted, you quickly find apache is down, start it, configure it to auto-start, and move on. There is already a monitor on the websites hosted by that apache instance as well as a monitor on how many apache threads are currently running; What purpose would another monitor serve? How often would it run? This is a prime example of how a data addict can spin out of control – too many useless monitors will mask a more important issue.
• Over Automation: Automation is a wonderful thing, however, it’s possible to have too much of a good thing. In one instance, there was a coldfusion server which would crash often. Rather than trace out the root cause, restarts were automated, then forgotten about. A few years later, it was found that the coldfusion servers were restarting every twenty minutes, and no one knew about it – no one except the users. If it takes 20 seconds to restart, and that’s 26280 twenty-second interrupts over the course of a year – that can translate into a bad user experience and loss of sales. Make sure that automation is audited and verifiable, and doesn’t cause more trouble than it prevents.
• Over Communication: While it is important to communicate with stakeholders, it is possible to over communicate. Stakeholders don’t need to know that there are 130 defunct apache processes caused by a combination of a bug in mod_jk and the threading configuration in JBoss – all they need is “Site availability is intermittent – we’ve located the root cause and are working on a solution. More information to follow.” Details aren’t needed. Likewise, not every single person should be notified when an alert goes off – does your backup administrator need to know when a web server goes down? No. Does the DBA need to know when an SSL cert is about to expire? No. Tailor the messages to the correct audience. Most monitoring systems allow you to configure contact groups – use them.
• Complexification: There are dozens of relationships between services, hosts, hostgroups, contacts, servicegroups, notification windows, dependencies, parents, etc. Try as you might, it’s usually impossible to perfectly model every relationship. Don’t become distracted by perfecting the configuration – focus on maintainability, scalability and accuracy. If you can’t add new systems and monitors, your configuration is too complex.
• Reporting vs Monitoring: Reports are the more successful cousin of Alerts. They may superficially appear similar, but serve entirely different purposes. Monitors should only be used to track and trend data and to notify if there is a problem, whereas reports take the collected data and massage it into an aggregated format. Monitors shouldn’t send out scheduled alerts. They can collect data, but they shouldn’t be used to present it to users. You’d be surprised how often someone asks for a monitor to send a nightly report. That slippery slope will turn your monitoring system into crystal reports.
• False Positives: False positives are the scourge of the monitoring world. There are many causes, but the reaction is always the same – start to investigate, realize that it’s a false positive, and lose interest, knowing that nothing is broken. The problem is that a false positive leads to lazy behavior – if you’re pretty sure it’s a false positive, you don’t bother looking into it, figuring it will clear on it’s own. This trains people to have a “wait and see” mentality when alerts go off, causing unneeded delays when a major issue appears.
• Apathy: It’s 2am on a Saturday, and you get paged that the CPU on a utility server is pegged. Without looking, you know that it’s the backup process copying the home directories, so you ignore it. The following Monday at 10am the QA JBoss instance stops responding. You know that it will clear within minutes because the QA team always rebuilds the QA instance Monday morning. When you get monitors constantly failing and recovering on their own, you start to ignore the pages that come in because you know they’re unimportant. It’s only a matter of time before you miss something important. If you have a situation that promotes apathy towards alerts, resolve it before something important is missed.

Don’t be [A]pathetic

I mentioned apathy above, but there’s a bit more to it – it’s not just admins that become apathetic.  If an issue is identified, action must be taken to correct it. The coldfusion example mentioned above is a great example of  company apathy – failure of the business unit to prioritize it and failure of IT to push back hard enough.  A former manager once had someone laugh because his team had ignored my manager’s bug report for a full year.  That’s not funny; it’s pathetic.

When management fails to address an issue; be it a known system problem or something as simple as morale from a lost team member, it shows the team that they don’t care. It soon becomes a vicious cycle of uncaring when managers no longer care that the site is down, which in turn causes developer apathy.  Developers then don’t care about code quality, leading to buggy code. Sysops stop caring that alerts are going off, leading to downtime. By the time the cycle is broken, it’s far too late – you’ve established a bad reputation with your customers.

Often times this will start with unreasonable development expectations, causing devs to cut corners, QA to be rushed, and monitors to be forgotten. There is a balance that must be maintained between getting code out the door and making sure that the code can stand up to the abuse it will receive when it goes live.  It’s a team effort, and everyone must care (and keep caring) to keep the systems running.

Wow. Well, that’s a lot more than I intended on writing. I should state that I am guilty of 75% or more of the bad behaviors listed here. I hope that this will help start discussion on how to better improve monitoring systems.

If you have feedback, suggestions or enhancements, please leave them in the comments.

(Thanks to jdrost, jslauter, keith4, pakrat, romaink, and my wife Jackie for their peer review/editing.)

Less is more.

That’s the common joke about less- It provides the same functionality as the older utility, more; but oh, how much more than more!

Less allows you to easily scan backwards as well as forwards- something more is not too good at (though it is possible). Less also allows you to navigate with arrow keys, page up and page down, home and end.

Less provides a quick way to view files as well- Many editors (like vim) need to read an entire file first, and often create a temporary copy (for editing) of the file. This isn’t a problem until you try to browse a 56 gig log file of a rabid JBoss instance- if your root directory isn’t large enough, chaos ensues. Less circumvents this by only loading a page at a time. Yes- searches do take longer, but it beats crashing a system.

Useful Commands:

• R: Repaints the screen, rereading the file from the disk. useful if the file is constantly changing (like a logfile)
• / and ?: Just like in vim, used for searching forwards and backwards
• g and G: Jumps to the top or bottom of the file, just like vim
• :n and :p: Jump to the next/previous file (if you pass less more than one file)
• q: quits. slightly important
• ! [cmd]: runs a given command outside of less- useful if you can’t remember the output of another command and need a quick reminder (ifconfig, tail, route, ls, etc)

Useful Flags:

• -N: Shows line numbers
• -s: squeeze adjacent blank lines into one
• -R: shows raw control characters, a.k.a. “turns color on!”

As usual, there are dozens of more options, flags, variables and behavior- you’d be surprised what less can do for you.

Your browser is not supported.

We do not support your browser yet…

If you are on a Windows machine please use:
* Internet Explorer 6 and up
* Firefox 1.5 and up
* Safari Beta 3 and up

If you are on a Mac please use:
* Safari
* Firefox 1.5 and up

Your current browser is Mozilla 1.9.2.5

There are several problems with this:

1. I am using firefox 1.5 and up, specifically, "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.5pre) Gecko/20100501 Ubuntu/9.10 (karmic) Namoroka/3.6.5pre" I can only presume that since it doesn’t say “firefox”, it fails.
2. You’re parsing my connection string wrong-1.9.2.5 is related to my profile version, not browser version. Don’t do that.
3. Even after I change my connection string to Firefox/1.5, it STILL fails. The error message says Windows or Mac; maybe it’s failing because it is looking for one or the other.
4. The site is completely useless if that page comes up- no contact info to report a problem, nothing. I can’t even tell you that you’re accidentally blocking me.

I end up having to use the following connection string:
"Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.0.8) Gecko/20061025 Firefox/1.5.0.8"
and the sad part was YOUR SITE LOOKED FINE. This kind of ungraceful degradation irritates the hell out of many people. If you don’t support my browser, I get it- you don’t want to give people a bad experience. You know what you do rather than make the site completely useless? You pop up a little window saying:

Sorry, we do not support your browser yet…

For better browsing experience, we suggest you try the following:
If you are on a Windows machine please use:
* Internet Explorer 6 and up
* Firefox 1.5 and up
* Safari Beta 3 and up

If you are on a Mac please use:
* Safari
* Firefox 1.5 and up

Your current browser is Mozilla 1.9.2.5
[x] continue anyways…

Note that the little [x] is a close button- VERY important on a warning like that. This allows the users to continue on viewing your website on the off chance that you accidentally mislabeled them.

This is 2010. This is the future. An issue like this was annoying back in 1998- there’s no reason a pretty site like yours should be doing this. Get with the program.

Sincerely,
- Jesse

1. Port Expansion: I mentioned there were 7 ports we use; originally there were only 6, but I enabled SNMP to better trend JVM usage (24 JVMs use a lot of RAM). As we add more features, more ports could be needed.
2. Application Expansion: We originally started out with 9 instances, and after 5 months added another three. More will be possible in the future.

The required flexibility makes numbering conventions difficult. The solution we came up with is to key the information and use it to build the port number. In order to do that, we had to make a few assumptions:

1. Environments are sequestered to their own hardware, however they’re on the same networks- i.e. dev and qa instances will never be on the same physical server, but will share multicast ranges
2. There will not be more than 5 instances in a cluster- any more than that would put theoretical ports out the 65536 range; besides, replication shouldn’t be past 4 nodes, right?
3. Ports below 1024 are off limits
4. We are constrained by IP addresses; one IP per server.

From here we could allow the following constraints:

1. Environments can use the same ports, EXCEPT for multicast- we don’t want QA and prod replicating data!
2. All multicast should use 1 for their Instance ID

With that in mind, port numbers will be laid out with the following structure: XYYZZ

X = Instance ID (e.g. 1-4)
YY = application ID (Helium,Hydrogen,Lithium, etc )
ZZ = Port Type (jmx, ajp, snmp, etc)


1 Instance1
2 Instance2
3 Instance3
4 Instance4



01 Hydrogen
02 Helium
03 Lithium
04 Beryllium
05 Boron
06 Carbon
07 Nitrogen
08 Oxygen
09 Fluorine
10 Neon
11 Sodium
12 Magnesium



01 Shutdown
02 JMX
03 HTTP
04 AJP
05 SNMP
06 REPL
07 MCASTDEV
08 MCASTQA
09 MCASTUAT
10 MCASTPREPROD
11 MCASTPROD


This allows us to define the following:

• Hydrogen AJP for instance 1 is 10104 (1+01+04)
• Oxygen SNMP for instance 4 is 40805 (4+08+05)
• Sodium Multicast for Prod 3 is 11111 (1+11+11) (remember, all multicast use 1 for their instance)

Other than the Multicast port kludge, this way seems to work pretty well. Numbers are easy to parse visually and can be maintained with a centralized key on the wiki. Anyone have any suggestions or examples of how they’ve solved this problem?

Let me know if you find it useful.

 
#!/usr/bin/perl
# read your ldap repo and generation nagios config files for contacts
# and contact_groups
use strict;
use Net::LDAP;
use Data::Dumper;
 
# Set all your custom configs here:
my $ldaphostname='ldap.example.int';
my $groupdn='ou=People,ou=Groups,dc=example,dc=int';
my $peopledn='ou=People,ou=active,ou=Users,dc=example,dc=int';
my $contactdir='/etc/nagios/contacts';
 
my $ldap = new Net::LDAP ($ldaphostname);
$ldap->bind() || die "failed to bind";
 
#search for groups
my $mesg = $ldap->search(
        base => $groupdn,
        filter => "(objectclass=posixgroup)",
        attrs => ['cn','memberuid'],
        );
 
#loop through each entry
while (my $group= $mesg->pop_entry() ){
 
    #set the name of the group
    my $cn=$group->get_value('cn');
 
    #create a comma-separated list of group members
    my $members=$group->get_value('memberuid', asref => 1);
    $members= join( ',', @$members);
 
    # create our configuration block
    my $contactinfo="
        # We only have one contact in this simple configuration file,
        # so there is no need to create more than one contact group.
 
        define contactgroup{
            contactgroup_name       $cn
            alias                   $cn
            members                 $members
            }\n";
 
    # write this block to a file named after the group cn
    print `echo "$contactinfo" > $contactdir/group_$cn.cfg`;
}
 
#search for users
my $mesg = $ldap->search(
            base => $peopledn,
            filter => "(objectclass=person)",
            attrs => ['uid','cn','mail','description'],
           );
 
#loop through each entry
while (my $user= $mesg->pop_entry() ){
 
    my ($pager,$addr);
    # grab the four fields we're interested in
    my $uid=$user->get_value('uid');
    my $cn=$user->get_value('cn');
    my $mail=$user->get_value('mail');
    my $desc=$user->get_value('description');
 
# Note that I store the user's SMS address and Instant Messenger
# addresses in the description field in the following format:
# SMS:1234445555@some.url;
# IM:morgajel@yahoo,morgajel@gtalk;
# for those interested, you can send email to an sms address 
# with help from <a href="http://sms411.net/how-to-send-email-to-a-phone">http://sms411.net/how-to-send-email-to-a-phone</a>
    #strip pager info out of desc and format it
    #(note it's in sms format):
    if ($desc =~/SMS:([^;]+);?/){
        $pager= "        pager           $1\n";
    }
    # strip addresses out of desc and format them:
    if ($desc =~/IM:([^;]+);?/){
        my @multiple_addr=split(/,/,$1);
        my $count=1;
        # break each address out into an addressX line
        # (for informational purposes);
        foreach my $addr (@multiple_addr){
            $addr=$addr."        address$count        $addr\n";
            $count=$count+1 ;
        }
    }
 
    #generate the first half of the contact info
    my $contactinfo="
    define contact{
        contact_name    $uid              ; Short name of user
        use             generic-contact
        alias           $cn               ; Full name of user
        email           $mail
$pager
$addr
    }";
 
    # write this block to a file named after the user uid
    `echo "$contactinfo" > $contactdir/$uid.cfg`;
}

(11:33:42 AM) Jeff: so many gone
(11:34:10 AM) Jeff: jesse
matt
carrie
lisa
stacey
kurt
roger
jim
naveen
brandon
lee
bink
tristan
jason
jen
sean
drew
radha
VJ
jackie
andy
noelle
whip
reedy
jeremy
keith
ryan
pankaj
nirupa
asad
santhi
rick
limer
meads
mick
(11:34:42 AM) Jesse: you should post that as a facebook status update. one comma separated list.
(11:39:06 AM) Jeff: lol
(11:39:12 AM) Jeff: those are just people i knew and worked with in IT
(11:39:18 AM) Jeff: i’m not counting others I didn’t work with or know in IT

SUBJECT: [#24413540] Suspension Notice CPU Limits Reached (1st Warning)
Hello,

Due to the amount of CPU and/or memory resources used by your account, we were forced into suspending the account. Please understand that we only suspend an account as a last resort and want to help you track down the cause as quickly as possible.

Shared servers (MegaPhase, Prophase, and ANHosting accounts) are only allotted 10% of the server’s resources (CPU and Memory) at any given time. This
restriction is put in place to keep balance on the server. The first step is to
try to isolate what content on your website is causing the overage. If this
seems unexpected, it could be a recent change to your site (such as installing
a new wordpress plugin, mailing list application, etc). Any information you can
provide about recent changes can help us to isolate the problem. However, we
were able to retrieve the following files and/or scripts that likely caused the
suspension:

————————

literaryescapism.com 2.33 1.38 0.1
Top Process %CPU 34.0 [php]
Top Process %CPU 25.0 public_html/index.php
Top Process %CPU 23.0 public_html/index.php

————————

This was the beginning of an email I received from our hosting company, telling me that Jackie’s site was suspended due to usage. I’m not disappointed so much that they suspended it (which was the proper course of action), but in how they responded to me. Here’s my response to the warning email:

I’m trying to figure out if this is new behavior from a wordpress exploit or if it’s honestly just grown to this point; do you have any trending data for my cpu usage?

Side note- the access-logs linked in my home directory appear to go nowhere:
access-logs -> /usr/local/apache/domlogs/user
that path doesn’t exist.

I would like to help in any way I can. As a side note, I’ve noticed quite a few “errors connecting to mysql” in the apache logs; could there be an underlying issue with mysql that’s causing processes to back up and chew cpu time?

I’m also seeing some defunct php processes from other users, leading me to wonder if this is more of a systemic problem than one with just my account.

(sorry, I’m an apache/linux admin myself- let me know if there’s anything I can do to assist. I can be reached on gtalk, aim or yahoo.)

And their response:

Hello,

Unfortunately we do not have any further data except for what was given in the e-mail:

literaryescapism.com 2.33 1.38 0.1
Top Process %CPU 34.0 [php]
Top Process %CPU 25.0 public_html/index.php
Top Process %CPU 23.0 public_html/index.php

Your account has been unsuspended so you can troubleshoot this issue. Please let us know if you have any further questions.
–
Best regards,

xxxxxxxxxxxxxx
Technical Support Representative
Hosting Services, Inc.

It’s sort of dumbfounding to be completely shut down like that from a TSR. I would have hoped that he would have seen that I A) know what I’m talking about and B) would have booted me up to someone who could better assist me. Instead he unlocks the account (which, btw, was never locked or suspended as far as I could tell from the time I got the email and checked until he sent the notice that it was unsuspended) and says “sorry, we have no data to help you figure out what happened.” Note that he didn’t address my questions regarding the broken symlink, mysql availability, or defunct processes.

Flustered, I responded with this:

Is it still using such high CPU load? I noticed several processes on the server that were having issues when I looked; many of which were not mine- The load on the server was 114 when I checked it last night. Regardless of server, that’s high, and I wasn’t one of the high processes listed.

I suspect this was a systemic issue- perhaps mysql went down causing multiple sites to freak out. If that’s the case (and there was a problem with the system itself), troubleshooting from my end is near impossible, especially since you have no trending data on usage. To determine if this was a one-time issue or a building issue.

I’d like to help you guys, but ya gotta work with me here. This is what I do for a living- I’m the Apache/Linux administrator. My wife’s site has been running for 6 months with no major changes (other than simple wordpress upgrades, none of which
were in the last week) and no notable increase in traffic or bandwidth usage over the past week.

If you’re incapable of providing details as to whether or not there was a system issue (or even if other clients had similar issues at the same time) my only course of action would be to take our sites to another provider. I don’t want to do that; I’ve been happy with you guys so far.

Please, check and see if there were multiple clients on this server who received warnings as well; I suspect I wasn’t the only one. From where I’m standing the server does not look very healthy- I’m seeing multiple defunct processes:

nobody 6857 0.0 0.0 0 0 ? Z 08:46 0:00 [httpd]
1027 10565 15.0 0.0 0 0 ? Z 08:48 0:00 [php]
32003 10573 0.0 0.0 0 0 ? Zs 08:48 0:00 [cpsrvd-ssl]

Please let me know if there is a sysadmin I can work with to resolve this issue. Like I said, I don’t think this is a problem with my site necessarily.

We’ll see how they respond. Hopefully they’ll be useful- I just switched to them in October, so I really don’t want to have to find another host.