Archive for December, 2010

Revisiting Ziggy’s Book

I appeared to have stalled out on SPoE before I could finish it, so I’m flipping back to the reason for SPoE- Ziggy’s book. After taking a full year off of writing, I’m ready to revise the first of the two books. Last year I stalled out on the 3rd draft and was unable to get restarted; now I think enough time has passed to give me a fresh perspective.

Here’s my current method of attack. I’ve printed up the second draft of the entire book through lulu in a spiral-bound, full page format with plenty of room for making edits (~330 pages) . I’ll try to edit one chapter a week for 2 months (there are 8 chapters, but I’ll give 2 weeks to the first chapter). Part of my editing process will involve a set of pens that I’ll keep in a pouch along with it:

  • Black Pen – New additions, replacements
  • Red Pen – Corrections (spelling, punctuation)
  • Blue Pen – Ideas, notes, non-prose
  • Yellow Highlighter – Area that needs work
  • Green Highlighter – Area I like
  • Sharpie – Kill it with fire, I don’t wanna remember it.
  • Mech. Pencil – Overall structures, themes, constructs.

I’ll go through each chapter several times during it’s allocated week;

  1. Read-through: a personal read to reacquaint myself.
  2. Red Pen, Pencil: correcting painful errors, identifying patterns and motifs.
  3. Verbal Read-through: read aloud to identify bad structure.
  4. Yellow Highlighter, Green Highlighter,Blue Pen: marking areas I like or want to change, flesh out ideas.
  5. Black Pen, Sharpie: changes to be made, sentences that should be obliterated.

I’m hoping that a nice, even pace will keep me from burning out, and a logical set of tools will keep the process fresh. With luck, it’ll give me the drive to create a manuscript polished enough to show others. I’m hoping that breaking each chapter into five run-throughs will make it bite-size enough that I can do a little bit each night.  After I finish this process for each chapter, I’ll make the changes and create a new, third draft.

With the start of the 3rd draft, I’ll have to look at pacing and structure issues, but I’ll worry about that when I get to it.

Humble Bundle #2

So this year I managed to snag the Humble Bundle package;  If you’re not familiar with it, it’s a pack of Indie Games bundled together. There are five good reasons why you should consider getting it:

  • Name your own price ($5000 or $0.01 – your call)
  • You choose who the money goes to (charities and/or developers)
  • If you pay more than the average donation, you get Humble Bundle #1 from last year.
  • All games work on Windows, Mac and Linux
  • No DRM

Our Christmas funds were already drained, but Jackie and I agreed that we could spare $15.  I’m going through and trying the games now and thought I’d write up my notes on them:

Osmos

You’re a little bubble that propels itself by ejecting some of it’s mass in a given direction; the more mass you eject, the faster you go. As you float around, you see other blobs- some smaller, some larger. When one blob touches another, the smaller of the two is absorbed by the larger, bur as your mass grows, velocity slows. It becomes a balancing act of launching yourself to get blobs slightly smaller than yourself; if you eject too much, when you hit your target, you’ll be smaller and will be absorbed by your target (and thus dying).

There are two different types of play; “become the biggest” “absorb the ___”. Become the Biggest is played in a square or round sandbox, or it may be orbiting a massive blob; each has it’s own challenges.  Absorb the ___ involves trying to absorb moving targets- other AI blobs or repulsers (runs away from you). There may be more modes of gameplay, but I haven’t gotten to them yet.

I’ve only dug into it a bit, but it’s very addictive from what I’ve played. The only issue I’ve had is it occasionally freezes;  I have to alt-tab out of it and kill it when it happens.  It’s only happened twice, so it’s not that bad. I am grateful that the devs decided to package it as a .deb file so installing it is as simple as doing a sudo dpkg -i. Overall it’s an excellent game and easily worth $15 by itself. I could also see it ending up on the Wii due to it’s simplicity and controls.

Braid

Braid is one of those horrible horrible games like Tetris or Columns where the gameplay sinks into your brain and your start envisioning it everywhere you go.  The crux of the game is that your character can rewind time;  while it appears to be a side-scrolling platformer, you can’t actually die; so it’s closer to a puzzle game despite the running and jumping.

Each level you open up unlocks a new “feature” items that sparkle green ignore the reverse flow of time; even though you run backwards, the badguys keep moving, or the key stays where you left it. purple sparkles cause a shadow version of you to re-perform the actions you just performed; if you need to get in a gate but the purple lever is too far away and the gate closes before you get there, you can run to the gate, run to the lever, then pull it; rewind until you’re back to the gate and watch the shadow you run over and pull the lever. while he’s doing that, you can run through the gate.

I haven’t gotten very far, but the storyline is very interesting; we all wish we could rewind time and undo our mistakes, but eventually you will become a prisoner of other people’s expectations. Very fun, and again, easily worth $15. The controls are a little funky on my laptop, but this would be a great xbox arcade game. They just have a bin file which creates a dir on your system; it woulda been nicer if it had a .deb file so I could track it.

World of Goo

I’ve heard a lot about this game, but this is the first time I’ve seen it. It was actually part of Humble Bundle #1, but since I paid over the average ($7.53 at the time I purchased it), I got HB#1 as well. It’s a basic physics/building game; think tinkertoys made out of jello. The goal is to get these little goo balls into a tube; usually this is done by building some type of bridge or tower that the gooballs can climb. when you place a gooball near an existing structure, it forms a goo strut; if it only connects to one, it’ll flop like a joint. if you place it next to two, it’ll form a triangle which is relatively sturdy.

There are many different types of goo; some that drip like snot, some that can only be placed on a structure once, some that can’t form structures, etc. To top it off, if you build an awkward tower out of the gooball, the struts will bend and give way, causing the tower to collapse.

It’s a fun game, broken into many levels and chapters, but not really as enthralling as Braid. I could see this on the Wii for $15 or so by itself.

Revenge of the Titans

Revenge of the Titans is a simple tower defense game based around an alien invasion; the graphics are stylized  but simple, and the gameplay is nothing special. I’ve played probably close to 50 or so tower defense games;  It’s a little more fast paced than most and seems to just be one solid, continual wave.  It’s a well made and decent game, but I’m sorta burned out on the genre.

Samorost2

Samorost2 is a Myst-like puzzle game. Unlike the others so far, this one was flash based. The story revolves around a guy living on a small planet with his dog. When aliens stop to steal his pears and kidnap his dog, he follows them back to their base to rescue him. Each phase of gameplay involves a static screen with minor animations; you identify something on the screen that’s interactive, like a flower or ladder, and when you click on it, it triggers an animation. The gameplay is based on trying different combinations of events and timings. I was able to beat it in around an hour and a half, which makes it the shortest of the games so far. The artwork is great, but in the end it’s “just a flash game.”

Still worth a play for the artwork alone.

Lugaru HD

Lugaru appears to be a single-player  3rd person RPG, similar to WoW. The graphics are a bit rough, but it looks interesting nonetheless. Unfortunately, the controls aren’t great on a laptop, but if I had a mouse I’d probably be more interested in it. Walking through the tutorial, I got a feel for the combat, which seems crazy and fun. I’ll have to spend more time on it later.

So thats just a quick sample of what’s in the pack- I haven’t had time to play the other games. This pack is easily worth $15 several times over, and I sorta feel like I.  If you haven’t heard of the Humble Bundle, now is your chance to check it out. Hurry though, it’s not available after Christmas day.

SPoE: Hibernate Search and Spring 3 Integration Example

You’d think that documentation on creating a search engine for a Spring 3 MVC website would be plentiful- search boxes are on most sites these days, and it should be trivial to implement some type of search functionality in Spring MVC.  Unfortunately, there is very little direct documentation on the subject,  so I thought I’d write up what I did so others can benefit (and hopefully I hit enough keyword combos to have it be found).

Prerequisites:

  • You want to implement a search box that scans a given model looking for a term.
  • You have a decent grasp on java, Spring 3 and are using Hiberate Search 3.1.
  • You have all your classpaths loaded, and a functional, if not skeletal MVC site.

Step 1: Grok the basics of Hibernate Search

Start off by reading a little bit of the documentation for Hibernate Search, mainly “Example 1.5. Example entities after adding Hibernate Search annotations.” Go ahead and annotate your model as well as you can using their example as a guide- it doesn’t have to be perfect, just make sure you include a field you’d like to search.

Step 2: Add Hibernate Search Configuration

Next you have to add some hibernate properties. You can do this in your persistence.xml, hibernate properties, or in your *-servlet.xml where you define your Hibernate Session Factory (which is what I did). Here’s an example of the props I used- note that my indexBase is in Tomcat’s /tmp/ directory- that may not be the best place for it, but it works for the time being:

<bean id="mySessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
...
    <property name="hibernateProperties">
      <props>
        ...
        <prop key="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</prop>
        <prop key="hibernate.search.default.directory_provider">org.hibernate.search.store.FSDirectoryProvider</prop>
        <prop key="hibernate.search.default.indexBase">${catalina.base}/tmp/indexes</prop>
      </props>
    </property>
</bean>

Step 3: Pass SessionFactory to Controller

After you add these properties, you’ll need to autowire your SessionFactory bean into your controller:

<bean id="searchController">
    <property name="sessionFactory" ref="mySessionFactory" />
</bean>

You’ll also need to add the following to your SearchController to get the sessionFactory autowired:

@Autowired
public SessionFactory getSessionFactory() {
    return sessionFactory;
}
public void setSessionFactory(SessionFactory pSessionFactory) {
    this.sessionFactory = pSessionFactory;
}

Step 4: Implement Search

Now for the final step- to implement the actual search.  I’ll be searching the title and content of my Snippet Class. In my SearchController I’ll add the following:

@RequestMapping
public ModelAndView quickSearch(@RequestParam("q") final String searchQuery) {
...
    //FIXME need to sanitize user input
    //FIXME make sure to handle Exceptions
    ModelAndView mav = new ModelAndView();

    //Open a Hibernate Session since none exist. This was the cause of much grief.
    Session session = sessionFactory.openSession();
    //Then open a Hibernate Search session
    FullTextSession fullTextSession = Search.getFullTextSession(session);
    //Begin your search transaction
    Transaction tx = fullTextSession.beginTransaction();

    //Create a list of fields you want to search
    String[] fields = new String[]{"title", "content", "lastModifiedDate"};
    //Define what type of parser and Analyzer you want to use
    MultiFieldQueryParser parser = new MultiFieldQueryParser(fields, new StandardAnalyzer());
    // Create a low-level Lucene Query from our parser using our searchQuery passed in from the user
    org.apache.lucene.search.Query query = parser.parse(searchQuery);
    // Run a Hibernate Search query using the lucene query against our Snippet class
    org.hibernate.Query hibQuery = fullTextSession.createFullTextQuery(query, Snippet.class);

    // List the results
    List results = hibQuery.list();
    // Commit the transaction
    tx.commit();
    // Close your hibernate session
    session.close();
    // pass your result set to your ModelAndView to be used.
    mav.addObject("results", results);

And just like that, a simple search in your Spring 3 MVC application using Hibernate Search.  I’m still a little fuzzy about why you need a transaction for a read-only search, but hey, it’s working, I can fine tune later. It goes without saying that there is no exception handling here- it’s not critical to the example.

The only downside to this implementation is that it doesn’t index existing content, only new content. You CAN do a full index by running:

    FullTextSession fullTextSession = Search.getFullTextSession(session);
    Transaction tx = fullTextSession.beginTransaction(); 
    List snippets = session.createQuery("from Snippet as snippet").list();
      for (Snippet snippet : snippets) {    
        fullTextSession.index(snippet);
      }
    tx.commit(); //index is written at commit time

But I’d suggest making an administrative tool/button/one-time-event out of it rather than implementing it here, because I sure there’s a slight performance hit in indexing thousands of lines of text.

I hope this ends up being of value to someone- if it does, please leave a comment letting me know, and if I have any mistakes above, please let me know that as well 😀

SPoE: Loadtesting, Emails and Connections

So as I mentioned earlier, loadtesting with 70 threads failed because the default MTA on Ubuntu (exim) is limited by default to 20 concurrent connections (smtp_accept_max). Since I was using under 20% of the cpu and the bottleneck was an unconfigured mailserver, I really wanted to see where my app bottomed out, meaning I needed to fix the mail issue and keep moving.

I should also mention that I switched to localhost to avoid getting myself in trouble by sending out 10,000 registration emails (when I checked the exim queue, there were 28.5k unsent email- oops).

I spent a couple hours trying to configure Exim with a higher smtp_accept_max, but I couldn’t figure out how to get the split configuration file to read smtp_accept_max, regardless of which of the 30 config fragments I tried placing it in. I also did a dpkg-reconfigure to switch it to the monolithic configuration, and it seemed to accept smtp_accept_max (according to exim -bP smtp_accept_max smtp_accept_max_per_host), however the RCPT was refused, so it was a non-starter. I fought for a few hours researching, configuring, and cursing before asking myself “why am I still bothering with Exim?” I don’t need massive configuration, I just need something to accept mail and toss it straight to /dev/null. So my first thought? “Oh, I’ll just install Sendmail.”

Anyone familiar with Sendmail knows that massive configuration is needed to get it to do anything. I had a base configuration for my servers, but it really wasn’t what I needed. Using the default config I tried the loadtest again, only to have it fail telling me “twilluser12123@localhost does not exist.”
After about 10 minutes I came to my senses and realized that I really didn’t want to configure sendmail properly because I didn’t know how long it would take and I couldn’t recall how to set up a catchall address at 1am.

When I got up the next morning, I installed postfix (another MTA like sendmail and exim4) that I’d used previously- I remembered it being relatively easy to set up a catchall address. Fortunately it was easy to set up, but I feared having the same connection limitation issues as exim’s default config.

I fired up the jmeter script with 70 threads, 210 second rampup, 100 loops, and an empty database. Each thread loop creates and activates and account, logs in and views it’s edit account page, then creates and views 5 snippets. The test completed successfully!

Here are the stats from the test:

  • 7k users were created
  • 35k snippets were created
  • 189k samples were taken
  • That’s a lot… so what did we find?

    JMeter

    • 12 ms Average page return
    • 7 ms Median page return
    • 490 ms Max page return
    • 0 errors in assertions.
    • 224.7 hits/second
    • 1082.1 KB/second throughput

    Tomcat

  • 239M of Tomcat memory in use (max)
  • 332M of Tomcat memory allocated (max)
  • 30% CPU (max)
  • 90 threads in use (max)
  • 6,477 classes loaded (max)
  • There were only 2 errors in the tomcat logs:

    Dec 11, 2010 11:06:34 AM org.apache.catalina.core.StandardWrapperValve invoke
    SEVERE: Servlet.service() for servlet dispatcher threw exception
    java.lang.IllegalStateException: removeAttribute: Session already invalidated
    	at org.apache.catalina.session.StandardSession.removeAttribute(StandardSession.java:1236)
    Dec 11, 2010 11:06:34 AM org.apache.catalina.core.StandardHostValve custom
    SEVERE: Exception Processing ErrorPage[exceptionType=java.lang.Exception, location=/WEB-INF/errors/uncaught-error.jsp]
    java.lang.IllegalStateException: Cannot reset buffer after response has been committed
    	at org.apache.catalina.connector.Response.resetBuffer(Response.java:691)

    Note that the performance metrics seem better than they should be since I’m testing locally and there’s no network latency. At this point I’m pretty sure the app is performing above my needs, so I have a couple of options:

    • Write more functionality
    • Improve quality of jmeter testing
    • Improve quantity of jmeter testing
    • Push existing tests even further to find out where it will break (90 threads? 120 threads?)

    I’m not sure what I’m going to do yet.

    SPoE: Performance and Errors Under Load

    One concern I have with SPoE is that, should it get popular, it must handle traffic to a reasonable degree. Since I deal with misbehaving Java apps at work all the time, I decided to test mine and see how it behaved under load. I’m running tomcat via eclipse, trending memory usage with VisualVM, and running the test with JMeter.

    I decided to start small- simply registering a new account. With only 5 pages, the test flew by and gave me a false sense of security. I upped the ante to registering a new account, logging in, viewing the user edit account page, then creating and viewing a series of snippets.  That’s when things fell apart. With 15 threads, 15 seconds of startup, and 100 loops, the first failure happened at around the 700 sample mark.  The failure happened when JMeter was attempting to create a snippet. The only problem is the error code that it spat back in the tomcat eclipse log:

    INFO: Server startup in 7158 ms
    Dec 9, 2010 10:09:31 PM org.apache.catalina.core.StandardWrapperValve invoke
    SEVERE: Servlet.service() for servlet dispatcher threw exception
    java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
     at java.util.ArrayList.RangeCheck(ArrayList.java:547)
     at java.util.ArrayList.get(ArrayList.java:322)
     at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter$RequestMappingInfo.bestMatchedPattern(AnnotationMethodHandlerAdapter.java:1017)
     at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter$RequestMappingInfoComparator.compare(AnnotationMethodHandlerAdapter.java:1104)
     at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter$RequestMappingInfoComparator.compare(AnnotationMethodHandlerAdapter.java:1)
     at java.util.Arrays.mergeSort(Arrays.java:1270)
     at java.util.Arrays.sort(Arrays.java:1210)
     at java.util.Collections.sort(Collections.java:159)
     at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter$ServletHandlerMethodResolver.resolveHandlerMethod(AnnotationMethodHandlerAdapter.java:611)
     at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:422)
     at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:415)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:788)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:717)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:366)
     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
     at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:177)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:91)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
     at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378)
     at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:167)
     at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
     at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
     at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
     at java.lang.Thread.run(Thread.java:619)
    

    As you can see, there’s nothing form com.morgajel.spoe in there. Here’s the controller snippet that fires when /snippet/create is hit:

    @RequestMapping("/create")
     public ModelAndView createSnippetForm() {
         LOGGER.info("showing the createSnippetForm");
         ModelAndView  mav = new ModelAndView();
         mav.setViewName("snippet/editSnippet");
         mav.addObject("editSnippetForm", new EditSnippetForm());
         return mav;
     }
    

    So wtf is causing the error? I don’t know. Now I’m not sure if I keep chasing this rabbit or I move on- on one hand, I hate hate HATE when there’s performance issues and devs ignore it; on the other, this is a side project that isn’t going into production anytime soon.

    host change

    yet again changing hosts. LE is getting too big for it’s britches.

    Go to Top