Stemming the flow of evincible Ignorance. We must try to understand for the sake of understanding.
SPoE
SPoE: Hibernate Search and Spring 3 Integration Example
12 years
You’d think that documentation on creating a search engine for a Spring 3 MVC website would be plentiful- search boxes are on most sites these days, and it should be trivial to implement some type of search functionality in Spring MVC. Unfortunately, there is very little direct documentation on the subject, so I thought I’d write up what I did so others can benefit (and hopefully I hit enough keyword combos to have it be found).
Prerequisites:
- You want to implement a search box that scans a given model looking for a term.
- You have a decent grasp on java, Spring 3 and are using Hiberate Search 3.1.
- You have all your classpaths loaded, and a functional, if not skeletal MVC site.
Step 1: Grok the basics of Hibernate Search
Start off by reading a little bit of the documentation for Hibernate Search, mainly “Example 1.5. Example entities after adding Hibernate Search annotations.” Go ahead and annotate your model as well as you can using their example as a guide- it doesn’t have to be perfect, just make sure you include a field you’d like to search.
Step 2: Add Hibernate Search Configuration
Next you have to add some hibernate properties. You can do this in your persistence.xml, hibernate properties, or in your *-servlet.xml where you define your Hibernate Session Factory (which is what I did). Here’s an example of the props I used- note that my indexBase is in Tomcat’s /tmp/ directory- that may not be the best place for it, but it works for the time being:
<bean id="mySessionFactory" class="org.springframework.orm.hibernate3.annotation.AnnotationSessionFactoryBean">
...
   <property name="hibernateProperties">
     <props>
       ...
     <prop key="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</prop>
       <prop key="hibernate.search.default.directory_provider">org.hibernate.search.store.FSDirectoryProvider</prop>
     <prop key="hibernate.search.default.indexBase">${catalina.base}/tmp/indexes</prop>
     </props>
   </property>
</bean>
Step 3: Pass SessionFactory to Controller
After you add these properties, you’ll need to autowire your SessionFactory bean into your controller:
<bean id="searchController"> Â Â Â <property name="sessionFactory" ref="mySessionFactory" /> </bean>
You’ll also need to add the following to your SearchController to get the sessionFactory autowired:
@Autowired
public SessionFactory getSessionFactory() {
   return sessionFactory;
}
public void setSessionFactory(SessionFactory pSessionFactory) {
   this.sessionFactory = pSessionFactory;
}
Step 4: Implement Search
Now for the final step- to implement the actual search. I’ll be searching the title and content of my Snippet Class. In my SearchController I’ll add the following:
@RequestMapping
public ModelAndView quickSearch(@RequestParam("q") final String searchQuery) {
...
   //FIXME need to sanitize user input
   //FIXME make sure to handle Exceptions
   ModelAndView mav = new ModelAndView();
   //Open a Hibernate Session since none exist. This was the cause of much grief.
   Session session = sessionFactory.openSession();
   //Then open a Hibernate Search session
   FullTextSession fullTextSession = Search.getFullTextSession(session);
   //Begin your search transaction
   Transaction tx = fullTextSession.beginTransaction();
   //Create a list of fields you want to search
   String[] fields = new String[]{"title", "content", "lastModifiedDate"};
   //Define what type of parser and Analyzer you want to use
   MultiFieldQueryParser parser = new MultiFieldQueryParser(fields, new StandardAnalyzer());
   // Create a low-level Lucene Query from our parser using our searchQuery passed in from the user
   org.apache.lucene.search.Query query = parser.parse(searchQuery);
   // Run a Hibernate Search query using the lucene query against our Snippet class
   org.hibernate.Query hibQuery = fullTextSession.createFullTextQuery(query, Snippet.class);
   // List the results
   List results = hibQuery.list();
   // Commit the transaction
   tx.commit();
   // Close your hibernate session
   session.close();
   // pass your result set to your ModelAndView to be used.
   mav.addObject("results", results);
And just like that, a simple search in your Spring 3 MVC application using Hibernate Search. I’m still a little fuzzy about why you need a transaction for a read-only search, but hey, it’s working, I can fine tune later. It goes without saying that there is no exception handling here- it’s not critical to the example.
The only downside to this implementation is that it doesn’t index existing content, only new content. You CAN do a full index by running:
   FullTextSession fullTextSession = Search.getFullTextSession(session);
   Transaction tx = fullTextSession.beginTransaction();Â
   List snippets = session.createQuery("from Snippet as snippet").list();
     for (Snippet snippet : snippets) {   Â
   fullTextSession.index(snippet);
    }
   tx.commit(); //index is written at commit time
But I’d suggest making an administrative tool/button/one-time-event out of it rather than implementing it here, because I sure there’s a slight performance hit in indexing thousands of lines of text.
I hope this ends up being of value to someone- if it does, please leave a comment letting me know, and if I have any mistakes above, please let me know that as well 😀
SPoE: Loadtesting, Emails and Connections
12 years
So as I mentioned earlier, loadtesting with 70 threads failed because the default MTA on Ubuntu (exim) is limited by default to 20 concurrent connections (smtp_accept_max). Since I was using under 20% of the cpu and the bottleneck was an unconfigured mailserver, I really wanted to see where my app bottomed out, meaning I needed to fix the mail issue and keep moving.
I should also mention that I switched to localhost to avoid getting myself in trouble by sending out 10,000 registration emails (when I checked the exim queue, there were 28.5k unsent email- oops).
I spent a couple hours trying to configure Exim with a higher smtp_accept_max, but I couldn’t figure out how to get the split configuration file to read smtp_accept_max, regardless of which of the 30 config fragments I tried placing it in. I also did a dpkg-reconfigure to switch it to the monolithic configuration, and it seemed to accept smtp_accept_max (according to exim -bP smtp_accept_max smtp_accept_max_per_host), however the RCPT was refused, so it was a non-starter. I fought for a few hours researching, configuring, and cursing before asking myself “why am I still bothering with Exim?” I don’t need massive configuration, I just need something to accept mail and toss it straight to /dev/null. So my first thought? “Oh, I’ll just install Sendmail.”
Anyone familiar with Sendmail knows that massive configuration is needed to get it to do anything. I had a base configuration for my servers, but it really wasn’t what I needed. Using the default config I tried the loadtest again, only to have it fail telling me “twilluser12123@localhost does not exist.”
After about 10 minutes I came to my senses and realized that I really didn’t want to configure sendmail properly because I didn’t know how long it would take and I couldn’t recall how to set up a catchall address at 1am.
When I got up the next morning, I installed postfix (another MTA like sendmail and exim4) that I’d used previously- I remembered it being relatively easy to set up a catchall address. Fortunately it was easy to set up, but I feared having the same connection limitation issues as exim’s default config.
I fired up the jmeter script with 70 threads, 210 second rampup, 100 loops, and an empty database. Each thread loop creates and activates and account, logs in and views it’s edit account page, then creates and views 5 snippets. The test completed successfully!
Here are the stats from the test:
That’s a lot… so what did we find?
JMeter
- 12 ms Average page return
- 7 ms Median page return
- 490 ms Max page return
- 0 errors in assertions.
- 224.7 hits/second
- 1082.1 KB/second throughput
Tomcat
There were only 2 errors in the tomcat logs:
Dec 11, 2010 11:06:34 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet dispatcher threw exception java.lang.IllegalStateException: removeAttribute: Session already invalidated at org.apache.catalina.session.StandardSession.removeAttribute(StandardSession.java:1236)
Dec 11, 2010 11:06:34 AM org.apache.catalina.core.StandardHostValve custom SEVERE: Exception Processing ErrorPage[exceptionType=java.lang.Exception, location=/WEB-INF/errors/uncaught-error.jsp] java.lang.IllegalStateException: Cannot reset buffer after response has been committed at org.apache.catalina.connector.Response.resetBuffer(Response.java:691)
Note that the performance metrics seem better than they should be since I’m testing locally and there’s no network latency. At this point I’m pretty sure the app is performing above my needs, so I have a couple of options:
- Write more functionality
- Improve quality of jmeter testing
- Improve quantity of jmeter testing
- Push existing tests even further to find out where it will break (90 threads? 120 threads?)
I’m not sure what I’m going to do yet.
SPoE: Performance and Errors Under Load
12 years
One concern I have with SPoE is that, should it get popular, it must handle traffic to a reasonable degree. Since I deal with misbehaving Java apps at work all the time, I decided to test mine and see how it behaved under load. I’m running tomcat via eclipse, trending memory usage with VisualVM, and running the test with JMeter.
I decided to start small- simply registering a new account. With only 5 pages, the test flew by and gave me a false sense of security. I upped the ante to registering a new account, logging in, viewing the user edit account page, then creating and viewing a series of snippets. That’s when things fell apart. With 15 threads, 15 seconds of startup, and 100 loops, the first failure happened at around the 700 sample mark. The failure happened when JMeter was attempting to create a snippet. The only problem is the error code that it spat back in the tomcat eclipse log:
INFO: Server startup in 7158 ms Dec 9, 2010 10:09:31 PM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet dispatcher threw exception java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter$RequestMappingInfo.bestMatchedPattern(AnnotationMethodHandlerAdapter.java:1017) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter$RequestMappingInfoComparator.compare(AnnotationMethodHandlerAdapter.java:1104) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter$RequestMappingInfoComparator.compare(AnnotationMethodHandlerAdapter.java:1) at java.util.Arrays.mergeSort(Arrays.java:1270) at java.util.Arrays.sort(Arrays.java:1210) at java.util.Collections.sort(Collections.java:159) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter$ServletHandlerMethodResolver.resolveHandlerMethod(AnnotationMethodHandlerAdapter.java:611) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:422) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:415) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:788) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:717) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549) at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:366) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:177) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:378) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:167) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:619)
As you can see, there’s nothing form com.morgajel.spoe in there. Here’s the controller snippet that fires when /snippet/create is hit:
@RequestMapping("/create")
 public ModelAndView createSnippetForm() {
    LOGGER.info("showing the createSnippetForm");
    ModelAndView mav = new ModelAndView();
    mav.setViewName("snippet/editSnippet");
    mav.addObject("editSnippetForm", new EditSnippetForm());
    return mav;
}
So wtf is causing the error? I don’t know. Now I’m not sure if I keep chasing this rabbit or I move on- on one hand, I hate hate HATE when there’s performance issues and devs ignore it; on the other, this is a side project that isn’t going into production anytime soon.
SPoE Update: What’s New?
12 years
I’ve been slowly chugging away at SPoE in my free time here and there and have added a few bits that I’d like to share.
- Welcome Page – now displays recently modified Snippets and Reviews (although reviews are unimplemented).
- Snippets – Now have a publish flag to hide snippets from the public. If a snippet is unpublished, no one can review it. Snippet lists can be paginated now.
- Accounts – users can register, activate, edit, their accounts and change and reset their passwords. They can set their proclaimed experience levels in writing and reviewing, change their email address, and add IM info.
- Annotations – username and passwords have annotated validation, despite error handling being somewhat lacking.
Again, it’s nowhere near complete, but it’s almost to the point of being usable.
SPoE: cleanup and status
12 years
I’ve been relatively quiet on SPoE because I’ve been focused. Users can now register, log in, log out, and view existing snippets. I’ve added a jquery navigation menu which looks slick in firefox but ugly in IE 7. I’ve also added wymEditor for adding snippets, despite the fact that it doesn’t save; I’m not sure if it’s the final choice, but it’ll work for now.
The card system was a complete catastrophe;Â I was too focused on learning the framework and building out parts as I figured it out. I’ve now reached a point where I have several tasks I can complete, so perhaps cards will be more useful.
The Wireframe Sketcher tool has been very useful in helping me visualize what needs to be created. The writer of the plugin deemed my previous open source work good enough to grant me a free license, to which I’m thankful.
Oh, and I switched from JBoss to Tomcat for development. Logging is broken and more of a pain, but the restart times make it worth the switch. This also allows me to get rid of my defaultDS config.
As always, you can check the status via Hudson, or play with the site at http://spoe.morgajel.com/ using the credentials jmorgan:hello. I wipe the db and deploy regularly, so don’t be surprised if things don’t always work.
Next tasks to work on:
- I want to be able to change passwords and other profile information [done]
- I want to be able to reset passwords via email [done]
- I want tomcat to properly log errors [done]
- save edited snippets [done]
- set up testng/integration testing
SPoE: Sketching in Eclipse
12 years
After my last round of mock-up tools fizzled out, I moved on to other things. Today I had a coworker inform me that Eclipse has a mock-up plugin. After giving it a spin, I have to say it’s the best of all the apps I’ve tried to use. I am amused that the solution to my problem has been sitting right under my nose this entire time. Sadly, it’s a non-free plugin, but I’m not sure what that means yet. If I can get a feel down for basic usage however, I’ll be happy.
SPoE: Dud fuses, mvc fun and Mockups
13 years
So SpringFuse was a dud. In the end it was just wasn’t a good fit for me. I’m trying to learn java and spring along the way and the library conflicts were getting out of hand. I dumped it and more or less zeroed out my checkout.
On the bright side, trying to debug springfuse taught me quite a bit about maven, java dependencies, using eclipse for debugging, and spring classes. By the time I grabbed mvc-basic from springsource samples and started looking at how it handled accounts, I had a fairly good grasp of how to not only understand spring, but debug common issues. All that said, I still know relatively nothing compared to some of the devs I know.
Finally, I found some great tools for wireframing. So far I’ve only played with Cacoo, but there are a few others I want to try. I need to start mocking up what I really want from this site.
SPoE Update: findbugs, emma and springfuse
13 years
So far, getting infrastructure in place has been much easier than actually doing any coding. As it stands now, I have added the following helpers:
- Cobertura and Emma: Code coverage.
- Findbugs: finding obvious bugs in my code
- PMD: Find other problems in the code.
So far that’s all I have- bells and whistles to distract me from doing actual work. Ideally they’ll help keep me honest about code quality, but this early in the game they don’t offer much value.
As far as development goes, I found a neat little site called springfuse which, when given a database schema, reverse engineers an entire site for you. It seemed pretty impressive, and I began using that as a baseline for development… the only problem with springfuse was that it used antiquated packages. Short off falling back to Spring 2.5, and an older version of hibernate, it would be a royal pain in the ass to get it all functional.
I’ve stripped out most of the springfuse code at this point, only keeping the base Account class and surrounding classes. If it is salvageable, I may also try and save Role and AccountRole. I’m not saying there’s anything wrong with using springfuse- it seems like a great time saver for someone familiar with spring, but I think it would cause me me grief than effort it would save.
In the mean time I’ll be reading up on spring security 3.0.x, and hope to get a simple account model in place well enough to create accounts and log in. With any luck this long weekend will help.
SPoE Roles Vs. Classes
13 years
One thing I knew going into creating SPoE was that there were two types of users, Reviewers and Authors. Authors are essentially Reviewers with an expanded view. Since I’ll be using Spring and MVC, it made sense to make Reviewer a model class and Author a child class. Then I could simply add the functionality to Author as needed.
I’m starting to rethink that; Spring Security has the concept of Roles, which I’d originally planned on reserving for regular users and admins, but now I’m thinking that it might be a superficial distinction. Perhaps I’d be better off with the following:
- Person Model: Everyone who visits the site belongs to this model. There are no roles by default
- Reviewer Role: Anyone logged in would receive this role. can view snippets and
- Author Role: Anyone who has submitted a snippet would receive this role in addition to Reviewer role
- Moderator Role: Given to a select few to make sure that people are behaving. can remove snippets and critiques.
- Administrator Role: assigns moderators, bans users, etc.
Knowing now how spring security works, this seems the better option. Any thoughts?