tag:blogger.com,1999:blog-42251244136483465282024-03-06T01:10:52.687-07:00theotherian's blogthe ramblings of a pedantic developer who is always looking for new ways to solve problemstheotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.comBlogger32125tag:blogger.com,1999:blog-4225124413648346528.post-22942757007294095342014-11-02T15:04:00.002-07:002014-11-02T15:04:33.751-07:00a java developer's interpretation of depression as one's own heap and garbage collector<h2>this will be more personal than usual</h2>
<p>Most of my posts are in the style of "here's how you accomplish x using y", but this post is going to be a significant departure. I, like many other people, suffer from depression; more so in the past few months than I have before in my life. I won't go into all the details as to why because not only are they irrelevant to the post but they're also not something I would dump on my audience.</p>
<p>Instead, I'm going to draw some similarities to the Java Virtual Machine, and how the heap and garbage collector, in my mind at least, closely resemble the ways I struggle to deal with depression.</p>
<h2>if you have no idea what the heap, garbage collector, or JVM is...</h2>
<p>Maybe you found this post by accident, but you were searching articles about depression. Maybe you use a different programming language. Maybe you're not a developer at all. In these cases, I'm going to give you an abstract, 10,000 foot view of what it is I'm associating to depression. If you already know these concepts, skip ahead to the next section. If you don't or want an entertaining analogy, read on.</p>
<p>If you indeed found this article because you're struggling with depression as well, then I'm sorry to hear that. Depression sucks, and it's crazy how debilitating something invisible to others can truly be.</p>
<p>I want you to imagine a house, filled with the typical things a house has, and a person living in it. To associate the technical ideas I'm going to delve into below, the house is the Java Virtual Machine, the person is a Java Application, and the things are the resources the person (or application) uses.</p>
<p>Some things in the house are quite large: sofas, beds, appliances. These are things people tend to keep for a long time, and are usually things a person uses frequently. Some things are small and get used and discarded frequently: food, packaging, disposable wares, etc; they're short lived possessions. A person's house is only so big, and eventually they can run out of space to put things. The more stuff you have, the longer it can take to clean your house as well. Bigger houses also take longer to clean (or are at least more expensive to clean; I suppose you could hire an army of cleaning people).</p>
<p>Some large items in your house may be things that offer little value that you decide to hold on to: that pool table you never use or that drum set you played a handful of times and abandoned after realizing the learning curve. Sometimes you buy a lot of new stuff that you plan on keeping around for a while, but there's too much large old stuff in the way to use it how you want to. It's even possible to get to a point where you don't really have room for new stuff <i>at all</i> and it's imperative you clear some stuff out, which can take serious time and effort.</p>
<p>All of this is analogous to how a Java Application runs: it's a person in a house with limited space, it needs some large things that are used frequently, but also takes in many small things that are consumed and discarded quickly or new large things that require space to exist. The problems are real too: you can run out of space and spend an awful lot of time trying to juggle what you have in a way that makes sense. In the examples below, the heap is the space you have, and the garbage collector is the process of clearing up space when you need to. In a Java application, the garbage collector will pause the application to reclaim those resources.</p>
<h2>similarities to my own mind</h2>
<p>I spent a fair amount of time thinking about this analogy, and it both amused and bothered me how much it fit the problem of depression.</p>
<p>I have plenty of experience troubleshooting Java applications that are performing poorly, often due to memory issues. Tell me what's happening, give me access to the server it's running on, and I can pull the problem apart and come up with what's wrong and how to fix it. At the risk of sounding pompous, as long as I have the right tools at my disposal, I'm <i>really</i> good at it too. Sometimes the application was holding on to large things that prevented new things from coming in. Sometimes the application had way too much disposable stuff coming in, or wasn't letting go of stuff it should dispose of. Whatever the case may be, these issues <i>prevented the application from accomplishing what it needed to do</i>, and I was able to identify the problem and correct the behavior once I understood the root cause.</p>
<p>Depression can have similar effects on my mind. There are large things that take up a significant amount of space, but really offer little value; they're a burden on the system and can prevent new things from coming in. Sometimes there's an abundance of short lived stimuli entering your mind, and all you can do is focus on those, even if they're noise: sort of like a mental DDoS. Some things are actually very disposable but you end up holding on to them because of an issue with how they're processed; I think of it almost like being a thought hoarder. Ultimately all of these things can slow you down to a crawl and prevent you from accomplishing the things you need to do.</p>
<p>These things are surprisingly similar in my mind, which ultimately begs the question: why can I see it when it's a process on a server and be completely blindsided and overwhelmed by it when it's happening in my own head?</p>
<h2>resource contention, overload, and thrashing</h2>
<p>There's a key similarity here in how the problems manifest: the resources of the system under duress are unavailable. The application isn't sorting things out itself: it may have a way of complaining that <i>something</i> is wrong via a failed health check or some error but can't adjust. Ultimately you're the one coming in, analyzing the problem, and figuring out how to adjust what's happening to make those resources available again. Meanwhile, the application is <i>struggling</i> and can't fulfill what it's designed to do; in extreme cases it becomes completely nonresponsive.</p>
<p>When you break it down, resources are being contended for and overloaded, which means they're not available as they should be for all the normal functions the application has to perform. Thrashing takes place when what little resources are left end up being consumed trying to alleviate the stress placed on the resources that are overburdened. This is what can cause things to come to a screeching halt for considerable periods of time.</p>
<p>Applications are lucky though; they can be restarted usually at little cost, and a restart can alleviate the problem at least temporarily. Not so in humans. I don't know what would even constitute the idea of a restart in that case. Moving to a new place and getting a new job? Getting baptized in a new church and going through some kind of metaphysical restart? Who knows...</p>
<p>Still though, a restart is a stop gap. Usually restarting an application only provides time before the problem comes back; the problem still hasn't been solved. The same can be true in us: you can do whatever it is you think is a "restart" but only delay the inevitable.</p>
<p><i>What can we do?</i></p>
<h2>the application has to change</h2>
<p>If you're talking about fixing an application, it means changing it. True, the literal Java application can't change itself, but I think you can see where I'm going with this.</p>
<p>The application needs to be able to manage its resources more effectively. The application has to be able to identify when considerable resources are being used for things that offer no value. The application has to understand when there's an abundance of noise coming in and try to discard it instead of hold on to it. The application has to be able to accept new things. <b>The application needs to manage its resources in a way that allows it to perform what it's designed to do.</b></p>
<p>Sometimes small design tweaks can provide big gains. Some design changes are enormous and take time. In between these changes, the application still has to run though: it performs an important duty and can't just shut down during construction so to speak.</p>
<h2>so what has to change?</h2>
<p>I keep using terms like "large things that offer no value" as well as "lots of noisy stimuli" and I'd like to expand on that, at a personal level, to provide context. I'm going to emphasize that I'm not a therapist, I'm an engineer, and that everyone's problems are different, so this is purely an example and not some kind of empirical data.</p>
<p>I struggle a lot with the unknown. I dwell on the "what ifs" and "should haves" and "don't knows", struggling to reconcile what I should or could have done differently or how I need to prepare for them. Most of them I can't do anything about though; life will always do unexpected things that you can't be prepared for. I could get sick. I could be robbed. I could lose someone close to me. Can I prevent any of it? Well, there are some things I can do like eating well or having a security system. The more important question is: <i>can I control any of it?</i> The answer to that is simply: no. Despite that answer, I allow myself to be overwhelmed. I research compulsively trying to find answers to questions I can't hope to answer, and I consume significant amounts of time, and really as a resource time means <i>my life</i>, on doing so, when instead I could spend that resource on enjoyment. The end result of this process, for me, is depression. Not only do I become depressed contemplating all these things, but the realization of how much time and effort I spent in futility depresses me as well.</p>
<p>That struggle corresponds to a few of the analogies I've been using. For me, these thoughts accumulate as a significant amount of noise; to use the term I shared before they're a "DDoS on the brain." They're also large in scope and consume a lot of resources, so much that my brain stops focusing on other large things. What do I mean by that? Sometimes I forget to eat because I'm so absorbed in dwelling on the unknown. Sometimes I spend so much time on it I don't go to the store to buy food, or talk to my friends. Sometimes I lose so much time on something of so little value I either go to sleep late or stay awake thinking about it. I would consider all of those vital functions of myself as an application that I'm unable to perform due to a lack of resources.</p>
<h2>the metaphysical gc pause</h2>
<p>I don't always manage to do this, but sometimes I'm able to realize just how much of my resources I'm consuming on, well, <i>garbage</i>, and I have an opportunity to get back in control. For me, I equate this to a GC pause.</p>
<p>Like I mentioned above for those unfamiliar, the Java garbage collector will pause the application to reclaim resources so that the application can continue to run. To do this, it has to identify what should stay and what should go, and remove the things that should go. I try to go through the same process. I look at the state of my mind and think "what is currently circulating in here that's not helping me do what I need to do?" and try to purge it. Some of it is still strongly referenced and I can't quite get rid of it yet. A fair amount of it is new enough though, that I can clear it out. I'm not saying this is easy to do; it's only something I've started to do recently and I'm slowly getting better at it. What I am saying though is that it helps me to pause, examine the context of what's happening in my head, and truly identify the garbage so that my resources are available both for what I need to do to keep running and to take new things in.</p>
<h2>so who's the person looking at the heap dumps and profiling things?</h2>
<p>One of the points I made before is that, with an application, I'm the one figuring out what is wrong with <b>it</b> rather than myself, and how that makes a huge difference in my ability to understand what's wrong.</p>
<p>If we can do that for computer programs, why can't computer programs do the same for us? I have a plan for this, and that's why I'm asking for a small donation to start my work towards creating an algorithm to implement a cerebral garbage collector that...</p>
<p>I'm kidding. :) (but be honest, you were thinking "man, wouldn't that be nice!")</p>
<p>In all seriousness, talking to a therapist has been helping me tremendously. Doing so isn't an easy step to take; in fact I avoided it for a long time and can honestly say that in my case it was due to a bunch of irrational fears. It's important to find a therapist you click with as well; who seems to have a decent grasp on who you are as a person and can communicate with you beyond just words. I was lucky in that I knew someone who recommended a therapist who was perfect for me, and I know others may not be so lucky.</p>
<p>Having a therapist has been hugely beneficial because, to come up with another programming metaphor, I was able to hand over all of my runtime statistics to him, and he was able to identify the bottlenecks and resource contention that was causing me not to run properly. Gradually I've been able to adjust to account for this and have made improvements, but I know it's a long process, and that it's important to fix the bigger problems that require major design changes and not just rely on the small tweaks.</p>
<p>I should mention too that having a therapist doesn't imply taking medication. In my case it was the total opposite: I told my therapist from the start that medication was only on the table if I arrive at an impasse that I absolutely must get past but can't without a prescription, and he was completely open minded and empathetic towards my position.. That's not a one-size-fits-all situation for anyone, nor would I ever judge anyone for agreeing or disagreeing to medicate. I have my reasons for why that works for me; I wanted to call it out for those who may fear a therapist for this reason as I once did. You have choices in how you get better, and data will help you with those choices. The most important choice though is deciding you want to get better, and taking steps to get there.</p>
<h2>the road ahead</h2>
<p>I know the "life's too short" expression is something people have heard to death by now, but I wanted to call out it here. It's not worth suffering and making yourself miserable if you can avoid it. I plan on continuing to give my therapist the data he needs, to get feedback on where the resource contention is the worst, to figure out what data is garbage, and to tune my garbage collection to make the most use of the resources I have. I know that someday, like all other "applications", I will terminate (it feels like there's a joke there about developers being nicer to operations people). When I do, I want to be remembered as an application that, while it may never run perfectly, accomplished many great things and improved the health and function of the collective system overall. I don't want to be the application that doesn't produce much because its resources are tied up doing things that don't benefit it or anything else. I don't want to be the application that stops running, but others don't notice because it prevented itself from really standing out because it was its own worst enemy.</p>
<p>I guess what I'm trying to say is: I don't want to be Internet Explorer.</p>
<p>And no, this article wasn't just a massive run towards an Internet Explorer joke. It just felt good to end on a comedic note.</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com15tag:blogger.com,1999:blog-4225124413648346528.post-27382399024676080642014-10-30T23:04:00.000-06:002014-10-30T23:04:03.040-06:00how do I select message data from my iPhone backup?<h2>don't worry, it's easy (promise)</h2>
<p>Messages on the iPhone are stored in a sqllite file inside of the backup. If you have no idea what a sqllite file is, well, then you have a little more homework to do, namely install sqllite so you can... read a sqllite file. :)</p>
<p>Once you have that installed, you need to grab the file within the latest subdirectory you can find inside the directory <code>Library/Application\ Support/MobileSync/Backup/</code> called <code>3d0d7e5fb2ce288813306e4d4636395e047a3d28</code>. This file has your texts in it. I suggest you copy it somewhere else in case you corrupt it by mistake using sqllite. Once you have it, just run <code>sqllite [your backup filename]</code> to load the data.</p>
<h2>getting the data</h2>
<p>The following query will retrieve the texts stored in this file, in a nice pipe-delimited format:</p>
<script src="https://gist.github.com/theotherian/0be39a4b303063751c85.js?file=messages.sql"></script>
<p>If there's more data you're looking for, this will tell you the columns in the table (sqllite doesn't use <code>desc</code> or <code>describe</code>):</p>
<script src="https://gist.github.com/theotherian/0be39a4b303063751c85.js?file=pragma.sql"></script>
<p>The output you get should be usable in Excel or any other number of tools if you want a different backup of your texts.</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com7tag:blogger.com,1999:blog-4225124413648346528.post-33638669854407928642014-07-14T22:37:00.002-06:002014-07-14T22:37:34.715-06:00underscan problems using hdmi with your macbook pro retina? here's a potential fix<h2>an answer buried in the middle of nowhere</h2>
<p>Just went to hook up my monitors to my Macbook Pro Retina, one with DVI to Display Port, the other with HDMI, and my problem was immediately apparently: I was in underscan town...</p>
<p>Luckily I happened to stumble on an answer online, but it was a little obscure. Mind you, my monitor had the same option as the poster's monitor (a Samsung in my case) in the post below, so your results may vary, but it fixed the issue:</p>
<blockquote>Okay, I was having the same problem... Trying to connect Macbook Pro to a 24in monitor... And I was SO disappointed I had to use the "underscanning" feature in the Displays box to make the bigger screen slightly smaller so it fit everything in there... This was horrible to me because it also made things less clear.
<br><br>
BUT I FIXED THE PROBLEM!
<br><br>
Go into your little settings within your ACTUAL MONITOR... And there's a section that talks about "Input"... Mine says "Input Select".. And where it mentions "HDMI", it gives an option of PC or AV... It was set to AV!! So I set it to PC and life became wonderful again! Spread the word. This seems to be a common problem but not everyone realizes it's the monitor.</blockquote>
<p>The answer was found at <a href="http://forums.macrumors.com/showthread.php?t=1313094">http://forums.macrumors.com/showthread.php?t=1313094</a>. I figured I'd toss a post up to try and make the solution a little more SEO relevant to those suffering from the same problem.</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com8tag:blogger.com,1999:blog-4225124413648346528.post-58972372283558323282014-05-31T18:54:00.000-06:002014-05-31T18:54:29.948-06:00caching in java: getting better performance and precision from your golden hammer<h2>inspiration</h2>
<p>This post is actually a written form of a presentation I did at a corporate tech convention. The inspiration for it came from a common tendency for caching to be treated as a golden hammer for performance: if we were executing a database query frequently, we were caching it.</p>
<p>To be blunt, <b>it's just not that simple.</b> Sure, memory is relatively cheap, and it may make things go faster out the door, but that doesn't mean you're using your resources well. It also doesn't mean your caching approach makes sense in the long run either: how effective is your hit ratio when that data grows? Sometimes people don't think about acceptable levels of staleness in data either and just blindly cache for hours when it can negatively impact the experience of your application.</p>
<p>All of these data points mean <b>there is no simple answer to caching.</b> This isn't just a one-size-fits-all solution: you have to be precise and think about how your data is going to be used, how it will grow over time, and how stale it can get. To reference <i>Idiocracy</i>, I view caching as a toilet water solution. If you haven't seen the movie, here's the gist (spoilers ahead): 500 years in the future, a man from the past wakes up from cryostasis and is the smartest man in the world. He finds out the crops are dead (I won't say why) and suggests that people start watering them instead of what they were doing. In this dystopian future, people only associate water with toilets, so the response from everyone is "Like from the toilet?" After water solves the problem, discussion of other problems leads people to suggest "Hey, let's try putting toilet water on it!" The moral of the story is this: you can't blindly apply the same solution to every problem just because it solved one that sounds similar.</p>
<p>This post is designed to suggest questions to ask rather than provide answers for everything. At the end I'll present a checklist you can try out to see if it helps you.</p>
<h2>what are the costs of caching in java?</h2>
<p>Java memory management is a massive topic by itself, so we're going to only go skin deep here. If you're not familiar with how the heap is organized, here's a diagram (note: permanent generation is going away in Java 8 as a result of being collapsed into tenured):</p>
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8se0ugy8mqa4Hsw1xmTutV6ty_KQQ_SOtpg40NDcZ_H5dh6leYH5krMHRz9jGgaZpHkvZHmXjPILNk3I05-R96E9llknork0a_Q_qgu5GE6WFYhwQ9t5H0Sc4FiZVkyUFoegIDXNxS5Es/s1600/heap.PNG" />
<p>We're not going to worry about eden versus survivor space in this post. What we're most interested in is the difference in young vs tenured, and how things end up getting stored over time. So...</p>
<ul>
<li>The young generation stores short lived objects, and is garbage collected (or GC'd) very frequently. For an example of short lived objects, think of all the objects you create during a request in a web application that are unnecessary after the request is complete. Young generally uses far less space than the tenured generation, because most applications don't create so many short lived objects that you need a massive young generation.</li>
<li>As the garbage collector scans the objects in the young generation, if it sees objects that are still in use after checking several times, it will eventually move that object over to the tenured generation.</li>
<li>As your tenured generation has more data added, the amount of memory you use will grow to the maximum number you specified in your <code>-Xmx</code> setting for your JVM instance. Once you've reached that point, the garbage collector will collect from the tenured generation when necessary.</li>
<li>Different garbage collectors have different approaches when it comes to the following statement, but at a high level, once something has to move out of young generation and into tenured, if there isn't enough space in tenured, the garbage collector has to clear some. This means your application will suffer from some pausing, and this pausing will increase (sometimes dramatically) in either frequency, duration, or both with the size of your heap. The garbage collector has to rummage through the objects in the tenured generation, see if they're still being used by your application (strongly reachable), and remove them if they're not. Sometimes it has to shift the objects that remain (compacting vs non-compacting garbage collection) if there are several small blocks of memory free, but a single large object needs to move.</li>
<li>Generally, <b>data you cached will live indefinitely in the tenured generation.</b> I won't get into other cases here (though if you're interested check out my posts on <a href="http://www.theotherian.com/2014/01/strong-soft-weak-phantom-references-java.html">other reference types</a> and <a href="http://www.theotherian.com/2013/08/understanding-javas-native-heap-or-c-heap.html">off-heap storage</a>), because mainstream stuff like Guava Cache and ehcache will put stuff here.</li>
</ul>
<p>So there you have it: ultimately the garbage collector is going to have to manage what you cache, which isn't necessarily a bad thing, but can become one if you're not careful. If you're going to force the garbage collector to do more work, you need to get value out of keeping that data in memory.</p>
<h2>so what should we cache?</h2>
<p>To the best of our ability: things that are frequently used and/or expensive to load. The former is the difference between a high and a low cache hit ratio, and we want a high cache hit ratio if possible. We need to know what data is accessed frequently or most expensive to load though in order to understand what to cache. Here are some criteria I encourage people to apply to their data:</p>
<ul>
<li>Is there some data that is explicitly more popular? For example: if we're a retail website and there's a set of products we know we're going to promote in a highly visible place, that data is going to be explicitly popular because we're directing people to it.</li>
<li>Is there some data that is implicitly more popular? For example: if we're a social media site, we probably have a bias towards new content rather than old due to the way we present it. In that case, we can prioritize newer content since we know that, at least for some period of time, it will be popular until newer content replaces it.</li>
<li>If you were to graph traffic against some unique identifier in your data, where does the <a href="http://en.wikipedia.org/wiki/Long_tail">tail</a> start? Often times the tail of the graph starts early on in the data set, and should give you an indication of what data is very valuable to cache, and what data will often be a cache miss if you were to cache it.</li>
<li>Is your cache distributed or local? If it's local, you're going to increase the cache misses to be equal to the number of hosts running that cache for any given entry. If it's distributed you may be able to cache more of the tail effectively.</li>
<li>Is the data expensive to load? If it is you may want to consider reloading that data asynchronously once it expires, rather than blocking on a reload. I've written more about blocking and non-blocking caches <a href="http://www.theotherian.com/2013/11/non-blocking-cache-with-guava-and-listenable-futures.html">in another post.</a></li>
</ul>
<p>Bear in mind the cost of loading data and preventative measures in terms of exposing yourself to a DDoS, or Distributed Denial of Service attack. Chances are you can't cache all of your data, and you should be careful to not put yourself in a position where cache misses and/or expensive load times can make your application unavailable if hit frequently enough. This subject is beyond the scope of this post, though I may write about it at some point in the future.</p>
<h2>why will data live indefinitely if cached?</h2>
<p>Interestingly, I had a discussion with a colleague about this just a few weeks ago, and it revealed a massive misconception that he and probably other people have about caching.</p>
<p>Usually, caches don't eagerly expire data by default; they do so lazily depending on their <a href="http://en.wikipedia.org/wiki/Cache_algorithms">caching algorithm.</a> Here's an example: say we have a cache that can hold 10 items, and that the max age is 30 minutes. If we load 10 items into the cache around the same time, and see how many items are in the cache 40 minutes later without having accessed them <i>at all</i> in between, we'll still have 10 items in the cache. By default, with technologies like ehcache and Guava Cache, there's no thread running in the background checking for expired entries. Since the entries are stored in such a way that they're strongly reachable, they're not eligible for garbage collection and will <b>always</b> stay in your tenured generation.</p>
<p>You may think this isn't a big deal, but it can be in practice. Let's say someone didn't pay attention when they set up a cache and decided it could hold 10,000 entries. Maybe the data set in the backing storage was small at first and no one saw a problem. Let's say that, over time, the data set grows to 10,000 entries, and that those entries are gradually hit over the runtime of the application, but only 100 or so are hit with any reasonable frequency. In that case, you're going to shove the <i>entire</i> data set in memory over time, and you're <i>never</i> going to get that memory back, even if no one accesses that data again for as long as the application runs.</p>
<p>Guava allows you to create an expiry thread that will make old data eligible for garbage collection (I haven't found the same in ehcache; comment if you're aware of this feature). Still though, you're probably better off sizing your cache more effectively for your traffic and data than you are eagerly evicting, depending on what problem you're trying to solve.</p>
<p>Bottom line: think about how much stuff you're tossing in memory, how much space it will take up, and how valuable it is to have it in memory.</p>
<h2>wrapping up for now</h2>
<p>This post is less of a deep dive into caching and more of a contextual overview of the impact of caching. I hope it helps clear up some confusion that I know I've run into multiple times. The goal of this post is really to identify why caching can actually go wrong over time, and why all of us really need to put thought into how we're using memory. The golden hammer principle can often apply to caching, and if you're not careful, you can really hurt yourself if you're swinging that hammer around a little too freely. :)</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com4tag:blogger.com,1999:blog-4225124413648346528.post-56899140491359977612014-04-30T18:47:00.000-06:002014-04-30T18:47:21.630-06:00pro tip for developers: take some time to code up some well known data structures and algorithms yourself<h2>remember all that stuff you learned in school?</h2>
<p>It's probably stuff you take for granted when you use data structures built into higher level languages like Java, Scala and C# or libraries such as Apache Commons, Google Guava, and the like. Ultimately that stuff is the foundation of your knowledge as a developer: you don't think about it all the time but the fundamentals are ingrained in you.</p>
<p>That said, I bet if you cracked open a textbook today and decided to read about one of the classics, it may stump you for a bit. Do you remember the details of Dijkstra's algorithm? You may remember it's used for shortest paths in a graph, but you may not remember how it works. What about hash based data structures? Do you remember what makes a good hash function, or different strategies for handling hash collisions? How about string sorting or searching approaches?</p>
<p>If some of these aren't fresh in your head, you shouldn't feel bad. You may have a notion of how they work on a high level still, along with the computational and storage impacts, but you may not remember how to implement them offhand. Going back a few sentences, if you've ever reread these from a textbook, I'm willing to bet you didn't grasp all of it on the first pass. It's pretty dense material, and usually the best algorithms somehow cheat or have some type of trick that gets around some of the computational expense of the problem that you need to get your brain in sync with again.</p>
<p>Now you may be thinking "what's the point of all of this? I almost never have to be hands on with this in my day to day job, <i>that's</i> why it's not at the ready." This is probably the case, but it very well might not be if you're in an interview. High caliber companies will probably ask you about this kind of stuff, and may very well ask you indirectly as a word problem instead of saying "show me how a trie works."</p>
<h2>why you should take the time to write some of this code yourself</h2>
<p>Ultimately, we all learn a little bit differently, and we speak about problems in ways that only we understand. How much easier do you find your own code to read and comprehend versus someone else's?</p>
<p>For that reason, I'd recommend writing up some solutions yourself, with your own comments and flair, so that if you ever need to refer to it, you don't end up cracking open a big textbook and try to relearn from someone else's words. I had to do this in recent history for a task, we'll call it "homework", where having my own implementation helped me pull the solution out of the archived storage in my head and back into my brain's cache. Everything was familiar to me once I read my own code and my own comments; I even called out the things that tripped me up the first time I revisited the subject matter that tripped me up again when I looked at my textbook.</p>
<p>Big players will ask you this kind of stuff; Google is notorious for this and will typically suggest study guides to candidates, and there's also a famous <a href="http://steve-yegge.blogspot.com/2008/03/get-that-job-at-google.html">blog post</a> about how to prepare. I've known people who studied for <i>months</i> for Google interviews, and I'm willing to bet that they wrote out these algorithms themselves to prepare. So, given that, why not be ahead of the game?</p>
<h2>stuff worth focusing on</h2>
<p>Perhaps I've sold you on this idea. If I have, here's what I'd suggest for resources and problems to focus on to help you.</p>
<p>First, you should buy <a href="http://www.amazon.com/Algorithms-4th-Edition-Robert-Sedgewick-ebook/dp/B004P8J1NA/">Algorithms</a> by Robert Sedgewick and Kevin Wayne. It's one of the better algorithm books I've read (still dense, but reasonable), and has a <i>lot</i> of code samples and illustrations to help you understand.</p>
<p>Second, I'd recommend dabbling in the following as focus points:</p>
<ul>
<li>Time-order complexity and Big-O notation (document the code you write with this, as fine grained as you can)</li>
<li>Sorting</li>
<li>Hashing and hash based data structures</li>
<li>Trees (binary/red-black)</li>
<li>Graphs and graph traversal</li>
<li>String sorting and searching</li>
<li>Data structures with interesting properties like min heaps and tries</li>
<li>Concurrency, for example map/reduce or a blocking work queue with a poison pill termination condition</li>
</ul>
<p>All of these are pretty broad subjects, but they yield a lot in terms of the number of computer science problems related to them. Again, further emphasizing the point, writing code for the subjects above on your own gives you a persistent way to communicate to yourself in a very high fidelity capacity, and provides you some nice examples if you're tasked with having this knowledge at the ready.</p>
<p>I mean what I said about that textbook too; it's a very well written book that covers a wide variety of classic problems and algoritms, and you owe it to yourself to add it to your collection. :)</p>
<p>No matter what position you're in, I hope this advice helps you in your career in the long haul. If you have questions, or if you've done this and had it help you, please feel free to comment below!</p>
theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com6tag:blogger.com,1999:blog-4225124413648346528.post-15781224538813241332014-03-31T21:54:00.000-06:002014-03-31T22:05:17.408-06:00using spring boot, spring websockets and stomp to create a browser chat service<h2>when automagical becomes autotragical...</h2>
<p>I have a love/hate relationship with frameworks that are automagical.</p>
<p>On one hand, they can save you a lot of time and effort, preventing you from writing tons of boilerplate code, and just less code in general (every line of code is one you have to support, right?).</p>
<p>There's a dark side to this however. Often times automagical frameworks have rules, usually modeled by conventions. If you violate these, the framework will slap you across the face with a stacktrace. Sometimes the stacktrace is meaningful. Sometimes it's outright esoteric. Sometimes it tells you something that you can't accurately discern without digging into the bowels of the code <i>that you're trying to leverage without creating bowels yourself.</i></p>
<p>In this case, Spring was guilty of said crimes on several occasions.</p>
<p>Don't get me wrong; I'm quite fond of Spring. However, in this case the solution involved going roughly shoulder deep into Spring's bowels to find simple, yet not obvious solutions to problems. Here's a list of some of the problems I faced, in the event that someone out on the intarwebs is searching for a way to make sense of it all:</p>
<ul>
<li>Spring Boot, at least when I wrote my code, seems to <a href="https://github.com/spring-projects/spring-boot/issues/245">despise Tomcat 7.0.51</a> (though this appears to have been resolved now). This is particularly important, because in 7.0.51 they <a href="http://tomcat.apache.org/tomcat-7.0-doc/changelog.html#WebSocket">fixed a bug</a> that prevented pure websockets from working correctly with... Spring (among other things no doubt).</li>
<li>When using Spring's WebSocket API, their scheduling API <a href="http://stackoverflow.com/questions/21758116/why-does-spring-4-only-allow-one-taskscheduler-in-a-context">ends up hijacked</a>. If you need to add your own scheduled bean, you need to override things (as shown in my code).</li>
<li>If you have a queue for your messages, and you don't <i>have the URI start with 'queue'</i>, you can't use <u>user specific message queues</u>. Regular queues will work just fine.</li>
<li>The STOMP heartbeat configuration doesn't work for browser clients since you have to deterministically configure which clients you have that will need to support a heartbeat on the backend. I had to roll my own here.</li>
<li>You can't inject the <code>SimpMessagingTemplate</code> into your configuration class or it's bootstrapping path if you're using constructor injection. This can be important if you're trying to set up a <code>ChannelInterceptor</code> to filter out messages before they hit a resource in your application. (Also, how important is it to shave off the 'le' exactly? It <i>really</i> can't just be called <code>SimpleMessagingTemplate</code>???)</li>
<li>When I was using Spring 4.0.0.RELEASE, <i>none of this was working correctly</i>. Switching to 4.0.1.RELEASE magically fixed all my problems. I won't ask how or why.</li>
</ul>
<p>There were several other issues that I've long since repressed. I wrote this code a while ago, and became so irritated with the number of gotchas that I didn't feel motivated at all to blog about it.</p>
<p>All of that aside, Spring's implementation is still leaps and bounds better than what Jetty is offering. For reasons I can't comprehend, the people who write the Jetty API seem to think that every class should be polymorphic to every other class, making it ridiculously confusing to set up. I cannot emphasize enough how many times during my use of their API that I thought "wait... this class is an instance of that?"</p>
<h2>if you just want a working demo to play with</h2>
<p>Then feel free to <a href="https://github.com/theotherian/spring-stomp-chat">grab my code on Github.</a> Did I mention that every example I found online was broken out of the box?</p>
<h2>getting your configurations built</h2>
<p>We're going to use four different features: web sockets, security, scheduling, and auto configuration. Each one will have its own configuration class, as shown below:</p>
<h4>auto configuration, aka spring boot</h4>
<script src="https://gist.github.com/theotherian/9906304.js?file=Application.java"></script>
<h4>security, because we need to log in to chat right?</h4>
<script src="https://gist.github.com/theotherian/9906304.js?file=SecurityConfig.java"></script>
<h4>scheduling, because we have a heartbeat to schedule with the web socket api</h4>
<script src="https://gist.github.com/theotherian/9906304.js?file=ChatSchedulingConfigurer.java"></script>
<h4>web sockets, because obviously</h4>
<script src="https://gist.github.com/theotherian/9906304.js?file=WebSocketConfig.java"></script>
<p>There are a few things worth pointing out in this configuration. The <code>/chat</code> and <code>/activeUsers</code> paths will correspond to Spring MVC controllers, while the <code>/queue</code> and <code>/topic</code> paths will correspond to the messaging API's pub/sub model.</p>
<h2>designing the application</h2>
<p>Before we get into the implementation, I'd like to address the design and talk about some of the details of using STOMP here.</p>
<p>First, you may be wondering what STOMP is. In this case, it's not an off-broadway production; it stands for Simple (or Streaming) Text Oriented Messaging Protocol. There's more about this on <a href="http://en.wikipedia.org/wiki/Streaming_Text_Oriented_Messaging_Protocol">Wikipedia</a> and <a href="http://stomp.github.io/">the STOMP website</a>, but in a nutshell it's a command based protocol for sending messages, interoperable with any provider or client implementing the STOMP spec. In the case of this example, it can use web sockets or not; it doesn't matter since the spec works either way.</p>
<p>STOMP and Spring allow us to set up queues, and more interestingly user specific queues. Semantically this is what you'd want in a chat application: each user can subscribe to a queue of messages coming to them, and the front end can route them into the correct chat window. In this case, our front end will simply subscribe to <code>/user/queue/messages</code>, and on the back end Spring will create a dynamic URL to map those messages to that matches the session of the user. The nice thing here is that each user subscribes to the same queue, but they only get their own messages. Even more sophisticated is that Spring can map multiple sessions for one user, so you can be signed in at multiple locations and get the same chat. All of this maps up with Spring Security as well, so as long as you have that configured all of your sessions will be mapped to the correct user queue, all messages received by the client will show up in a chat window corresponding to the other person in chat.</p>
<p>STOMP and Spring also allow us to set up topics, where every subscriber will receive the same message. This is going to be very useful for tracking active users. In the UI, each user subscribes to a topic that reports back which users are active, and in our example that topic will produce a message every 2 seconds. The client will reply to every message containing a list of users with its own heartbeat, which then updates the message being sent to other clients. If a client hasn't checked in for more than 5 seconds (i.e. missed two heartbeats), we consider them offline. This gives us near real time resolution of users being available to chat. Users will appear in a box on the left hand side of the screen, clicking on a name will pull up a chat window for them, and names with an envelope next to them have new messages.</p>
<h2>writing an implementation</h2>
<p>Since we're writing a chat client, we need some way of modeling a message, as shown below:</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=ChatMessage.java"></script>
<p>We also need a place to send messages to; in this case a controller just like you would create in Spring MVC:</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=MessageController.java"></script>
<p>There's some interesting stuff happening here. First, we're injecting an instance of <code>SimpMessagingTemplate</code> into our controller. This class is what allows us to send messages to individual user queues via the method <code>convertAndSendToUser</code>. In our controller we're also assigning the sender ourselves based on the session information that Spring Security has identified, as allowing the client to specify who the sender was produces the same problem the email spec currently has. It's worth noting here that the message is sent to both the sender and the recipient, indicating that the message has passed through the server before being seen in the sender's client's chat window. We ignore the case of sending a message to the recipient in case you're messaging yourself for some reason so that you don't get double messaging (though I should probably just ditch that case altogether).</p>
<p>Lastly, I'd like to draw your attention to the <code>@MessageMapping</code> annotation, which binds the method the annotation is on to the path we set up in our configuration class earlier.</p>
<p>That's actually all we need server side to send and receive messages believe it or not. Next is determining who's signed in.</p>
<p>We're going to start with a controller that receives a heartbeat from users who are signed into chat:</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=ActiveUserController.java"></script>
<p>In this case, we're receiving a message that contains information in the header about who the user is, and mark their most recent heartbeat in a class called the <code>ActiveUserService</code>, which can be seen below:</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=ActiveUserService.java"></script>
<p>When we call the <code>mark</code> method, we set an updated time for when the user last checked in, which is stored in a Google Guava <code>LoadingCache</code> instance. We also have a method that will aggregate the user names of all the active users, based on the metric of them checking in within the last 5 seconds. This is the information we want to send back to our topic, which is handled with the code below:</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=ActiveUserPinger.java"></script>
<p>In this class, we combine the service and the template to send a message to a topic every two seconds, letting all clients know which users are available to chat. In return, users can reply when receiving a message, telling the server they're still active, creating a round trip heartbeat.</p>
<h2>putting it all together with a user interface</h2>
<p>And here's where I step out into the realm of things I actively try to avoid: creating a user interface. I am using jQuery here, which makes the task far easier, but I'm still admittedly not great in this department.</p>
<p>Before we go down that path, it's worth calling out that I used two JavaScript files that were present in the <a href="http://spring.io/guides/gs/messaging-stomp-websocket/">Spring guide for using their messaging API with websockets</a>: sock.js and stomp.js. The first sets up the connection to the server, while the second implements the STOMP spec. They're both <a href="https://github.com/spring-guides/gs-messaging-stomp-websocket/tree/master/complete/src/main/resources/static">available for download (and originally found) at Github</a>.</p>
<p><b>Warning and full disclosure:</b> I'm a <i>lousy</i> JavaScript developer. The code below is likely horribly inefficient and could probably be rewritten by a UI developer in about 10 lines. That said, it does <i>work</i>, and that's good enough for me for now. If anyone reading this would like to advise me on improvements I'm open to feedback. I'll probably try to clean it up a bit over time.</p>
<h4>connecting</h4>
<p>Before we can do anything, we need to connect (this fires when the document is ready):</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=connect.js"></script>
<p>In this method, we do three very important things:<p>
<ol>
<li>We capture the user name for the session (this is used for figuring out who we're chatting with and for coloring text in the chat window).</li>
<li>We subscribe to a queue that's bound to our user, and call <code>showMessage</code> when we receive one.</li>
<li>We subscribe to a topic that sends out all the active user names, and call <code>showActive</code> when we get an update.</li>
</ol>
<p>There's something slightly peculiar here worth calling out: notice that we construct the <code>SockJS</code> instance with <code>/chat</code> as the URL, yet you may notice that the other JavaScript below sends messages to <code>/app/chat</code>. Spring seems to have some kind of handshake here that I don't fully understand. If you change the value in this constructor, it will fail saying that <code>/app/chat/info</code> doesn't exist, so Spring appears to expose the resource <code>/chat/info</code> here to facilitate a handshake with SockJS.</p>
<h4>showing a message</h4>
<p>Whenever we receive a message, we want to render it in the chat window for that user, which is done in the code below:</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=message.js"></script>
<p>First, we figure out which chat window we should load based on comparing the recipient to our username. We then create a span with the new message, colored to indicate the sender, append it to the window, and scroll to the bottom of the window to show the latest message. If we're updating a chat window that's not currently visible to the user, we render an envelope character next to their name in the user list to indicate that there are pending messages. Below is the code to create the envelope:</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=newmessage.js"></script>
<h4>getting the chat window</h4>
<p>In the example above we obtain a chat window to update with message, but we have to create it if it doesn't already exist, as shown below:</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=window.js"></script>
<p>We create a div to wrap everything, create a div for displaying message, and then create a textarea for typing in new messages. We automatically hide created chat windows if another one already exists, because otherwise we'd interrupt the user's current chat with another person. We create binds for two events: hitting 'enter' and clicking 'Submit'. In both cases, since the event references a DOM object that has a unique id referencing the user you want to send the message to, I'm capturing that and using it to route. JavaScript's variable scoping is something I often mix up in practice, so I'm relying on the DOM to tell me who to send the message to instead. The method for doing this is shown below:</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=send.js"></script>
<p>We send the message to <code>/app/chat</code>, which routes to our <code>MessageController</code> class, which will then produce a message to the appropriate queues. After sending the message, we clear our the textarea for the next message.</p>
<h4>showing active users</h4>
<p>As I mentioned above, whenever the server tells us about the active users, the client reports back that it's active, allowing us to have a heartbeat. The first method in the code below sends the message back to the server, while the second renders the active users:</p>
<script src="https://gist.github.com/theotherian/9906304.js?file=active.js"></script>
<p>There's a lot going on here, so let me break it down:</p>
<ul>
<li>First, capture who was previously selected.</li>
<li>Second, capture which users had pending messages. We'll want to re-render this with the new list.</li>
<li>Create a new <code>div</code> for the next user list.</li>
<li>As we process users, preserve the one that was previously selected by assigning the appropriate CSS class (<code>.user-selected</code>)</li>
<li>Bind a click event to each user that will hide the current chat window, removed the select class from all users and show them as unselected, remove the pending messages status, pull up the chat window for that user, and mark that user as selected.</li>
<li>If a user was added to the new list that previously had messages pending, redisplay the envelope icon.</li>
</ul>
<p>There's really almost no HTML to this example; virtually everything is generated as messages come in. I opted for a lot of re-rendering of data to avoid an overabundance of conditional checking, though I don't know if this is good in practice or not: again, I'm not particularly good with JavaScript</p>
<h2>try it out!</h2>
<p>Like I mentioned, this is <a href="https://github.com/theotherian/spring-stomp-chat">available on Github</a> for you to play with. I know there's a <i>lot</i> covered in this blog post, and I'll probably update it several times to fill in gaps as a read it and/or get feedback. I hope this helps you get started using Spring and STOMP for doing some pretty sophisticated messaging between the browser and the server.</p>
theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com33tag:blogger.com,1999:blog-4225124413648346528.post-84358378936266441032014-02-28T22:41:00.000-07:002014-02-28T22:41:09.754-07:00using jackson mixins and modules to fix serialization issues on external classes<h2>the problem</h2>
<p>Let's say you have some classes coming from another library that you need to serialize into JSON using Jackson. You can't manipulate the source code of these classes one reason or another, and you have a problem:</p>
<p><i>These classes don't serialize correctly</i></p>
<p>There are a number of reasons this can happen, but in this post we're going to focus on two examples: two classes that have a recursive relationship to one another, and a class that doesn't conform to the bean spec.</p>
<h2>dealing with recursive relationships</h2>
<p>Let's say we have two classes, <code>User</code> and <code>Thing</code>, as shown below. <code>User</code> has a one to many relationship with <code>Thing</code>, and <code>Thing</code> has a many to one relationship back to its parent, <code>User</code>:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=User.java"></script>
<script src="https://gist.github.com/theotherian/9285448.js?file=Thing.java"></script>
<p>Given these classes, let's say in a unit test we create users using the following code, establishing the recursive relationship:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=CreateUser.java"></script>
<p>Now that we can create a user, let's try serializing:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=SerializationFailureTest.java"></script>
<p>If you run that method, your test will fail, and you'll see a nice CPU spike on your computer because this just happened:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=serialization-stacktrace"></script>
<h2>fixing recursive relationship issues with mixins</h2>
<p>As it turns out, Jackson has an awesome feature called mixins that can address this type of problem (remember, we're assuming <code>User</code> and <code>Thing</code> are not modifiable).</p>
<p>Mixins allow you to create another class that has additional Jackson annotations for special serialization handling, and Jackson will allow you to map that class to the class you want the annotations to apply to. Let's create a mixin for <code>Thing</code> that specifies a <code>@JsonFilter</code>:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=ThingMixin.java"></script>
<p>Now you might be thinking, "What's that 'thing filter' string referencing?" We have to add this mixin to the object mapper, binding it to the <code>Thing</code> class, and then we have to create a filter called "thing filter" that exludes <code>Thing</code>'s field of <code>user</code>, as shown in the test below:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=SerializationSuccessTest.java"></script>
<p>If you run this test, you'll see that it passes.</p>
<h2>dealing with a class that doesn't conform to the bean spec</h2>
<p>Let's say we have two other classes, <code>Widget</code> and <code>WidgetName</code>. For some reason, the person who wrote <code>WidgetName</code> decided to not conform to the bean spec, meaning when we serialize an instance of <code>Widget</code>, we can't see data in <code>WidgetName</code>:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=Widget.java"></script>
<script src="https://gist.github.com/theotherian/9285448.js?file=WidgetName.java"></script>
<p>Let's say we're creating widgets using the code below:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=CreateWidget.java"></script>
<p>If we try to create a widget like this and serialize it, we won't see the name. Here's a test that will serialize:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=ModuleFailureTest.java"></script>
<p>And here's the output:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=unserializable-field-output"></script>
<h2>fixing classes that don't conform to the bean spec with modules and customer serializers</h2>
<p>We can address this problem pretty easily using a Jackson module that provides a custom serializer for <code>WidgetName</code>. The serializer can be seen below, and it uses the <code>JsonGenerator</code> instance to write the value from the <code>WidgetName</code> argument:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=WidgetNameSerializer.java"></script>
<p>We now need to wire this up in order to use it. Below is a test that creates a <code>SimpleModule</code> instance, wires up our custom serializer to it, and registers the module within our <code>ObjectMapper</code> instance:
<script src="https://gist.github.com/theotherian/9285448.js?file=ModuleSuccessTest.java"></script>
<p>If you run this test, you can see the correct output for the serialization:</p>
<script src="https://gist.github.com/theotherian/9285448.js?file=successful-serialization"></script>
<h2>conclusion and resources</h2>
<p>All the resources used in this example can be found on Github at <a href="https://github.com/theotherian/jackson-mixins-and-modules">https://github.com/theotherian/jackson-mixins-and-modules</a>. If you end up having to serialize data that's outside of your control and is causing you problems, you should be able to get a lot of mileage out of these two solutions.
theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com2tag:blogger.com,1999:blog-4225124413648346528.post-23166307096526914222014-01-23T23:25:00.000-07:002014-01-23T23:25:39.652-07:00writing your own tech blog part 1: the first 10k hits are the toughest<h2>a milestone and a retrospective</h2>
<p>Today, my blog exceeded 10,000 views. I'll concede that some of them are spam/bots, but most aren't. So, in a flagrant self endorsement, I'm going to write a blog post about writing blog posts. It's like Inception but your dead wife isn't trying to kill you in your subconscious.</p>
<p>I'm pretty excited that I've been able to not just hit that milestone, but that I've been keeping with it and getting some comments and feedback. I had a lot of motivation and ideas when I started writing, and I've had some opportunities to figure out what works, what doesn't, and some goals that work well. I wanted to share these in an effort to help others who are thinking about writing a blog, or have one and are stuck or not feeling motivated.</p>
<h2>things that work</h2>
<p>Let's start off on a positive note: ways to succeed personally and publicly with this. Here's a list of observations and advice that I found worked well for me and was reflected in the comments I received and the page views I've attracted.</p>
<ul>
<li>Try picking topics or problems that are somewhat mainstream, but also something people tend to get stuck on. For example, the post I wrote about <a href="http://www.theotherian.com/2013/07/creating-resource-filters-with-jersey.html">Jersey 2.0 resource filters</a> gets a decent amount of traffic, and it's the kind of thing that people often set and forget or run into issues getting started with. I found the documentation around this feature of Jersey to be lacking and had to do a fair amount of trial and error to get things working correctly, so I figured others had the same problem. I also use Jersey a lot, which brings me to my next point...</li>
<li>Pick things you're interested in. This probably sounds obvious, but I want to emphasize it. If you write about something you're genuinely interested in, it'll help with your motivation. It can also help broaden your horizons, because as you're writing you may start to think of new directions and features you want to explore. Sometimes it'll help you solve a problem, which leads me to...</li>
<li>If you solved a problem that seemed tricky to figure out, like it was something you'll forget and need again, or can help someone else in the same boat, just write about it. Seriously; make yourself a quick set of rough notes in a text editor and write about it later. I was surprised at how often I came back to one of my posts by doing this, and it's helped me professionally in that I've referred coworkers to my blog for guides on how to do certain things. You may be seeing a pattern here, because my next point is:</li>
<li>When you write a post, try to be as complete as possible with your examples and resources. Usually I try to do things with Maven and try to include a pom file which makes it easier to reproduce my work on your machine should you choose to mess with it. Try to avoid leaving out steps, or better yet, after you post something, start from a blank slate in an IDE and try running through your example; you may be surprised by what you realize you left others to figure out on their own.</li>
<li>Spend some time on self promotion and SEO. I typically tweet my blog posts, often times to coworkers to help get feedback since I'm lucky enough to work with some incredibly smart people. I also look at what people search on that leads them to my site, and what I search for when I'm researching for a post. Sometimes I will custom craft the URL for a post to try and get the most lift. As one example of having strong SEO, if you search for Jersey 2.0 filters, my post is the 4th result after a java.net article and two dzone articles. Against titans like that, I'll consider 4th place an achievement. (On an amusing note, my colleague Sanjay's post is 8th in the results. I'll be sure to tease him about this tomorrow, hehe)</li>
</ul>
<h2>setting goals</h2>
<p>There are really two goals that I would advocate you take into consideration when writing a blog. I'm sure you'll have lots of goals, but there are two I think are particularly important.</p>
<ol>
<li>In the words of Interpol, <i>pace is the trick</i>. Set some <i>modest</i>, achievable goals for yourself, and don't try to go at your blog at some frenetic pace. You don't want to burn yourself out; you want to find that balance of productivity and desire that gives you a steady flow of work. For me, I set a goal of writing one post a month. Some months I don't write one, some I write two. By doing that though, I feel like I can achieve that goal and never end up dreading it. I don't feel like I'm setting an unreasonable goal, and I don't feel like I'm being lax on myself: it's enough to keep things moving.</li>
<li>Once you get into a groove and get some solid material up, <i>your blog can be a massively important extension of your resume</i>. I've had multiple interviews since putting my URL in the header of my resume, and in several of them one or more technical interviewers told me "I took some time to look at your blog, and I really like the work you shared." It's a serious advantage to your cause, because you're demonstrating expertise in your field. I see 8+ page resumes all the time, and they're a nightmare to deal with. Often times they're crafted to get past filters, listing every technology and buzzword possible, but really they're just a massive obfuscation of what the candidate is good at. If you can stick with a blog and show what you're made of, that 1-2 page resume can focus on a smaller, stronger set of accomplishments, and your many useful, well written blog posts can do the rest of the talking. More than just a manifestation of skill, it demonstrates that you care about your craft, and are contributing something that can help others.</li>
</ol>
<h2>these ways lead to madness</h2>
<p>There are ways to put yourself on the road to failure as well. Some of these are the counter case to my points above and may have already been inferred by you, but I think they're still worth discussing.</p>
<ul>
<li>Don't overdo it. Like I said before, find a pace that works for you, but more importantly recognize what pace doesn't. For a while I tried to do two posts a month, and I started to feel burdened with it. Luckily I was able to realize this and have since backed down (as I said I shoot for one a month), but I think had I not realized this I may have put myself off.</li>
<li>Avoid overcomplicated material or examples. If you want to do something large, break it up into smaller manageable parts. I did this with my multipart guide on Maven archetypes, because writing all of that at once would have been repulsive. I broke up the subject into three logical portions that weren't so short that they added no value but also not so long that people gave up on them.</li>
<li>Don't skimp on research for your posts. If you're writing something and you think someone may have already solved it in a different way or you feel like you're missing something, <i>dig deeper</i>. I've had multiple experiences where I was pretty far into writing a post and felt something was off where that turned out to be true. Often times I spend 10-20 hours researching something before I write about it to try and understand as much as possible before I publish. Throwing some half-baked scribble up just to add a post could discredit you in the eyes of your audience.</li>
<li>Try not to ignore, disregard, or fight in comments. I'll concede that I'm guilty of this in one case: I replied but didn't follow through on uploading more of my code to help someone. You want people to contribute back to you; sometimes they may offer additional resources you didn't know about, other times they may be struggling with something you wrote and it could help you realize that you glossed over some important details. If people read your blog and just see a bunch of comments with no answer they could think that you don't care or have abandoned the blog. You could get the occasional troll as well. If you do, try to keep things civil as your blog is a reflection of you. People may argue or even be combative; try to do your best to remain objective. If there's no resolution to the argument, there's no shame in calling that out. Saying "I think we just have two different ways of approaching this" or "I understand your point, but I don't agree with it. I do appreciate you commenting though" is a perfectly professional thing to say and can wrap things up just fine.</li>
</ul>
<h2>next stop: 100k (hopefully)</h2>
<p>I hope sharing these observations and opinions helps. I'm a big fan of knowledge sharing and seeing developers help and contribute to one another, and I think blogging is a fantastic way to accomplish that. I encourage anyone who takes their career as a developer seriously to give writing their own tech blog a shot; you may be surprised just how much you'll learn in the process.</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com1tag:blogger.com,1999:blog-4225124413648346528.post-26915947653340793472014-01-20T22:03:00.001-07:002014-01-20T22:03:29.706-07:00random things I learned about solr today<h2>sometimes I just go where the search results take me</h2>
<p>I was doing some research on SolrCloud tonight, and wound up learning about enough disparate things that I figured I'd put together a quick page summarizing what I'd read along with some links. If nothing else this post is going to just end up being notes for my own memory, but if somehow this helps someone else along the way all the better.</p>
<h2>so, what did you learn?</h2>
<h4>regarding data consistency and cap theorem</h4>
<p>SolrCloud (or distributed Solr) claims to use a CP model for data, which surprised me. CP means consistent and partition tolerant and refers to CAP theorem; if you aren't familiar with it you should read about it. The more I read about this though, I would disagree that "CP" is correct unless my understanding of CAP is flawed.
</p>
<p>
According to <a href="http://www.lucenerevolution.org/sites/default/files/lucenerevolution%20solr%20cloud%20talk%5B1%5D.pdf">this</a>, SolrCloud "is a CP system - in the face of partitions, we favor consistency over availability." <a href="http://lucene.472066.n3.nabble.com/consistency-in-SolrCloud-replication-td4020379.html">This discussion </a> gets things a little more clear, clarifying that SolrCloud "favors consistency over availability (mostly concerning writes)."
</p>
<p>To expand on what this means, you need to have at least a high level understanding of Solr's sharding capabilities, which is about all I have at this point. When you shard, you have a leader for certain documents as well as replicas. When you go to update a document, Solr will route the request to the leader and then propagate the change to the replicas. If you happen to look up data from replicas as well as the leader, then you'll actually be using an eventual consistency model. One request that hits the replica can get a stale document compared to what the leader has if the leader hasn't finished distributing an update to the replicas in the event of a real time get.
</p>
<p>The "A" is missing in this equation because it's possible that update requests will be rejected under certain conditions. SolrCloud uses ZooKeeper to elect a leader, and ZooKeeper will not allow a split brain condition to happen if part of the cluster goes down. If ZooKeeper doesn't agree on a leader due to a partition of the cluster and a potential split brain condition, update requests will be rejected, i.e. availability is sacrificed in favor of remaining consistent and being partition tolerant. However, <i>availability is still maintained for read operations</i>; the cluster will not reject those requests unless you've partitioned in such a way that there's no shard or replica corresponding to a particular document.</p>
<p>To wrap things up, I found the assertion of a CP model surprising when it's using the same eventual consistency model that AP data stores use such as CouchDB. To Solr's credit, changes should be distributed to replicas extremely fast and soft commits happen within seconds meaning the eventual consistency window is quite small, so the odds that it will create a problem are small.</p>
<h4>soft commits, hard commits, real time gets and the transaction log</h4>
<p>This is merely a terse summary of the documentation around <a href="https://cwiki.apache.org/confluence/display/solr/RealTime+Get">real time gets</a> and <a href="https://cwiki.apache.org/confluence/display/solr/Near+Real+Time+Searching">near real time searching</a>, but since it falls under the "things I learned and may likely forget tomorrow morning" umbrella I'm writing about it.</p>
<p>First, it's important to call out that when you update a document in Solr that doesn't make it automatically available within searches. As of Solr 4, you can access a fresh version of a resource after it's been updated by using a real time get as long as you have the transaction log enabled. The transaction log is not unlike what databases use to track changes, and to be honest Solr can behave more like a database than I thought as a result of this feature. Enabling real time gets makes Solr behave more like a NoSQL database.</p>
<p>If you've updated a document, then you have two options to make the changes searchable: a hard commit or soft commit. A hard commit is expensive: it pushes changes to the file system (making them persistent) and has a significant performance impact. A soft commit is less expensive but not persistent. All updates are persistent if you have the transaction log enabled. According to Solr's documentation, it's reasonable to have soft commits automatically happen within seconds while hard commits are restricted to a much longer interval (maybe 10-15 minutes).</p>
<p>You need to be aware of a few things when using the transaction log, as documented <a href="http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/">here</a>. First, all your updates are written to the transaction log before a successful response is returned to a client. Second, performing a hard commit will persist all changes in the transaction log. Third, <i>not</i> performing a hard commit periodically can result in having a huge transaction log that can potentially kick the crap out of your Solr instance on startup should it try to persist changes potentially on the order of gigs. So, keep an eye on how large you're allowing your transaction log to become, lest you send Solr into a tailspin on startup.</p>
<h4>block joins make searching relational</h4>
<p>If you've ever wanted a nice parent-child relationship on your documents, it's here. I'm not going to talk about this too much myself because I have a tenuous understanding of how to query this in Solr so far, and there are awesome resources <a href="http://blog.mikemccandless.com/2012/01/searching-relational-content-with.html">here</a>, <a href="https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-BlockJoinQueryParsers">here</a>,
<a href="http://blog.griddynamics.com/2013/09/solr-block-join-support.html">here</a> and <a href="http://www.slideshare.net/MarkHarwood/proposal-for-nested-document-support-in-lucene">here</a>. One thing worth calling out is that apparently this won't work correctly in JSON until version 4.7 according to <a href="https://issues.apache.org/jira/browse/SOLR-5183">this jira ticket</a>.</p>
<h2>that's it for now</h2>
<p>There's a lot more I'm planning on reading up on regarding Solr in the next few weeks, meaning there's a decent chance of more posts like this as well as in-depth follow ups to help people get started with certain features. In the meantime, feel free to share anything you think I or others should dedicate some time to learning about Solr next!</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com6tag:blogger.com,1999:blog-4225124413648346528.post-72444836001469172632014-01-01T14:12:00.000-07:002014-01-01T14:12:25.851-07:00strong, soft, weak, and phantom references: the double rainbows of java<h2>did you just make a double rainbow metaphor?</h2>
<p>Yes, because like the <a href="http://www.youtube.com/watch?v=OQSNhk5ICTI">double rainbow video</a>, you may look at soft, weak or phantom references and say to yourself "what does it mean?!" It's easy to mix up soft vs weak if you're new to them, and it's also easy to be confused by soft references since they are often referred to as a "poor man's cache." Phantom references, on the other hand, offer such a different type of functionality that you'll probably never need to use them... unless you need to at which point they're literally the only thing that can do what they do in Java.</p>
<p>Hopefully, by the end of this post, you'll just look at them and say "full on!" (another video reference). This isn't even a triple rainbow; this is a quadruple rainbow we're dealing with. The goal is to help provide a clear but concise summary of what these do, when you would use them, and what their impact is on the JVM and garbage collector.</p>
<p>Before we continue though, a few words of warning about these types:</p>
<ul>
<li>These types have a direct impact on Java's very sophisticated garbage collector. Use them wrong, and you can end up regretting it.</li>
<li>Since these expedite when something is eligible for garbage collection, you can end up getting null returned when you weren't previously.</li>
<li>These should only be used in specific cases, where you're absolutely positive you need the behavior they offer. You should be no means look at these and see a general replacement for what you're doing or a myriad number of ways to change your code.</li>
<li>If you're going to use these at all, do a code review with someone else, preferably a senior developer, principal developer, or architect. They're powerful, impactful, and easy to use incorrectly, so even if you quadruple checked your code and think to yourself "Nailed it!", have someone else go over it with you; a pair of fresh eyes on code can make all the difference in the world.</li>
</ul>
<h2>strong references, and a brief jvm crash course</h2>
<p>Any reference (or object) you create is a strong reference; this is the default behavior of Java. Strong references exist in the heap for as long as they're reachable, which means some thread in the application can reach that reference (object) without actually using a <code>Reference</code> instance, and potentially longer depending on the necessity for a full GC cycle. Any reference you create (barring things that are interned, which is another discussion) is added to the heap, first in an area called the young generation. The garbage collector keeps an eye on the young generation all the time, since most objects (references) that get created are short lived and eligible to be garbage collected shortly after their creation. Once an object survives the young generation's GC cycles, it's promoted to the old generation and sticks around.</p>
<p>Once something ends up in the old generation, the garbage collection characteristics are different. Full GC cycles will free up memory in the old generation, but to do so they have to pause the application to know what can be freed up. There's a lot more to talk about here, but it's beyond the scope of this post. Full GC pauses can be <i>very</i> slow depending on the size of your heap/old generation, and generally only happen when its absolutely necessary to free up space (I say generally because the JVM's <code>-client</code> and <code>-server</code> options have an effect on this behavior). An object can exist in the old generation and no longer be strongly reachable in your application, but that doesn't mean it's necessarily going to be garbage collected if your application doesn't have to free up memory.</p>
<p>There are multiple reasons why the JVM may need to free up memory. You may need to move something from the young generation to the old, and you don't have enough space. You may have an old generation that's highly fragmented from many small objects being collected, and your application needs a larger block of contiguous space to store something. Whatever the reason, if you can't free up the space you need in the heap, you'll be in <code>OutOfMemoryError</code> town. Conversely, if you're low on memory you can also create conditions where you're creating a barrage of young and old garbage collection passes, often referred to as thrashing, which can tank the performance of your application.</p>
<h2>weak references</h2>
<p>I'm going to deviate from the canonical ordering in the title of this post and explain weak references before soft, because I think it's far easier to understand soft <i>after</i> you understand weak.</p>
<p>Think of a <code>WeakReference</code> as a way of giving a hint to the garbage collector that something is not particularly important and can be aggressively garbage collected. An object is considered "weakly reachable" if it's no longer strongly reachable and only reachable via the referent field of a <code>WeakReference</code> instance. You can wrap something inside of a <code>WeakReference</code>, which is then accessible via the <code>get()</code> method, as shown in the example below:</p>
<script src="https://gist.github.com/theotherian/8209190.js?file=WeaklyReferenced.java"></script>
<p>If the instance inside of <code>value</code> is only reachable via <code>value</code>, it's eligible for garbage collection. If it's garbage collected, then <code>value.get()</code> will return <code>null</code>. The garbage collector has a level of awareness of weak references (and for all <code>Reference</code> types for that matter) and can be more strategic about reclaiming memory as such.</p>
<p>Now, you may be asking yourself: "when would I use weak references?" Most of the other resources on the web say one of two things: <code>WeakHashMap</code> is an example of how to use them, and using them for canonicalized mappings. I think both of these are poor answers for a few reasons: <code>WeakHashMap</code> is <i>dangerous</i> to use if used incorrectly (<a href="http://docs.oracle.com/javase/7/docs/api/java/util/WeakHashMap.html">read the JavaDoc</a>), and I highly doubt that the average person who is just learning about weak references will read "use them for canonicalized mappings", slap their hand on their forehead and exclaim "Oh! Of course!"</p>
<p>That said, there's a very practical example of <a href="http://www.ibm.com/developerworks/java/library/j-jtp11225/">using weak references via WeakHashMap</a> written by Brian Goetz that I will attempt to paraphrase. When you store a key-value pair in a <code>Map</code>, the key and value are strongly reachable as long as the map is. Let's say we have a case where, once the key is garbage collected, the value should be too: a clear example of this is a parent-child relationship where we don't need the children if we don't have the parent. If we use the parent as the key to a <code>WeakHashMap</code> instance, it ends up wrapped in a <code>WeakReference</code>, meaning that once the parent is no longer strongly reachable anywhere else in the application it can be garbage collected. The <code>WeakHashMap</code> can then go back and clean up the value stored with the key by using a <code>ReferenceQueue</code>, which I explain further down in this post.</p>
<p>Previous to that paragraph, I mentioned <code>WeakHashMap</code> can be dangerous, and I'd like to expand on that. It's not uncommon that someone may think a <code>WeakHashMap</code> is a good candidate for a cache, which is likely a recipe for problems. Usually a cache is used as a means to store data in memory that has a (potentially huge) cost to load, meaning the value is what you want to have long-lived and not necessarily the key, which is probably quite dynamic in nature. If you use a <code>WeakHashMap</code> without long-lived keys, you'll be purging stuff out of it quite often, and probably cause a large amount of overhead in your application. So, if you're going to use <code>WeakHashMap</code>, the first question you <b>must</b> ask yourself is: how long-lived is the key to this map?</p>
<h2>soft references, sometimes referred to as the "poor man's cache"</h2>
<p>The differences between a <code>SoftReference</code> and a <code>WeakReference</code> are straightforward on the surface but quite complex behind the scenes. Just like the definition of "weakly reachable", a reference is considered to be "softly reachable" if it's no longer strongly reachable and is only reachable via the referent field of a <code>SoftReference</code> instance. While a weak reference will be GC'd as aggressively as possible, a soft reference will be GC'd only if an <code>OutOfMemoryError</code> would be thrown if it wasn't reclaimed, or if it hasn't been used recently. The former case is pretty easy to understand: none of your strongly referenced objects are eligible for GC and you can't grow the heap any more, so you have to clear your soft references to keep your application running. The latter case is more complex: a <code>SoftReference</code> will actively record the time of the last garbage collection when you call <code>get()</code>, and the garbage collector <i>itself</i> records the last time a collection occurred inside of a global field in <code>SoftReference</code>. Recording these two points provides the garbage collector with a useful piece of information: how much time has passed from the GC before the value was last accessed versus when the most current GC occurred.</p>
<p>Here's an example of using a <code>SoftReference</code>:</p>
<script src="https://gist.github.com/theotherian/8209190.js?file=SoftlyReferenced.java"></script>
<p>The JVM also provides a tuning parameter related to soft references called <code>-XX:SoftRefLRUPolicyMSPerMB=(some time in millis)</code>. This parameter (set to 1000ms by default) indicates how long the value in a <code>SoftReference</code> (also called the referent) may survive when it's no longer strongly reachable in the application, based on the number of megabytes of free memory. So, if you have 100MB of free memory, your "softly reachable" object may last an additional 100 seconds by default within the heap. The reason I say "may" is that it's completely subject to when garbage collection takes place. If the softly reachable referent kicked around for 120 seconds and then became strongly reachable again, that time would reset and the referent wouldn't be available for garbage collection until the conditions I've mentioned were met again.</p>
<p>Now, regarding the "poor man's cache" label...</p>
<p>Sometimes you'll find questions online where someone will ask about building a cache where data can be expired automatically, the topic of soft references will come up, and then some will be scolded and told that you should use a cache library that has Least Recently Used (LRU) semantics like ehcache or Guava cache. While both of those as well as many other caching libraries have far more sophisticated ways for managing data than just relying on soft references, that doesn't mean that soft references don't have value in regard to caching.</p>
<p>In fact, ehcache has a bit of a problem in this regard: <i>everything</i> it caches is strongly referenced, and while it does have LRU eviction, that eviction is lazy rather than eager. This means that you could have data that isn't being used sitting around in memory, strongly referenced and not eligible for GC, and not forced out of the cache because you haven't exceeded the maximum number of entries. Guava cache, on the other hand, has a builder method of <a href="http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/cache/CacheBuilder.html#softValues()"><code>CacheBuilder.softValues()</code></a> that allows you to specify that values be wrapped in a <code>SoftReference</code> instances. If you're using a loading cache, the value can be repopulated if it's been garbage collected automatically. In this case, soft references play nicely with a robust caching solution since you have the advanced semantics of LRU and maximum capacity along with the lazy cleanup of values that aren't being used frequently by the garbage collector.</p>
<h2>phantom references: the tool you'll never need until you need it</h2>
<p>Think of phantom references as what the <code>finalize()</code> method should have been in the first place.</p>
<script src="https://gist.github.com/theotherian/8209190.js?file=PhantomlyReferenced.java"></script>
<p>Similarly to <code>WeakReference</code> and <code>SoftReference</code>, you can wrap an object in a <code>PhantomReference</code> instance. However, unlike the other two types, the constructor for <code>PhantomReference</code> requires a <code>ReferenceQueue</code> instance as well as the instance you're wrapping. Also unlike the other two types, the <code>get()</code> method of a <code>PhantomReference</code> always returns <code>null</code>. So, why does <code>get()</code> always return <code>null</code>, and what does a <code>ReferenceQueue</code> do?</p>
<p>A phantom reference only serves one purpose: to provide a way to find out if its referent has been garbage collected. An object is said to be "phantom reachable" if it is no longer strongly reachable in the application and is only reachable via the referent field of a <code>PhantomReference</code> instance. When the referent is garbage collected, the phantom reference is put on the reference queue instance passed into its constructor. By polling the queue, you can find out if something has been garbage collected.</p>
<p>Extension of phantom references can be used to provide metadata about what was garbage collected. For example, let's say we have a <code>CustomerPhantomReference</code> class that has a referent of type <code>Customer</code> and also stores a numeric id for that customer. Let's also assume that we can do some resource clean up after a customer is no longer in memory in the application. By having a background thread poll the reference queue used in the <code>CustomerPhantomReference</code> instance, we can get the phantom reference back providing us the numeric id of the customer that was garbage collected and perform some cleanup based on that id. This may sound very similar to the example I provided with weak references at face value, so allow me to provide some clarification. In the case of weak references, we were making other data available to be GC'd. In this case, you may have some resource cleanup you want to perform that's functional in nature rather than just making something no longer strongly reachable.</p>
<p>Given that, it should be clear that the reason the constructor of a <code>PhantomReference</code> instance requires a <code>ReferenceQueue</code> is that a phantom reference is useless without the queue: the <i>only</i> thing it tells you is that something has been garbage collected. Still though, what about <code>get()</code> returning <code>null</code>?</p>
<p>One of the dangers of the <code>finalize()</code> method is that you can reintroduce strong reachability by leaking a reference to the instance the method is being executed from. Since <code>PhantomReference</code> will only return <code>null</code> from its <code>get()</code> method, it doesn't provide a way for you to make the referent strongly reachable again.</p>
<h2>so what do reference queues do in regard to weak and soft references?</h2>
<p>We already know that soft and weak references provide a way to have things garbage collected when they would normally be strongly reachable. We also know from phantom references use a reference queue as a way to provide feedback for when something is garbage collected, which is really the purpose of phantom references to begin with. So why would we want soft and weak references to be queued up too?</p>
<p>The reason is actually quite simple: your soft and weak references are <i>still strongly referenced.</i> That's right, you could potentially end up hitting an <code>OutOfMemoryError</code> because of an overabundance of now useless <code>SoftReference</code> or <code>WeakReference</code> instances which are strongly referenced though the value they effectively proxied was garbage collected.</p>
<p>Using a <code>ReferenceQueue</code> allows you to poll for any type of <code>Reference</code> that has been garbage collected and remove it (or set it to null). There's an example of this visible in <code>WeakHashMap.expungeStaleEntries()</code> where the map polls its <code>ReferenceQueue</code> whenever you call <code>size()</code> or whenever <code>getTable()</code> or <code>resize()</code> is called internally.</p>
<h2>additional resources</h2>
<a href="https://www.artima.com/insidejvm/ed2/gc17.html">Garbage Collection, by Bill Venners</a><br>
<a href="https://weblogs.java.net/blog/2006/05/04/understanding-weak-references">Understanding Weak References by Ethan Nicholas</a><br>
<a href="http://www.informit.com/guides/content.aspx?g=java&seqNum=680">Memory Thrashing by Steven Haines</a><br>
<a href="http://www.cubrid.org/blog/dev-platform/understanding-java-garbage-collection/">Understanding Java Garbage Collection by Sangmin Lee</a><br>
<a href="http://www.codeinstructions.com/2008/09/weakhashmap-is-not-cache-understanding.html">WeakHashMap is Not a Cache! by Domingos Neto</a><br>
<a href="http://www.ibm.com/developerworks/java/library/j-jtp11225/">Plugging Memory Leaks with Weak References by Brian Goetz</a><br>
<a href="http://jeremymanson.blogspot.com/2009/07/how-hotspot-decides-to-clear_07.html">How Hotspot Decides to Clear Soft References by Jeremy Manson</a><br>
theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com3tag:blogger.com,1999:blog-4225124413648346528.post-64068615543673746122013-12-22T23:29:00.000-07:002013-12-22T23:29:37.596-07:00kryo vs smile vs json part 1: a misguided shootout<h2>this may be my most frustrating post so far</h2>
<p>First, a little background.</p>
<p>At some point, even when you can scale horizontally, you start to examine aspects of your application that you can easily take for granted in the grand scheme of things for performance gains. One of those points when dealing with web services is serialization. There's general knowledge that Java serialization is slow, and XML is bloated compared to JSON. JSON is a pretty safe pick in general: it's readable, lightweight, and fast. That said, what happens when you want to do better than JSON in your RESTful web service?</p>
<p>A colleague and I came to this point recently, where the majority of his transaction overhead was spent unmarshalling requests and marshalling responses. This application comes under very high load, so the obvious conclusion was "well, there's a clear place to start to improve things." From there, we started looking at Apache Thrift, Google ProtoBuf (or Protocol Buffers), Kryo, Jackson Smile and, of course as a control, JSON. Naturally, we wanted to invest some time comparing these to each other.</p>
<p>I looked around online a lot at performance benchmarks and found some data dealing with Kryo, ProtoBuf and others located at <a href="https://github.com/eishay/jvm-serializers/wiki">https://github.com/eishay/jvm-serializers/wiki</a>. The data presented there was very low level, and my goal was quite literally to produce the least sophisticated comparison of these frameworks possible, ideally using the 4-6 line samples on their respective wikis. My reasoning for this was that there is likely a common case of people not investing a huge amount of time trying to optimize their serialization stack, but rather trying to seek out a drop-in boost in the form of a library.</p>
<p>This is where the frustration comes into play. My results don't quite match what I've seen elsewhere, which caused me to question them several times and revisit the benchmarks I was performing. They still don't quite match, and to be honest I'm questioning the benchmark code I linked to after <a href="https://github.com/eishay/jvm-serializers/blob/master/tpc/src/serializers/BenchmarkBase.java">discovering calls to System.gc() all over the place</a>, but I feel like I have enough data that it's worth posting <i>something</i> up here.</p>
<h2>the experiment: use cases, setup, metrics, and the contenders</h2>
<p>Let's talk about the use cases I was trying to cover first:</p>
<ul>
<li>Don't go over the network. Do everything in memory to avoid external performance influences in the benchmark.</li>
<li>Serialize an object that is reasonably complex and representative of something a web service may use.</li>
<li>Serialize objects that have both small and large data footprints.</li>
<li>Use the most basic setup possible to perform the serialization and deserialization.</li>
</ul>
<p>The setup was:</p>
<ul>
<li>Run a "warm up" pass before gathering metrics to remove initial load factors on JVM startup that won't be a constant issue, and to fragment the heap slightly to both simulate real-world conditions and not give a potential advantage to a single framework.</li>
<li>Run a series of batches of entities to gather enough data to arrive at a reasonable conclusion of performance.</li>
<li>Randomize the data a bit to try and keep things in line with real-world conditions. The data is randomized from a small data set, with the assumption being that the differences in size are small enough and the batches are large enough to get a reasonably even distribution, meaning the metrics will converge on a figure that is a reasonable measurement of performance.</li>
</ul>
<p>The following metrics were recorded:</p>
<ul>
<li>Measure the average time to serialize and deserialize a batch of 100,000 entities.</li>
<li>Measure the average size of a response.</li>
<li>Measure the average time of an individual serialization/deserialization</li>
</ul>
<p>Lastly, the contenders:</p>
<ul>
<li>Kryo, using the sample found here: <a href="https://github.com/EsotericSoftware/kryo#quickstart">https://github.com/EsotericSoftware/kryo#quickstart</a></li>
<li>Jackson Smile, using the example found here: <a href="https://github.com/FasterXML/jackson-dataformat-smile#usage">https://github.com/FasterXML/jackson-dataformat-smile#usage</a></li>
<li>Jackson JSON, using the example found here: <a href="http://wiki.fasterxml.com/JacksonInFiveMinutes#Examples">http://wiki.fasterxml.com/JacksonInFiveMinutes#Examples</a></li>
<li>Jackson Smile JAXRS Provider, which had to be inferred.</li>
</ul>
<p>The use of the Jackson Smile JAXRS provider may seem odd, but I have a good reason. The basic Smile example is only a few lines, while the Smile JAXRS provider class is almost 1000 (!!!) lines. There's a lot of extra work going on in that class, and felt it was worth comparing because 1) many people could end up using this adapter in the wild and 2) perhaps there are some optimizations that should be benchmarked.</p>
<h2>code</h2>
<p><b>All of the code used in this can be found at <a href="https://github.com/theotherian/serialization-shootout/tree/master/serialization-shootout">https://github.com/theotherian/serialization-shootout/tree/master/serialization-shootout</a></b></p>
<p>Here's a tree representation of what the entity being serialized/deserialized, Car, looks like:</p>
<script src="https://gist.github.com/theotherian/8092057.js?file=Car"></script>
<p>Here are the harnesses being used:</p>
<script src="https://gist.github.com/theotherian/8092057.js?file=KryoHarness.java"></script>
<script src="https://gist.github.com/theotherian/8092057.js?file=JsonHarness.java"></script>
<script src="https://gist.github.com/theotherian/8092057.js?file=SmileHarness.java"></script>
<script src="https://gist.github.com/theotherian/8092057.js?file=SmileJaxrsHarness.java"></script>
<h2>the results: normal size objects</h2>
<p>By normal, I mean on the smaller size; most data is in order of 10's of bytes:</p>
<script src="https://gist.github.com/theotherian/8092057.js?file=normal-object-results.log"></script>
<p>Key data points:</p>
<ul>
<li>Kryo and Smile are clearly more performant than JSON in terms of time spent and size of payload.</li>
<li>Kryo and Smile are close: Kryo performs better but Smile is slightly smaller.</li>
<li>Kryo has the fastest raw serialization/deserialization performance by a significant amount over both Smile and JSON.</li>
<li>The Smile JAXRS provider is <i>significantly</i> slower than its raw counterpart.</li>
</ul>
<h2>the results: large size objects</h2>
<p>For this comparison, I added portions of Wikipedia articles as part of the object, all equal in length:</p>
<script src="https://gist.github.com/theotherian/8092057.js?file=large-object-results.log"></script>
<p>Key data points:</p>
<ul>
<li>Kryo is best in breed by a wide margin here, handling batches in 1.2s vs 1.9s for both Smile and JSON. Serialization and deserialization are both significantly faster.</li>
<li>Variance in size is practically nonexistent between all the frameworks.</li>
<li>Smile JAXRS <i>really</i> looks like a dog here, taking 2.6s to handle a batch and showing surprisingly poor deserialization performance.</li>
</ul>
<h2>the winner: kryo (with HUGE MASSIVE caveats)</h2>
<p>Kryo clearly has some advantages here, but it also has one major disadvantage: <b>Kryo instances are not thread safe.</b> Did you hear that?</p>
<h4>KRYO INSTANCES ARE NOT THREAD SAFE!</h4>
<p>This caused me to show the same amount of rage DateFormat did years ago. BFD you may say, thinking "Just create a Kryo instance each time!" Well, what if I told you that <b>each batch of the <u>normal size objects</u> takes a whopping <u>NINE SECONDS</u> when I moved the creation of the Kryo object inside the harness' method.</b></p>
<p>No sir; if you're going to use Kryo you need to have thread local storage for your Kryo instances or you are going to be in for some serious pain. Depending on the load of your application, you may want to pre-create them as a pool within a servlet initializer that is scaled to the number of threads you have in your container.</p>
<p>Quite frankly I'm astonished that there's so much overhead encountered on an instance that isn't thread safe, but I also haven't delved into the API enough to know what the reasons are behind this. Still though, it creates some very annoying design implications that you'll need to make sure are accounted for correctly in your application.</p>
<p>Part of me would sooner call Smile the winner since it doesn't have this particular issue, but after looking at the JAXRS provider for it I'm left scratching my head.</p>
<p>However, when it comes to larger entities, Smile offered marginal improvement over JSON, whereas Kryo clearly won that round.</p>
<p>Based on the results in the first pass, I think Kryo showed the most improvement, but also a fair number of warts.</p>
<h2>next steps</h2>
<p>I'm far from finished here, but felt compelled to get something published. I plan on doing the following things next:</p>
<ul>
<li>Getting feedback from others about my approach and the data to see if I'm way off the mark.</li>
<li>Potentially benchmarking ProtoBuf here too. It's more painful to set up, but worth experimenting with to get more data.</li>
<li>Figuring why Smile JAXRS is so miserably slow.</li>
<li>Messing around with Kryo's optimization (an example of this is <a href="https://code.google.com/p/thrift-protobuf-compare/source/browse/trunk/tpc/src/serializers/kryo/KryoOptimizedSerializer.java?r=121">here</a>).</li>
<li>Looking at other BSON libraries.</li>
</ul>
<p>I do genuinely feel like I'm missing some critical piece of data or type of test here, so if you see anything that could stand to be addressed, please let me know in the comments!</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com6tag:blogger.com,1999:blog-4225124413648346528.post-68562103536570879112013-12-02T22:06:00.001-07:002013-12-02T22:06:53.388-07:00making guava cache better with jmx<h2>caching with jmx is just so much better</h2>
<p>If you've never used this before, you're missing out. Being able to remotely check statistics on your cache to measure its effectiveness, as well as being able to purge it at runtime is invaluable. Sadly Guava doesn't have this baked in the way ehcache does, but it's relatively easy to add.</p>
<p>Most of my work is a slightly different take on some work a fellow Github user named kofemann produced (<a href="https://gist.github.com/kofemann/6702006">located here</a>) which contains the JMX beans and bean registration logic. I made a few alterations to the code, pulling the registration out into a separate class (I really didn't like the bean doing all that work in the constructor) and adding a refreshAll method.</p>
<h2>taking advantage of refresh after write functionality</h2>
<p>If you've <a href="http://www.theotherian.com/2013/11/non-blocking-cache-with-guava-and-listenable-futures.html">read my previous blog post</a> about the awesomeness that is Guava's refresh after write functionality, then you'll see how it can be advantageous when it comes to JMX management. If you didn't read my post (shame on you), then it's worth calling out using refresh after write allows for asynchronous loading of cache values, meaning you never block barring the initial loading of the cache.</p>
<p>This can be used via JMX management as well by iterating through the keys of the cache and calling refresh for each one, which will load new values without causing clients of the cache to block (as opposed to purging the cache). Purging a cache is a dangerous thing to do under certain circumstances, since missing values will trigger loading events that will block clients at runtime and potentially overwhelm either your application server or even your underlying data storage. I would argue that ehcache is particularly bad because of potential read contention caused by write blocking. To clarify, several threads in your application can block waiting for cache values to be reloaded, and all of those blocking threads will then compete over a limited number of read locks after the write lock has been released, potentially causing a CPU spike and considerable latency in your application under the worst conditions. When I say worst conditions, I'm speaking from very recent and harrowing experience, so I have the lumps to say with the utmost certainty this can happen. :)</p>
<h2>the implementation</h2>
<p>For JMX you need an interface and an implementation. The interface can be found <a href="https://gist.github.com/theotherian/7540383">on my Gist</a> and doesn't really need to be shown in the post. The implementation is below; it's really a wrapper around Guava's CacheStats object and the cleanup/invalidateAll methods, as well as my refreshAll method:</p>
<script src="https://gist.github.com/theotherian/7540383.js?file=GuavaCacheMXBeanImpl.java"></script>
<p>As I said before, refreshAll has the advantage of not causing your application to potentially lock up due to cache contention; everything will load up in the background. Depending on how you have your thread pool set up for performing refreshes, you can also throttle how hard you're hitting your data store by restricting the number of concurrent fetches of data by limiting the threads available.</p>
<h2>registering your cache in jmx</h2>
<p>This is pretty straightforward: just pass your cache (in this case a LoadingCache because of refreshAll) to the method shown below and you'll expose it via JMX for statistics and management:</p>
<script src="https://gist.github.com/theotherian/7540383.js?file=LoadingCacheJMXManager.java"></script>
<h2>feedback</h2>
<p>Let me know if this works for you; I plan on using this soon in a high load environment, so I'll follow up with any results I find to help out my readers. I feel kind of bad bagging on ehcache so much recently, but it's caused me enough gray hair over the last month that I plan on focusing several blog posts around caching.</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com1tag:blogger.com,1999:blog-4225124413648346528.post-88398557325575121582013-11-14T22:48:00.000-07:002013-11-14T22:48:44.302-07:00non-blocking cache with guava and listenable futures (or why I hate ehcache some days)<h2>trying to scale? just throw more cache at the problem!</h2>
<p>Yes, I'm going to use the term "cache" in lots of ironic and financially metaphoric ways in this post. My apologies.</p>
<p>Caching makes a lot of things possible in everything we do on the Internet and on computer systems in general. That said, caching can also get you into trouble for a variety of reasons such as how wisely you use memory, how performant your cache is under contention, and how effective your cache is (i.e. cache-hit ratio).</p>
<p>If you're using Java, chances are you've heard of ehcache at some point. While there's a lot that ehcache does well, there's a particular aspect of it that in my experience doesn't scale well, and under certain conditions can take down your application. In fact, part of the reason I'm writing this blog post is the aftermath of raising my arms in the air and screaming after examining a performance issue related to ehcache which caused a failed load test <i>today</i>.</p>
<h2>mo' (eh)cache, mo' problems</h2>
<p>When reading data from ehcache, you end up blocking until a result can be returned (more on that <a href="http://ehcache.org/documentation/apis/constructs">here</a>). While this is necessary for the first time you fetch data from the cache, since something has to be loaded in order to be returned, it's probably not the case that you need to block for subsequent requests. To clarify this, if you're caching something for 1 hour and it takes 5 seconds to load, you probably don't care about loading data that's 1 hour and 5 seconds old, especially if the alternative is blocking the request to your application for 5 seconds to do it as well as every other request trying to load that data.</p>
<p>Unfortunately, and if I'm wrong here I hope someone will call me out in the comments, <b>ehcache blocks every time the data needs to be reloaded.</b> Furthermore, it uses a <a href="http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/ReentrantReadWriteLock.html">ReadWriteLock</a> for all reads as well along with a fixed number of mutexes (2048 by default), so you can end up with read contention as well given enough load. While I understand the decisions that were made and why, there are cases where it isn't ideal and you don't want to grab <i>any</i> locks to create blocking conditions.</p>
<h2>making your cache work for you</h2>
<p>To be fair, this problem really manifests itself when you have high contention on specific keys, particularly when reloading events occur. In most cases ehcache performs perfectly fine; this post isn't meant to be a general condemnation of a very popular and useful library. That said, in order to solve the problem we don't really want to block on reads or writes; we want to refresh our data in the background, and only update what consumers of the cache see when we have that refreshed data.</p>
<p>This can be accomplished by having a thread go and reload the data while the cache returns stale entries until new data is available. This accomplishes both the goals of not requiring read or write locks outside of the initial population of the cache, which is unavoidable. Even better, Guava Cache has all of this functionality baked in.</p>
<h2>refresh after write and listenable futures in guava cache</h2>
<p>Guava Cache can handle this by telling the <code>CacheBuilder</code> via <a href="http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/cache/CacheBuilder.html#refreshAfterWrite(long, java.util.concurrent.TimeUnit)">refreshAfterWrite</a> to refresh entries by calling the <a href="http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/cache/CacheLoader.html#reload(K, V)">reload</a> method in the <code>CacheLoader</code> instance used to construct your <code>LoadingCache</code> instance. The reload method returns a <a href="http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/util/concurrent/ListenableFuture.html"><code>ListenableFuture</code></a>, which is the same as a regular <code>Future</code> but exposes a method to register a callback. In this case, the callback is used to update the value in the cache once we've finished retrieving it.</p>
<p>Here's an example of this in action:</p>
<script src="https://gist.github.com/theotherian/7478882.js?file=NonBlockingCache.java"></script>
<p>The sleeps are in there to create artificial latency to show what this looks like in action. If you run this you'll see the asynchronous load events kick off and can witness the five seconds of latency in between that event firing and the data being updated. You should also notice that reads keep succeeding in the meantime. There is a small spike the first time an asynchronous load fires, which I assume is a one-time cost resource allocation within Guava Cache.</p>
<p>There is one point to consider when doing this, which is how to shut down your refresh thread. In my example I used a <code>ThreadFactory</code> (courtesy of <a href="http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/util/concurrent/ThreadFactoryBuilder.html"><code>ThreadFactoryBuilder</code></a>) to set my refresh thread as a daemon thread, which allows the JVM to shut down at the end. I also used the <code>ThreadFactory</code> to name the thread, which I would recommend as a general practice to make debugging easier on yourself whenever you're creating thread pools. In my example there aren't any resource concerns, so it doesn't matter if the thread is terminated, but if you had resource cleanup to perform for some reason you'd want to wire up a shutdown hook to your <code>ExecutorService</code> in your application since the pool would exist eternally.</p>
<p>For a use case like this, you'd want to be judicious about how many threads you're willing to allocate to this process as well. The number should scale somewhat to the maximum number of entries and refresh interval you choose so that you can refresh in a timely manner without consuming too many resources in your application.</p>
<h2>conclusion</h2>
<p>If you've come across this problem, then I hope this post helps you get past it. To reiterate what I said before, ehcache is a solid API overall, it just doesn't handle this case well. I haven't tested the Guava Cache implementation under high load conditions yet, so it's certainly possible that it has issues I've left out of the post, but from a face value standpoint it addresses the issues I've seen with ehcache in a way that doesn't involve rolling your own solution from scratch.</p>
<p>Feel free to share any feedback or things I may have missed in the comments!</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com4tag:blogger.com,1999:blog-4225124413648346528.post-90561534247482497792013-09-28T16:19:00.000-06:002013-09-28T16:19:07.967-06:00handlebars server side and client side with jersey<h2>uniting the client and the server side... sort of</h2>
<p>In this post, I'm sharing some code I've been working on to try and help address an issue I think is prone to happen with many web applications: keeping server side and client side rendering somewhat consistent.</p>
<p>Web users have grown accustomed to rich web experiences with plenty of nice interactions without page refreshes. Search engines, on the other hand, continue to require that data be rendered directly in the browser (crawlable AJAX is orthogonal to this detail) if you want to have any SEO relevance. You end up left with a dilemma: you have to render some of your data server side to keep people coming to your site, and you need to be able to render additional data client side to keep people coming to your site. While these are easily achievable using loads of technologies, keeping them consistent is slightly more challenging.</p>
<h2>handlebars and helper functions</h2>
<p>Disclaimer: I'm a <a href="http://handlebarsjs.com/">handlebars</a> fan boy, and an equally big fan boy of Edgar Espina's <a href="https://github.com/jknack/handlebars.java">handlebars-java</a>, a server-side implementation of the templating engine. I think handlebars adds <i>just enough</i> to <a href="http://mustache.github.io/">mustache</a> to get around some of the violations of DRY and provides better modularity of the view overall.</p>
<p>The biggest, most versatile tool in handlebars' shed is, in my opinion, <a href="http://handlebarsjs.com/#helpers">helper functions</a>. They provide a way to modularize view logic in a nice reusable way. A good example of this is alternating classes when rendering a list of items.</p>
<p>Helper functions get even more powerful in handlebars-java, where you can implement them in Java on the server side. Need to fetch data from the database? Knock yourself out! Need to make a web service call? Go for it! You can also use helper functions written in JavaScript on the server side as well (Edgar was nice enough to implement this after I <a href="https://github.com/jknack/handlebars.java/issues/188">made a request</a>), so you don't have to worry about your Front End Developers having their code left out.</p>
<h2>compilation and precompilation via handlebars-java</h2>
<p>This probably sounds confusing at first blush; how could there be compilation <i>and</i> precompilation? Typically for the client side, you'd want to <a href="http://handlebarsjs.com/precompilation.html">precompile</a> your templates; asking your clients to do the compilation for you is, well, a bit rude. This turns any handlebars template into a JavaScript function that will perform all the rendering you set up in your template. JSON objects are passed to the function to resolve the values referenced with the <code>{{}}</code> syntax.</p>
<p>Everything in handlebars-java is compiled via an ANTLR grammar, and all object graphs passed into it when rendering have their values resolved via reflection; no serialization into JSON takes place. Precompilation within handlebars-java does the same thing that precompilation with handlebars.js would do, except that it can render the precompiled template inline on the page for you on the server side. This may not sound like anything to write home about, but it provides several advantages:</p>
<ul>
<li>Your server can automatically recompile and re-precompile your templates as you make changes; effectively automating the process.</li>
<li>The caching mechanisms in handlebars-java are well designed, so you don't end up unnecessarily recompiling anything.</li>
<li>Upgrading handlebars.js can be automated since all templates will be re-precompiled (though this could also be undesirable since it could involve major regression testing).</li>
<li>Anything compiled server side for server side rendering can be compiled into JavaScript for client side rendering.</li>
</ul>
<p>Ultimately, you can reuse a good portion of your server side stuff on the client side as well. An example of this is presented further down.</p>
<h2>bringing jersey into the mix and mapping helper functions to resources</h2>
<p>If we're going to have all sorts of nice client side functionality, pulling down data after the page is rendered, we'll need some resources on the server side to handle it. At the same time though, if we're pulling in data via helper functions, it would be advantageous to keep the code execution paths the same. My proposal to you is that a Java handlebars helper function could shadow a Jersey resource if it's something that is expected to be available client side as well. My suggestion requires some amount of dogmatic conventions that may not be suitable for all audiences:</p>
<ul>
<li>Your <code>@Path</code> annotation and the name of the helper method should be the same.</li>
<li>The argument to your resource's GET should be the same as the argument to your handlebars function's <code>context</code> argument.</li>
<li>The helper method and the resource should exist in the same class.</li>
</ul>
<p>I think this makes more sense with an example. Let's say we have a site that allows people to post messages, and other people can subscribe to the messages they post. Since no large website has cornered the market on this yet, I figured it would provide a fun "what if" example.</p>
<p>Here's an example of what I would have written as a resource for messages:</p>
<script src="https://gist.github.com/theotherian/6746967.js?file=MessageResource.java"></script>
<h2>using handlebars templates with our... err... "functional resource"</h2>
<p>Let's take a look at what a Handlebars template rendered serverside might look like:</p>
<script src="https://gist.github.com/theotherian/6746967.js?file=home.html"></script>
<p>As you can see, we're using the server-side helper function within the <code>{{#latestmessages}}</code> tag, and passing the data to a partial called <code>messages</code> that looks like this:</p>
<script src="https://gist.github.com/theotherian/6746967.js?file=messages.html"></script>
<p>Now, ironically, the mobile example I have is larger than the desktop one because of the JavaScript involved with making the calls. Chances are that you would be dealing with a lot more data than my example shows under real conditions, so I'm asking for your forgiveness on the silliness of this disparity. That said, the mobile version:</p>
<script src="https://gist.github.com/theotherian/6746967.js?file=home.mobile.html"></script>
<p>Allow me to call out the important bits of the page:</p>
<ul>
<li>We're pulling in the handlebars JavaScript (full for the example rather than runtime) and jQuery. It's worth mentioning templates rendered server side don't have to pull in the handlebars JavaScript.</li>
<li>Inside of a <code>script</code> tag, we're calling the <code>precompile</code> function that exists server side, and precompiling our <code>messages</code> partial into the page as a JavaScript function.</li>
<li>On the server side, we're rendering a series of names we've looked up, and a link on each name. (Note: we're not loading any messages server side)</li>
<li>We're binding a function to click events on each name link that will hide other people's messages, and will then either fetch the selected person's messages or display messages that were already fetched for that person.</li>
<li>When fetching messages, we're calling <code>latestmessages</code> as a URI and appending the <code>name</code> to the URI, the same way we called <code>{{#latestmessages name}}</code> on the server side. This is the consistency that I've been trying to drive towards.</li>
</ul>
<h2>automatically mapping "functional" resources to their helper functions</h2>
<p>I'm really not thrilled at the notion of having resources representing functions; but in this case the direction I'm trying to go is having a function really just provide a means of fetching data and nothing else. Ultimately any Jersey resource maps an HTTP method to a Java method in the resource class, so in my mind they're not so far removed with the exception that handlebars effectively maintains all helper functions at a single level.</p>
<p>That aside, per another blog post I wrote about <a href="http://www.theotherian.com/2013/07/creating-resource-filters-with-jersey.html">resource filters in Jersey</a>, I'm going to create a filter that will register any resource that is also a helper function within handlebars-java and use the value in <code>@Path</code> as the name. In its current form I feel like this is a bit crude, though it is effective if you follow the convention:</p>
<script src="https://gist.github.com/theotherian/6746967.js?file=HandlebarsResourceFeature.java"></script>
<p>By doing this, you're creating a one-to-one relationship of a helper function to a resource for consistency purposes.</p>
<h2>an example project to mess around with</h2>
<p>I set up a project <a href="https://github.com/theotherian/handlebars-client-and-server-side">on Github</a> that you can use to mess around with this pattern of doing things. It's by no means perfect, and I plan on playing around with it more, but I think it can offer a fair amount of flexibility if you're willing to buy into the pattern as a whole.</p>
<p>Please feel free to share your thoughts on this approach; I'm interested to hear your feedback!</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com4tag:blogger.com,1999:blog-4225124413648346528.post-36966395938069944252013-08-17T09:32:00.000-06:002013-08-17T09:32:31.919-06:00jersey server side client and resource with in memory request handling<h2>who's to say who's crazy and who's not?</h2>
<p>This is a question from one of my favorite comedians, Otis Lee Crenshaw. He answers his own question with this (paraphrased):</p>
<blockquote><i>If you can play the guitar and harmonica at the same time like Bob Dylan or Neil Young you're considered a genius. Make that extra effort to strap a pair of cymbals to your knees and people will cross the street just to get the hell away from you.</i></blockquote>
<p>I guess what I'm saying is, that in this post, I think the cymbals may be firmly affixed.</p>
<p>I've spent a fair amount of time thinking about the problem this blog post is centered on, and I think that the solution I'm suggesting has value, though I could certainly understand people not liking it. <i>At all.</i> So, without further adieu...</p>
<h2>the problem</h2>
<p>Let's say you have a web application, and you have some data (a resource, if you will) that you're pulling in on the server side and spitting out into a view using a typical MVC pattern. For the sake of giving some context to this problem, let's say your data is messages posted by a user. Since there's no massive website that deals with this kind of data at unfathomable scale, I thought this would be a nice unique use case.</p>
<p>Your MVC application has worked well, but it could stand to be improved. At some point, you decide on pursuing either one or both of the following (since the net result is the same):</p>
<ul>
<li>There's not much value in displaying more than the last 20 messages for any given user up front on a page. Fetching additional posts on the client side via an AJAX call makes more sense.</li>
<li>This website should have a mobile presence, but data consumption is a concern. Loading smaller groups of messages on a user by user basis on demand is preferable for the user experience.</li>
</ul>
<p>In either case, a popular solution is to create a RESTful endpoint that exposes those messages. A simple call to <code>/messages/{user}?offset=x&limit=y</code> returning a nice lightweight JSON representation, and we've satisfied both cases. Problem solved!</p>
<p>Or is it...</p>
<p>You don't want to pull down <i>everything</i> this way, or at least not on the desktop site. There's value in pulling some of this data server side: your clients don't have to make additional connections to get the initial view, you don't have to consume additional resources on the server side for handling network traffic, and you may be able to more easily reuse other resources like database sessions.</p>
<p>At the same time, (I think) there's value in being consistent. Is the controller of your desktop view accessing your messages the same way your resources are? Are you sure you're serializing and deserializing to the same values client side as your are server side? You could programmatically invoke your resources and get typesafe return values, but is the behavior different when it's a request vs a method call? How are the execution paths different from one another?</p>
<p>Do they have to be different at all?</p>
<h2>an idea for a solution</h2>
<p>There's an obvious problem with making calls to the resource via an actual HTTP request when you're taking about a client and resource that exist in the same JVM: all the overhead of actually making a network connection via localhost. That may be in order of a few milliseconds, but those milliseconds add up, and <a href="http://blog.gigaspaces.com/amazon-found-every-100ms-of-latency-cost-them-1-in-sales/">studies have shown</a> end users have little patience for latency.</p> If you make enough calls, this can easily turn into 10's or even 100's of milliseconds of latency</p>
<p>As it turns out, you can avoid this network hop entirely, but it requires some (mildly kludgy) work.</p>
<p>Jersey provides a type called <code>Connector</code> for its client API. This type is used to handle connections that are created from <code>ClientRequest</code> instances. I mentioned in <a href="http://www.theotherian.com/2013/08/jersey-client-2.0-httpclient-timeouts-max-connections.html">a previous post</a> how to set up an Apache HTTP Client connector if you'd like to see an example of this.<p>
<p>More interestingly, the type <code>ApplicationHandler</code> can be injected using Jersey's <code>@Context</code> based injection system, and represents a hook into the Jersey server instance that is only a few levels removed from the methods of the servlet container itself. All of Jersey's routing logic is downstream from <code>ApplicationHandler</code>, so sending requests to it means you're largely getting the full HTTP request taste without the calories.</p>
<p>We're going to need to capture the <code>ApplicationHandler</code> instance at startup. Unfortunately, the code to do this is quite ugly in its current state. You could no doubt do this more cleanly using dependency injection, but I think you get the point. First we'll need a provider for Jersey that will allow us to capture the instance, and a way to construct <code>Connector</code> instances with that instance:</p>
<script src="https://gist.github.com/theotherian/6237777.js?file=ServerConnectorProvider.java"></script>
<p>Then we'll need to wire it up to the application:</p>
<script src="https://gist.github.com/theotherian/6237777.js?file=MyApplication.java"></script>
<p>Now, we need the connector itself.</p>
<h2>translating different request and response representations</h2>
<p>Most of the work done in the connector is ugly get/set logic; nothing sophisticated or glamorous. It still needs some explanation though.</p>
<p>At a high level, we need to implement the <code>apply</code> method, and inside get a <code>ContainerRequest</code> from a <code>ClientRequest</code>, pass it to the <code>ApplicationHandler</code>, and then convert a <code>ContainerResponse</code> to a <code>ClientResponse</code>. Here's the skeleton code:</p>
<script src="https://gist.github.com/theotherian/6237777.js?file=ServerSideConnectorApply.java"></script>
<p>Let's dig into building the request first:</p>
<script src="https://gist.github.com/theotherian/6237777.js?file=ServerSideConnectorBuildRequest.java"></script>
<p>We end up copying the main three pieces of data we need:</p>
<ul>
<li>The URI data (needed to construct)</li>
<li>The headers</li>
<li>The request body (or entity)</li>
</ul>
<p>I should call out that two of the arguments to the <code>ContainerRequest</code> constructor are null. The first is a reference to a <code>SecurityContext</code> instance, which is outside the scope of this post. The second is called <code>PropertiesDelegate</code>, which isn't actually Javadoc'd. The example works without it, though I may go back and dig into what it does later.</p>
<p>Now that we have the request, we need to send it to the <code>ApplicationHandler</code>:</p>
<script src="https://gist.github.com/theotherian/6237777.js?file=ServerSideConnectorSendRequest.java"></script>
<p>As you can see, we get a <code>ContainerResponse</code> instance back, which we'll need to convert to a <code>Response</code>, which is then used to create the <code>ClientResponse</code> instance we have to return:</p>
<script src="https://gist.github.com/theotherian/6237777.js?file=ServerSideConnectorBuildResponse.java"></script>
<p>Full disclosure: I don't know if I'm covering all the bases as far as what needs to be set on <code>Response</code>. I have for my example, but I don't know what I may be missing for other use cases at this point. I will update this post as I discover other data points that may need to be handled.</p>
<p>With this in place, we have <code>apply</code> fully implemented:</p>
<script src="https://gist.github.com/theotherian/6237777.js?file=ServerSideConnectorApplyFull.java"></script>
<p>There's a full implementation of the <code>ServerSideConnector</code> type <a href="https://gist.github.com/theotherian/6237777#file-serversideconnector-java">located here.</a> Another method has to be implemented for asynchronous functionality that closely resembles the <code>apply</code> method shown, along with a <code>close</code> and <code>getName</code> method that can be seen in the linked code.
<h2>use within a client and performance</h2>
<p>Here's an example client that handles resources that deal with <code>Message</code>s. As you can see, the invocation of the client is pretty cookie cutter, and the <code>Connector</code> implementation can be swapped out per the lines in the constructor:</p>
<script src="https://gist.github.com/theotherian/6237777.js?file=MessageClient.java"></script>
<p>At the beginning of this post I called out performance as a concern due to the latency of passing data over a network connection when it could be passed directly via memory instead. Let's see if the in-memory solution indeed performs better.</p>
<p>HTTP client requests:</p>
<script src="https://gist.github.com/theotherian/6237777.js?file=httpclient-requests"></script>
<p>In-memory requests:</p>
<script src="https://gist.github.com/theotherian/6237777.js?file=in-memory-requests"></script>
<p>Yep, I'd say it's faster.</p>
<h2>is this really necessary?</h2>
<p>Probably not, but I wanted to figure out how to do it. I do think there's value in hitting a consistent set of execution points for a single type of transaction, and that one way to keep it consistent is to have a single entry point, which this accomplishes (though with some overhead).</p>
<p>One major motivation for this was a continual challenge I've faced in regard to supporting both desktop and mobile versions of a website, and keeping those sites consistent.</p>
<p>Another use case I can think of for this is being able to break apart an application into more of a service oriented architecture as time permits. Up front it may not make sense to split an application up into too many services, but as your organization, web traffic, and resources grow, your need to split your application up to scale either to your customer base or your development staff will grow as well. Using a client right out of the gate, and being able to abstract interactions to the resources backing it from internal to external by changing two lines of code does, in my opinion, has value.</p>
<p>I'm interested to hear feedback about this solution, because I know it strays from the norm considerably. Like I said, the cymbals are firmly attached; are you planning on crossing the street?</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com7tag:blogger.com,1999:blog-4225124413648346528.post-2894960178229241312013-08-11T11:48:00.000-06:002014-03-31T21:55:18.986-06:00setting up jersey client 2.0 to use httpclient, timeouts, and max connections<h2>the goal</h2>
<p>I've been trying to get my head wrapped around Jersey 2.0 client after playing around with the server a fair amount, having some experience configuring the client for Jersey 1.x. There's typically a standard set of things I look to configure:</p>
<ul>
<li>Connection pooling via HttpClient</li>
<li>The read timeout</li>
<li>The connection timeout</li>
<li>Maximum number of connections</li>
<li>Maximum number of connections per host (or route)</li>
<li>JSON support</li>
</ul>
<p>I decided to figure out how to accomplish this in Jersey 2.0, which was less obvious than I had anticipated. It's important to note that <i>you absolutely should</i> set these values up in your application, because the defaults are a combination of being overly restrictive <i>and</i> overly generous. The defaults when using HttpClient are as follows:</p>
<ul>
<li>Infinite connection and read timeouts</li>
<li>By default when using Jersey's <code>ApacheConnector</code> you will get an instance of <code>BasicClientConnectionManager</code> which limits you to a single connection (though it is thread safe)</li>
<li>If you're configuring a <code>PoolingClientConnectionManager</code> instead, you'll have a maximum of 20 total connections with a maximum of 2 connections per host (or route)</li>
</ul>
<p>As I'm sure you've already realized, these defaults are not going to scale <i>at all.</i></p>
<h2>for every parameter a config, and for every config a parameter</h2>
<p>I've never been a fan of classes that contain a bunch of <code>String</code> based properties for an API because it can be difficult to figure out what goes where. There's no type to easily match against, so any method like <code>setProperty(String key, Object value)</code> could have anything set, and unless it's Javadoc'd that could be something from <code>FooProperties.*</code> or <code>BarProperties.*</code> for example.</p>
<p>(For the record, I like to define values like this with <code>enum</code> instances, sometimes with a common interface, to make it easier to find what should be used via an IDE)</p>
<p>I'm going to break down each part of the list above and how to go about configuring that feature by creating a class called <code>ClientFactory</code> one piece at a time.</p>
<h2>connection and read timeouts</h2>
<p>To begin, we need to start with a class called <code>ClientConfig</code>. Jersey uses this to configure client instances via its <code>ClientBuilder</code> API. We can set the connection and read timeouts with this class, as shown below:</p>
<script src="https://gist.github.com/theotherian/6205639.js?file=ConnectionAndReadTimeout.java"></script>
<h2>using httpclient with connection pooling</h2>
<p>Now let's set up a <code>ClientConnectionManager</code> that uses pooling. We should also set the limits on the number of connections a little bit higher, since 20 total and 2 per host is on the low side (we'll use 100 and 20 instead):</p>
<script src="https://gist.github.com/theotherian/6205639.js?file=HttpClientWithPooling.java"></script>
<p>You have to use the implementation <code>PoolingClientConnectionManager</code> to set the max values; the interface <code>ClientConnectionManager</code> doesn't provide these methods. You can also be much more fine grained about how many connections per host are allowed with the <code>setMaxPerRoute</code> method. For example, let's say it was ok to have 40 max connections when hitting localhost:</p>
<script src="https://gist.github.com/theotherian/6205639.js?file=MaxConnectionsPerRoute.java"></script>
<h2>configuring the jersey connector... with the config... which needs the connector</h2>
<p>This code is a little obtuse, but it's how you need to have <code>ClientConfig</code> know about HttpClient and have HttpClient know about the timeout settings. We need to create a new <code>ApacheConnector</code> from the configuration, which specifies the pooled connection manager, and then set the connector in the configuration. Hopefully this makes more sense looking at the code:</p>
<script src="https://gist.github.com/theotherian/6205639.js?file=ConnectorConfiguration.java"></script>
<h2>constructing the client and adding json support</h2>
<p>Now we have all the configuration we need to actually create the <code>Client</code> instance and set up JSON support:</p>
<script src="https://gist.github.com/theotherian/6205639.js?file=ClientWithJsonSupport.java"></script>
<h2>putting it all together</h2>
<p>Here's the full view of everything we set up to build the client. We're configuring the timeouts, setting up a pooled connection manager instance, restricting the maximum total number of connections and connections per host, adding an override for the maximum connections to localhost, and configuring JSON handling:</p>
<script src="https://gist.github.com/theotherian/6205639.js?file=ClientFactory.java"></script>
<p>As a point of reference, here are the relevant pieces for a Maven pom to get the correct dependencies for this example:</p>
<script src="https://gist.github.com/theotherian/6205639.js?file=pom.xml"></script>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com18tag:blogger.com,1999:blog-4225124413648346528.post-90336339939570936532013-08-10T23:33:00.000-06:002013-08-10T23:33:06.970-06:00using generic types in response entities with jersey 2.0 client<h2>a special type for a special case</h2>
<p>Java Generics are great, but they can truly be a beast to deal with once you start dealing with <code>Class</code> references that can't carry a generic type with them. In the case of using Jersey 2.0's client API, you may come into a situation where a resource returns a generified type, such as <code>List<Message></code>. You can't map the entity to <code>List<Message>.class</code>, but thankfully Jersey has a <i>very</i> easy way to handle this case:</p>
<script src="https://gist.github.com/theotherian/6203508.js?file=MessageClient.java"></script>
<p>That's it! Just binding it to <code>GenericType<List<Message>>(){}</code> takes care of it!<p>
<p>Source: <a href="http://jersey.576304.n2.nabble.com/How-can-I-parse-a-java-util-List-lt-gt-Is-it-supported-by-the-Jersey-client-td2300852.html">http://jersey.576304.n2.nabble.com/How-can-I-parse-a-java-util-List-lt-gt-Is-it-supported-by-the-Jersey-client-td2300852.html</a></p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com5tag:blogger.com,1999:blog-4225124413648346528.post-71177737052989698232013-08-10T21:21:00.000-06:002013-10-15T23:10:06.871-06:00understanding java's native heap (or c-heap) and java heap<h2>inspiration for this post</h2>
<p>Not that long ago, I was diagnosing an issue on Jenkins where I was seeing an <code>OutOfMemoryError</code> in a native API. "What hijinks be these?" I thought to myself, since while the memory footprint was high GC wasn't getting out of control. Of course, like so many things, I had to learn what the cause of these errors was within the context of a service outage.</p>
<h2>my first exposure to java's native heap</h2>
<p>When we think of the Java heap, we usually think of this chunk of memory that is kept in order for us by the Garbage Collector, and why wouldn't we? Any call to the <code>new</code> operator is allocating memory within the heap for whatever instance we're creating, and the Garbage Collector is keeping tabs on that instance so that, when it's no longer in use, the memory can be reclaimed <i>within the heap</i>. That last bit is important. I'm sure most people already know this, but it's still worth calling out that the heap doesn't shrink once it's grown, and will grow up to its max heap (-Xmx) size.</p>
<p>If you're using a 32-bit JVM, the max you can set your heap to is 4GB (or less depending on the OS), which is inclusive of the max heap and permgen size. Conversely, on a 64-bit JVM, you're limited by the machine as to what you set as the boundaries to your heap (depending on JVM implementation and <a href="https://wikis.oracle.com/display/HotSpotInternals/CompressedOops">CompressedOops</a>).</p>
<p>What you have left to work with, in both of these limitations, is the <i>free space</i> available to the native heap (or c-heap). I'm calling out that this is the free space available because the Java heap we've all grown to know and love is a section of the native heap; they're not mutually exclusive areas of memory. This space is used for native APIs and data, and it can most definitely run out.</p>
<p>Let's say you're using a 32-bit JVM, your OS can handle a 4GB heap, and you've allocated 3.5GB as the max heap and 384MB to permgen. Should you max those out, you've left your native heap with 128MB to do everything it needs to. In some applications this may not be a problem, but under certain conditions, say if you're heavily using IO, you could end up exhausting this memory, leaving you with an out of memory error in a native method. For example:</p>
<pre>
java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
...
</pre>
<p>There are a few more interesting details about the native heap that are worth pointing out:</p>
<ul>
<li>The native heap isn't managed by the garbage collector. The portion that makes up the Java heap is, of course.</li>
<li>Using <code>-XX:+HeapDumpOnOutOfMemoryError</code> won't actually work on <code>OutOfMemoryError</code>s caused by exhaustion of the native heap. There's a <a href="http://bugs.sun.com/view_bug.do?bug_id=6784422">bug ticket</a> logged for this which was, in my opinion incorrectly, closed as not reproducible.</li>
<li>A heap dump won't actually reveal what's happening in the native heap; you'll need process monitoring to figure that out.</li>
<li>Loading anything into or out of the native heap from the Java heap that isn't already a byte array requires serialization for insertion and deserialization for retrieval.</li>
</ul>
<h2>can you store stuff in the native heap from within your application?</h2>
<p>Honestly, I could write an entire blog post <i>just about off-heap storage</i> (in fact I started to write one here and stopped). I may very well write a post about that, but I'll leave you with the following advice: <b>Yes you can and probably shouldn't on your own</b>. There are a couple of ways to do this, one being <a href="http://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html"><code>ByteBuffer</code></a> (the "legit" way) and <a href="http://www.docjar.com/html/api/sun/misc/Unsafe.java.html">sun.misc.Unsafe</a> which you have to pry out of the JVM using reflection backdoors.</p>
<p>One detail that may not be obvious is that there are other settings for direct memory in the JVM. There's another flag that can be passed to the JVM called <code>-XX:MaxDirectMemorySize</code> which is different than heap size. Terracotta has an <a href="http://terracotta.org/documentation/4.0/bigmemorygo/configuration/storage-options">excellent write up about this</a>, which while it's for their product BigMemory touches on a lot of interesting data points that have to do with off-heap memory management.</p>
<p>I'd also like to point out that <code>ByteBuffer</code> delegates to a class called <code>Bits</code> which has some really sketchy implementation details when you allocate memory, so you shouldn't make calls to allocate any more often than necessary. Rather than type out the details, I'll just show you the code in all its glory (I put my name in the comments for the lines I wanted to draw your attention to):</p>
<script src="https://gist.github.com/theotherian/6203215.js?file=Bits.java"></script>
<p>If you were trying to store large blocks of data represented by byte arrays without blowing up your old generation heap space, using off-heap storage can be very beneficial. You can put all of the data in a <code>ByteBuffer</code> and read it from an <code>InputStream</code>, though that involves keeping track of offsets of data in the buffer and either writing your own <code>InputStream</code> to support the buffer or finding one that's already implemented in another project.</p>
<p>If you were trying to use off-heap storage as a cache for Java objects, you should probably look at something preexisting like <a href="http://www.hazelcast.com/docs/2.5/manual/multi_html/ch03.html">Hazelcast</a> or <a href="http://terracotta.org/products/bigmemory">Terracotta's BigMemory</a>. The two challenges you'll end up with trying to handle this yourself are serialization/deserialization since all objects must be converted to/from byte arrays, and managing how you're accessing that data directly from memory. The serialization and deserialization aspects can be painful from a performance standpoint, especially using Java's built in serialization. You can get significantly better performance using something like <a href="https://code.google.com/p/kryo/">Kryo</a> or <a href="https://github.com/FasterXML/jackson-dataformat-smile">Jackson Smile</a> which serializes to binary JSON. There's also <a href="https://code.google.com/p/fast-serialization/">fast-serialization</a>, which claims to be faster than Kryo and has some <a href="https://code.google.com/p/fast-serialization/wiki/QuickStartHeapOff">off-heap storage implemented</a> with more in the works. Hazelcast <a href="http://blog.hazelcast.com/blog/2013/06/13/comparing-hazelcast-3-serialization-methods-with-kryo-serialization/">recently did a comparison of Kryo and Smile</a>, and the results clearly show a noticeable improvement in performance. Accessing the data is also non-trivial, since you need to allocate large chunks of data and manage offsets yourself to fetch the correct data.</p>
<p>If you were trying to use off heap for dealing with IO, you should check out <a href="http://netty.io/">Netty</a>, which not only works very well and intuitively, but also <a href="http://netty.io/4.0/api/io/netty/buffer/package-summary.html#package_description">does the job better than <code>ByteBuffer</code></a></p>
<p>There's a really nice blog post at <a href="http://mishadoff.github.io/blog/java-magic-part-4-sun-dot-misc-dot-unsafe/">http://mishadoff.github.io/blog/java-magic-part-4-sun-dot-misc-dot-unsafe/</a> that goes through the many things you can and shouldn't do with <code>Unsafe</code> if you're interested. There's also a fantastic writeup about using <code>ByteBuffer</code> and dealing with all of its idiosyncrasies at <a href="http://mindprod.com/jgloss/bytebuffer.html">http://mindprod.com/jgloss/bytebuffer.html</a></p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com9tag:blogger.com,1999:blog-4225124413648346528.post-43780599118930675302013-08-07T22:08:00.001-06:002013-08-07T22:08:30.786-06:00having trouble with jersey 2.0 and servlet 3.0? you need jersey-core-servlet!<h2>a subtle problem</h2>
<p>I just came across this with some example code I'm working on, and the problem is easy to miss.</p>
<p>Let's say you want to use the JAXRS <code>@ApplicationPath</code> annotation for your Jersey application, and you don't want to use a <code>web.xml</code> file anymore, i.e. you want to programmatically define your servlet using Servlet 3.0 mechanisms. You have everything set up just like you've seen in all of the examples online, you run <code>mvn jetty:run</code>, and... Nothing.</p>
<h2>two dependencies</h2>
<p>There are two dependencies that serve similar purposes; adapting Jersey 2.0's <code>Application</code> class to a servlet instance. One of them includes compatibility for Servlet 2.x, and the other doesn't:</p>
<script src="https://gist.github.com/theotherian/6181277.js?file=jersey-container-servlet-core.xml"></script>
<script src="https://gist.github.com/theotherian/6181277.js?file=jersey-container-servlet.xml"></script>
<p>You'd think that this just means one has the ability to support Servlet 2.x and the other doesn't. In my experience, the case is that automatic Servlet 3.0 hooks won't actually work with <code>jersey-container-servlet-core</code> <i>at all</i>; it only works using <code>jersey-container-servlet</code>. Amusingly, the comment in the pom that is generated by the Jersey archetype (not the Grizzly one) is, at least in my mind, equally misleading:</p>
<script src="https://gist.github.com/theotherian/6181277.js?file=archetype-comment.xml"></script>
<p>In my mind, it makes more sense for this to say:</p>
<script src="https://gist.github.com/theotherian/6181277.js?file=ideal-comment.xml"></script>
<h2>proof</h2>
<p>I set up a very basic example of this behavior that you can feel free to mess around with if you like. The following two classes make up what is just about the most basic example for a Jersey app (though I use <code>ResourceConfig</code> instead of <code>Application</code> just because I like its flexibility better):</p>
<script src="https://gist.github.com/theotherian/6181277.js?file=JerseyApp.java"></script>
<script src="https://gist.github.com/theotherian/6181277.js?file=HelloResource.java"></script>
<p>To make this work, here's the pom we're going to use. In comments in the dependencies section, you can see which line you need to comment to experience the problem:</p>
<script src="https://gist.github.com/theotherian/6181277.js?file=pom.xml"></script>
<h2>simple, but not so obvious, solution</h2>
<p>Hopefully this helps you. This actually had me scratching my head (which means grinding my teeth as I gradually type harder) for the better part of a day before I realized I'd been bamboozled by similar dependencies.</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com10tag:blogger.com,1999:blog-4225124413648346528.post-49428845016714508162013-07-28T18:42:00.001-06:002013-07-28T18:48:03.210-06:00hibernate, joins, and max results: a match made in hell<h2>what are you ranting about now?</h2>
<p>Hibernate's ability to handle joins when specifying a max number of results via <code>Criteria</code> can cause some serious pain. The real problem here is that <i>nothing in the mappings will look wrong.</i> In fact, in several cases the mappings will return the correct data, but with some serious consequences. This is a case where you should probably learn how the hot dogs are being made.</p>
<h2>give me something to go off of</h2>
<p>For this example, we're going to be using the following two classes, which represent a one to many relationship:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=Person.java"></script>
<script src="https://gist.github.com/theotherian/5993277.js?file=Thing.java"></script>
<p>The tables those entities represent contain the following data, and will exist in an HSQLDB instance:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=person-data"></script>
<script src="https://gist.github.com/theotherian/5993277.js?file=thing-data"></script>
<p>In most cases, we're going to try and get data out of these tables with the following <code>Criteria</code>:
<script src="https://gist.github.com/theotherian/5993277.js?file=CriteriaBasic.java"></script>
<p>Note that an <code>Order</code> is being added here to prevent any lack of determinism in result sets from obfuscating what's happening.</p>
<h2>alright, so what happens?</h2>
<p>Nothing good. Let's start off mapping the <code>@OneToMany</code> as such:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManyEagerMappedByJoin.java"></script>
<p>We're using eager loading with a JOIN fetch type, and we're defining that the relationship is mapped by the <code>person</code> field in <code>Thing</code>.</p>
<p>Now, let's run our <code>Criteria</code> and take a look at the SQL generated:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManyEagerMappedByJoin.sql"></script>
<p>Looks like what would be expected. What does <code>persons</code> contain?</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManyEagerMappedByJoin.output"></script>
<p>Hold on just a second. Why are there <b>two</b> instances of 'Ian' being returned?</p>
<p>Well, the <code>limit</code> applies to the <b>entire result set</b>, not just the parent entities. Since the join is matching the two children, those are the first two rows returned. As a result, not only are we just getting one person back, but we're getting two of the same instance in the list.</p>
<p>Typically, if you're using a <code>JOIN</code> fetch with a <code>Criteria</code>, there's a way to eliminate duplicate root entities:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=CriteriaBasicWithDistinct.java"></script>
<p>Unfortunately, it doesn't work here. Since <code>maxResults</code> applies to the underlying result set and not the entities Hibernate is pulling in, adding this restriction actually just limited the list to be a single instance:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=CriteriaBasicWithDistinct.output"></script>
<h2>that was using join. what about subselect?</h2>
<p>I've seen many instances where people switch their <code>OneToMany</code> associations to use <code>SUBSELECT</code> rather than <code>JOIN</code> because they see duplicate parent entities and don't know about the <code>DISTINCT_ROOT_ENTITY</code> transformer. That said, let's take a look at what <code>SUBSELECT</code> does. Here's the mapping:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManyEagerMappedBySubselect.java"></script>
<p>Let's run it through the original <code>Criteria</code> setup again:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=CriteriaBasic.java"></script>
<p>Alright, how's that output looking?</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManyEagerMappedBySubselect.output"></script>
<p>Great! What queries did it run to get the data this time?</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManyEagerMappedBySubselect.sql"></script>
<p><b>WHAT?!</b> Now we have <b>four</b> queries? DO NOT WANT!</p>
<p>Breaking down the SQL a bit, we can see that the first query only selects records from <code>PERSON</code>, while the second selects only records from <code>THING</code>. But...</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManyEagerMappedBySubselectIn.sql"></script>
<p>It didn't just select the <code>THING</code> records for the <code>PERSON</code> records we wanted, <b>it selected every record from <code>THING</code></b>. The series of ids that the <code>IN</code> clause is selecting against is totally unbounded; the limit statement isn't applied.</p>
<p>Still, that doesn't explain the third and fourth queries, which are indeed the same. The reason these execute is that the <code>Thing</code> instances loaded reference <code>Person</code> instances which don't exist in the session yet. Since they're mapped by the <code>person</code> field in <code>Thing</code>, Hibernate will select each missing <code>Person</code> with a separate query.</p>
<p>Imagine if we had 1000 records in <code>PERSON</code>, each having two corresponding records in <code>THING</code>. <b>We would end up executing 1000 queries to get the data!</b> Generally the enemy with Hibernate is <code>n+1</code> selects; in this case you have <code>m-n+2</code> selects, where <code>n</code> is the number of results and <code>m</code> is the total number of records in the parent table.</p>
<h2>what if you use a join column instead of mapped by?</h2>
<p>Very well, let's change the mapping again:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManyEagerJoinColSubselect.java"></script>
<p><u>Side note: the <code>name</code> used in <code>@JoinColumn</code> appears to be case sensitive, at least in HSQLDB and Oracle. Using <code>"person_id"</code> instead of <code>"PERSON_ID"</code> will yield the same additional selects in the last example.</u></p>
<p>Now what queries ran?</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManyEagerJoinColSubselect.sql"></script>
<p>If you think we're out of the woods here, I assure you we're not. Using <code>@JoinColumn</code> changed the relationship such that we're not issuing individual statements for parents, however, we're still fetching in everything. Hibernate still runs the same subquery for <code>IN</code> as before, except that we join to the parent table up front which avoids the additional individual selects:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManyEagerMappedBySubselectIn.sql"></script>
<p>It's important to realize that <b>the entire contents of both tables are still being pulled into memory here.</b> Again, if you're dealing with a huge number of records, you could easily run out of heap space.</p>
<h2>what about using hql instead?</h2>
<p>You can try and run this query with HQL. However, the result is not much different than before. Here's the code to do so:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=HqlSelect.java"></script>
<p>Running it will generate the following SQL query:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=HqlSelect.sql"></script>
<p>The results returned are actually correct; the first two people and their associated things are in the list. Notice again that we have an unbounded query; the limit statement was never applied. Just before the SQL is logged, Hibernate logs a warning:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=HqlSelect.warning"></script>
<p>This means that Hibernate is fetching everything and then <b>trying to apply the first/max result restrictions in memory.</b> As you may imagine, this is also undesirable. This happens because of the <code>join fetch</code> used in the HQL query. Even more comforting is what the JPA spec has to say about this kind of interaction:</p>
<blockquote>
The effect of applying setMaxResults or setFirstResult to a query involving fetch joins over collections is undefined. (JPA "Enterprise JavaBeans 3.0, Final Release", Kapitel 3.6.1 Query Interface)
</blockquote>
<h2>well, if subselect, join, and hql don't work, that just leaves select</h2>
<p>Correct, however, <code>FetchMode.SELECT</code> by itself will cause an <code>n+1</code> select problem.</p>
<p>Luckily there's a way to mitigate that problem.</p>
<p>Hibernate has another annotation called <code>@BatchSize</code>. Unfortunately, the JavaDoc for this annotation is simply "<code>The batch size for SQL loading</code>" and doesn't really explain what it's doing. What the annotation really does is generate a select for the association against multiple ids (up to the number specified) and automatically uses the ids of entities that are in the session that haven't had their association loaded yet. In other words, if you were to get 5 instances of <code>Person</code>, and you had your batch size set to 5, Hibernate will issue a single select for instances of <code>Thing</code> that matched the 5 ids of the <code>Person</code> instances in the session when the any association needed to be loaded. If you had 7 <code>Person</code> instances in the session with a batch size of 5, Hibernate would issue two selects: one for the first 5 and another for the other 2.</p>
<p>Here's the code for this in action (you can use <code>FetchType.LAZY</code> or <code>FetchType.EAGER</code>, they both work):</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManySelectBatch.java"></script>
<p>Here's the SQL generated:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManySelectBatch.sql"></script>
<p>And here are the results, and importantly, the ones we wanted without anything extra being loaded:</p>
<script src="https://gist.github.com/theotherian/5993277.js?file=PersonOneToManySelectBatch.output"></script>
<p>The example above does use lazy loading just to call out that it will work in this case since the other examples were eager. If you were to try lazy loading in any other capacity, you'd end up running into other issues:</p>
<ul>
<li>Using <code>FetchMode.SELECT</code> will cause <code>n+1</code> selects without <code>@BatchSize</code> (and would still do so if you had a batch size of 1)</li>
<li>Using <code>FetchMode.SUBSELECT</code> will cause the same unbounded query we saw when using <code>SUBSELECT</code> eagerly, and will generate <code>m-n+2</code> selects if it uses <code>@OneToMany</code>'s <code>mappedBy</code> attribute, where <code>m</code> is the total number of records in the parent table</li>
</ul>
<h2>ok, so the problem can be solved. why the rant?</h2>
<p>The main reason I tried to outline so much in this post is that <i>the other examples don't look wrong.</i> Common sense would lead most to believe that the mappings were correct and that the limit would be applied as expected to the parent and children. You don't even see the issue unless you have <code>show_sql</code> turned on, your DBA yells at you for DDoSing the database with tiny individual selects, or your application runs out of memory due to loading huge result sets in memory. In all of those cases, you won't really notice the issue until your data gets a little larger, and by then you could be staring down a production outage depending on the scope of the problem.</p>
<p>I'd never fault someone for getting this wrong, because on paper the annotations seem so logical. There have been several bugs logged for this issue (<a href="https://hibernate.atlassian.net/browse/HB-520">HB-520</a>, <a href="https://hibernate.atlassian.net/browse/HHH-304">HHH-304</a> and <a href="https://hibernate.atlassian.net/browse/HHH-2666">HHH-2666</a>), and even Gavin King himself says in one of the tickets:
<blockquote>Any limit that gets applied to a query should also get applied to the subselect of any subsequent collection fetches.</blockquote>
</p>
<p>Using <code>@BatchSize</code> is syntactically trivial, but it does require some thought. You don't want to stick too high or too low of a number to the batch size. Too low and you'll generate many selects. Too high and you could end up loading more data than you needed, though this is probably more applicable to lazy loading than eager since eager is going to pull in the association for every parent in the session anyway. Having too high of a batch size could adversely affect database performance since the range of the total number of arguments provided to the query is going to be greater unless you have some unusual degree of uniformity in your data; i.e. a batch size of 100 will generate as many as 100 unique statements. As long as you have some notion of the number of entities you're fetching in certain areas where the number of results is bound by a max or a where clause, you should be able to pick a sensible number that will keep the total number of queries you run low.</p>
<p>Alternatively, you could create a separate entity to handle different association loading strategies. I've used this before to enforce a type safe contract that can control what mechanism you're using to load associations, as well as in a polymorphic capacity. If you had an area of your application where you knew you'd be using a max results limitation, you could have your persister logic look up entities using <code>FetchMode.SELECT</code> with <code>@BatchSize</code>, and use another <code>FetchMode</code> and/or <code>FetchType</code> on a different entity when you didn't need batching. Leveraging Hibernate's <code>@MappedSuperclass</code> annotation means you can leverage polymorphism for this use case nicely; declaring the getter for the association in the superclass and mapping the association at the field level in the subclass.</p>
<p>As far as my knowledge of Hibernate is concerned, using batching (or batching with separate entities) is the optimal case for dealing with joins combined with max result limitations. If anyone reading this knows of a better way, or can see a flaw in my logic or code above, please let me know in the comments!</p>
<p>Also, all the files used in this project can be found at <a href="https://github.com/theotherian/hibernate-gotchas">https://github.com/theotherian/hibernate-gotchas</a>. Feel free to check out the project and mess around with this yourself!</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com17tag:blogger.com,1999:blog-4225124413648346528.post-71518554344464768592013-07-27T17:38:00.000-06:002013-09-02T11:37:22.794-06:00creating an in-memory hsqldb instance for using hibernate in a maven build<h2>a simple setup for testing</h2>
<p>Lately I've been working on some Hibernate examples I plan on sharing soon, and I wanted to create a simple in memory instance of HSQLDB to test against. Trying to drill down to the bare necessities of what I needed proved to be a little scattered across several resources, so I thought I'd aggregate everything in one place.</p>
<p>This article is assuming a few things:</p>
<ul>
<li>You're using a Maven build</li>
<li>You're using Hibernate</li>
<li>You don't care about persisting to disk or database state</li>
</ul>
<h2>creating the server</h2>
<p>Setting up the server is mostly straightforward, but I do have a subtle change to make the Maven aspect easier:</p>
<script src="https://gist.github.com/theotherian/6096578.js?file=ServerApplication.java"></script>
<p>Note that the location is set to <code>target/mem:test</code>. This tells HSQLDB to create an in-memory only database, which will never persist changes to disk. That said, HSQLDB still writes out a few files (a log, script, and properties file) which uses the name of the database. Prefixing the name with <code>target/</code> will write to Maven's default build directory so that your workspace doesn't have a bunch of log data in it that you'll have to ignore from version control</p>
<h2>configuring hibernate</h2>
<p>Now that we can fire up the database, we can connect to it via Hibernate. The config file below will connect, but also has a subtlety similar to what was used above:</p>
<script src="https://gist.github.com/theotherian/6096578.js?file=hibernate.cfg.xml"></script>
<p>Here's where things get a little screwy, and there's a decent chance it's because I don't understand certain aspects of HSQLDB. In this case, the connection string is <code>target/hsql:mem:test</code>. As far as I can tell, <b>both the HSQLDB server and driver write to the same set of files, prefixed by the connection string or database name.</b> When you start up the database and connection pool, you end up with files in <code>target</code> with the names of <code>mem:test.*</code> and <code>hsql:mem:test.*</code>. Since both of these sets of files end up being written to, I figure it's not a bad thing that they're named differently. I do find it a little odd that the client logs data like this, but at least at the moment I don't care enough to see if this can be adjusted. If it can be I'll update this post to reflect how to do that.</p>
<h2>wrapping up...</h2>
<p>As stated above, I really wanted a sandbox to demonstrate a few Hibernate interactions with, so I really wanted to keep the setup as barebones as possible. There are probably certain things I could or should have configured differently, but the above will work to get you going. I never really use HSQLDB, so if you're reading this article and have some input on a better way to do this or any corrections that should be made, please leave your feedback in the comments :)</p>theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com0tag:blogger.com,1999:blog-4225124413648346528.post-18369546169762972412013-07-12T22:35:00.001-06:002013-07-12T22:35:33.880-06:00how to name your github gists<h2>yes, this is a bit of a hack</h2>
<p>I started using Gists to encapsulate all of my code, command line, and log related content rather than using <pre> tags or some styling tool to format code inline. I found that using Gists cleaned up the source of my blog considerably, and made it easier to edit content in general; converting to an HTML escaped version and then having to make changes afterwards was a pain at times</p>
<p>That said, I ran into a different pain, which was Gist's bizarre way of handling organization of data sets</p>
<h2>description? check. name? well...</h2>
<p>Gists to allow you to set a description for a set of files, which can help you identify what it is that's in there from a list of all your gists. However, it also organizes them from the alphabetically first file name in the set. If you had, for example, a bunch of Maven examples all with a file in the set called 'pom.xml', you'd have a hard time navigating your list of Gists. It also shows the contents of that file in the list, further confusing you.</p>
<p>However, that identification system can be used to your advantage if you're clever. If you create another file in your Gist which is actually a name for the set, but precede that name with a single space, it will be sorted to the top alphabetically. You can't have empty files though, so my advice would be to actually add your description there instead. As an example of how this looks, take a look <a href="https://gist.github.com/theotherian">at my Gists and how they're labeled.</a>
<h2>how can that file be excluded when using gists remotely?</h2>
<p>Gists offer a script tag based solution for embedding a Gist elsewhere. For example:</p>
<pre>
<script src="https://gist.github.com/theotherian/5954592.js"></script>
</pre>
<p>would embed all the files contained in the Gist at <a href="https://gist.github.com/theotherian/5954592.js">https://gist.github.com/theotherian/5954592.js</a></p>
<p>But what if you just want to embed an individual file, in this case the <code>togglewifi.applescript</code> file? In this case you can append <code>?file=filename</code> to the URL:</p>
<pre>
<script src="https://gist.github.com/theotherian/5954592.js?file=togglewifi.applescript"></script>
</pre>
<p>The end result is this:</p>
<script src="https://gist.github.com/theotherian/5954592.js?file=togglewifi.applescript"></script>
theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com0tag:blogger.com,1999:blog-4225124413648346528.post-80154641240724174382013-07-12T22:15:00.000-06:002013-07-12T22:15:25.552-06:00ditching blogger's dynamic views<h2>not all they're cracked up to be</h2>
<p>After starting this blog with the new Dynamic Views blogger offers, I've decided to go back to the classic templates.</p>
<p>Oddly enough, the sobering reality that they weren't for me happened when I tried to direct a colleague towards my blog for an answer to a question. Trying to explain where to click and how to navigate was surprisingly non-intuitive, whereas the classic template is sort of the model most blogs follow. They're straightforward, easy to navigate, and lightweight.</p>
<p>Another reason for reverting was Github Gists. While you can shoehorn them into blogs using Dynamic Views using JavaScript others have written to consume specially crafted code tags, I'd rather just use the direct links Github provides.</p>
<p>The other views that Dynamic Views offer didn't really do much for me or the blog either. Most of the views are largely inapplicable to a technical blog, and just add a layer of complication that's unnecessary. While I could see value in using them for other non-technical blogs, I think they're a poor fit for blogs heavy on text rather than images.</p>
theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com0tag:blogger.com,1999:blog-4225124413648346528.post-84470149621161536792013-07-02T23:29:00.001-06:002013-07-15T09:20:29.064-06:00creating resource filters with jersey and jaxrs 2.0<h2>in what situations would I use this feature?</h2>
<p>Let's say you have a series of preconditions for certain resources in a web application. For the sake of this example, let's assume we're using Jersey as the controller for a web application rather than just a purely RESTful web service. How can we solve this requirement?</p>
<p>For some simple cases it's not unheard of to see an inheritance model leveraged for this, where resources extend other resources that provide methods for certain preconditions. If single inheritance started to make this unwieldy, a composition or functional approach could be used instead, but you may end up inundating your resource with additional logic. In certain cases, it may be preferable to define this precondition with metadata, perhaps with an annotation.</p>
<p>In cases like this, you can leverage Jersey's dynamic binding of resource filters along with your own annotations to both define conditions as metadata and decouple resources from other business logic in your code base. Below is source code to demonstrate this.</p>
<h2>the annotation</h2>
<script src="https://gist.github.com/theotherian/5882658.js?file=IE6NotSupported.java"></script>
<p>Pretty straight forward: no fields, just retained at runtime and able to annotate methods.</p>
<h2>the filter</h2>
<script src="https://gist.github.com/theotherian/5882658.js?file=IE6NotSupportedFilter.java"></script>
<p>This is a little more interesting. Here we check the user agent to see if it's IE6, and if it is, we abort the request directly in the filter. Aborting the request is a new feature in jaxrs 2.0. In this case, we're sending a 412 response code (<code>PRECONDITION_FAILED</code>), and passing back an error page as the entity via the response.</p>
<p>As you might have guessed, since we're showing a <code>ContainerRequestFilter</code> here, there's also a <code>ContainerResponseFilter</code> that can also be used with resources. The response counterpart is passed both the <code>ContainerRequestContext</code> and a <code>ContainerResponseContext</code>. A typical example of a use for response filters is attaching supplemental headers to a response.</p>
<p>Now that we have our filter, we need a way of binding it to any resources annotated with <code>@IE6NotSupported</code></p>
<h2>the binding</h2>
<script src="https://gist.github.com/theotherian/5882658.js?file=ResourceFilterBindingFeature.java"></script>
<p>Now we're getting somewhere. Here we have an implementation of a <code>DynamicFeature</code>, another class that's new to jaxrs 2.0. For those unfamiliar with this class, here's a quick snippet of the javadoc:</p>
<pre>
/**
* A JAX-RS meta-provider for dynamic registration of <i>post-matching</i> providers
* during a JAX-RS application setup at deployment time.
*
* Dynamic feature is used by JAX-RS runtime to register providers that shall be applied
* to a particular resource class and method and overrides any annotation-based binding
* definitions defined on any registered resource filter or interceptor instance.
</pre>
<p>This is invoked for every method of every resource in your application <i>when the application starts up</i>. In the example above, we're specifically looking if the method is annotated with <code>@IE6NotSupported</code>. We could also change our annotation to support targets of <code>TYPE</code> as well as <code>METHOD</code>, and call <code>resourceInfo.getResourceClass()</code> and perform the same check.
<h2>the bootstrap</h2>
<p>Now that we have an annotation, a filter, and a way to link the filter to a resource, we need to tell our application to invoke this upon startup.</p>
<code id="gist-5882658" data-file="MyApplication.java"></code>
<script src="https://gist.github.com/theotherian/5882658.js?file=MyApplication.java"></script>
<p>Typically you'd have many classes mapped here for all the resources and providers in your application, but for the sake of the example we're just mapping the <code>ResourceFilterBindingFeature</code> class.
<h2>summary</h2>
<p>I hope this helps introduce you to the world of Jersey and jaxrs 2.0 filters. Personally, I find them hugely beneficial for mapping out both pre and post conditions that should be honored for resources without coupling them to the resource itself. I'm a big fan of annotations, and for things like resources I think there are several cases where a precondition can be expressed via metadata rather than in the resource itself.</p>
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
<script type="text/javascript" src="https://raw.github.com/theotherian/gist-embed/master/gist-embed.js"></script>
theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com0tag:blogger.com,1999:blog-4225124413648346528.post-10937427539685195342013-04-25T21:01:00.002-06:002013-04-25T21:01:44.038-06:00This just in: dope with blog forgets to renew domain, more at 11If any of you tried to hit my blog in the last week, I apologize for the lame domain parking page that came up. I thought I ordered 5 years of service for my domain, but apparently I only ordered one and it expired.<br />
<br />
I've now renewed it and will pay for the domain for 5 years in advance for real this time.<br />
<br />
Sorry...theotherianhttp://www.blogger.com/profile/05381212808252496737noreply@blogger.com0