BlogicBlog: View from the trenches

The blog has moved

2007-01-01T14:43:00.000-05:00

I have blogged here about Java and tech support issues since September 2004. Initially, the blog was anonymous as people were worried about negative repercussions from their workplace. Eventually, Sun and IBM and BEA (who I worked for at the time) saw the benefits and I have gone public.

After a year or so of blogging, I realised that I want to blog about more than just Java. Unfortunately Blogspot did not support the categories and I did not want to pollute JavaBlogs syndicator (where I am a member) with irrelevant content.

So I switched to the hosted WordPress account for my other topics some months back. And boy! was it nicer. Google is trying to catch up now, but it is too late for me.

The free WordPress account had one major disadvantage though: it had limited customization of the layout and plugins, something the original Blogspot provided for.

So, as part of the New Year resolutions, I have decided to consolidated the blogs into the new one at the personal domain. I have moved entries from both blog there, but not without some lost comments. Hopefully, I will figure out how to get them across eventually.

I will still be a member of the JavaBlogs, since the new installation (also a Wordpress) allows to syndicate on the category level, rather than the full blog level.

Happy New Year and I will see you at the new blog.

BlogicBlogger Over and Out

Ouch: Thinking about hiring an offshore team?

2006-12-25T22:05:00.000-05:00

Check who they are training with. Yakov Fain's notes from an offshoring training presentation are scary, and not in a good way.

BlogicBlogger Over and Out

Microsoft's way of fixing a security issue with a feature

2006-11-21T23:07:00.000-05:00

... is dropping the feature.

I have an iPAQ rx3115 with a built-in WiFi. I had it for a while and it is a nice unit, if one forgives Windows Mobile 2003 (this is possible on a good day). I also have a wireless router, so can download books and generally browse internet from anywhere in the flat without turning the main computer on.

One of the other benefits of the built-in WiFi was synchronization over IP, which for me was specifically over the WiFi. This feature is no more.

Microsoft decided that this feature is a security risk because people's email, files and notes were transmitted unecrypted and could be snooped upon. So, with their new focus on security they had to fix this problem and fix it urgently.

So they dropped the feature in the update.

I am not sure what more to say. They decided and they executed. They did not bother mentioning this in the release notes, they did not provide it as either a disabled-by-default option or even as a registry setting. Nope. Just gone! And to make sure I don't endanger myself, they removed all old versions of the ActiveSynch as well.

One brave engineer decided to explain the reasoning behind the feature drop in his personal blog. Whatever! I can't say it any better than multiple comments on his blog already did and a fat lot of good it did to anybody.

One thing for sure - this does not bode well for Vista, where such changes don't even require people installing the updates.

BlogicBlogger Over and Out

Java has gone open source with GPL2

2006-11-13T10:08:00.000-05:00

A great news all around.

I hope Sun will also host a public cross-referencing source code navigator for the code, rather than just a subversion directory. Something like Sorcerer might be a good start. We know the idea is good, because Linux had one for ages.

Oh, and a new line in a resume to strive for: Java source code contributor.

BlogicBlogger Over and Out

Example based code quiz from O'Reilly Labs

2006-11-01T21:59:00.000-05:00

O'Reilly labs have created a quick, but addictive code quiz that tests programmer's knowledge based on him being able to look at code example and figure out which book it might have come from.

I run through it in one breath and did not do too badly (280 points). I mostly had troubles with VB/ASP (no surprise), but also with some examples that were ambiguous.

O'Reilly is going to do some serious data mining from the game results, but I wish they would also provide the dataset to the public, obfuscated or otherwise. It would be interesting to play with it, a la Netflix Prize. And/or they could allow the results to be embeddable in the page for the brag value.

BlogicBlogger Over and Out

Obscure bulk format converters of OpenOffice.org

2006-10-22T15:06:00.000-04:00

I needed to convert multiple MSWord documents into the OpenOffice.org format and then later to convert those documents further into HTML. How to do it with a single document is obvious; it took some searching to figure out how to do it for multiple documents.

The magic menu turned out to be File/Wizards (Document Converter and Web Page correspondingly). In a retrospect, it is not unobvious. The only reason it took time is because I very rarely use wizard interfaces of any software.

Be that as it may, the wizards themselves are quite powerful and useful and work just fine on large file sets. The Web Page wizard's actual purpose is to generate a pretty table of content for the documents, but it will convert all the documents into HTML as a side-effect. So, I just deleted the generated TOC files and kept the converted HTML documents.

BlogicBlogger Over and Out

Online typing speed test

2006-07-04T15:25:00.000-04:00

Nice, simple typing speed test page. Even with tired hands, I managed to pull good 86 wpm with only a 3 mistakes.

I remember buying the TypeQuick software long time ago to learn touch typing, when it still came on floppies and had some really strange copy protection. I remember the pain it was to uninstall the application back onto the floppy disk to make sure the license could be transferred over to the new computer.

At least, I could transfer the license back then. This days, Windows will not even let you do that legally. Talk about monopoly abuse.

BlogicBlogger Over and Out

JavaOne 2006 sessions are up with sound and transcripts

2006-07-02T13:08:00.000-04:00

The sessions are finally up and like last couple of years have sound synchronized with the slides and transcripts. Like last couple of years, there are some transcription errors. Not surprising given the accents, etc. Still, I wish Sun had given the speakers a way to double check and correct the transcriptions. Given that they will be up for a long time, it is worth making sure things look really well.

I have checked my own session (TS-1669) and it is not too bad. I had more raspy voice than I have normally, but that's probably because of all the talking and partying that happened beforehand.

BlogicBlogger Over and Out

JavaOne session evaluation of my talk

2006-06-29T07:01:00.000-04:00

I have received the session evaluations from the talk I did at JavaONE(TS-1669) (slides are available). Differently from the evaluation of the talk two years ago, this time it is more scientific. I guess pushing the evaluation paper on every attendee has paid of.

I am quite happy with the result. The top two quartiles for overall quality were 4.03-4.29 and 4.29-4.89. My mark is 4.24. I did not make it into top 25% of the sessions, but I am somewhere in the top 30%.

I had a fairly large room (620 people capacity). It was fully booked, but I was told the number of people who really show up is between 35% and 70%. My count was ~~296~~ (Update: 432), which is still respectable, especially considering that this year there were several sessions on troubleshooting.

My other stats are fairly good as well. They are as follows (track, global averages are in brackets):

Overall: 4.24 (4.01, 4.12)
Matching of the presentation and the abstract: 4.29 (4.11, 4.17)
Speaker quality: 4.31 (4.01, 4.14)
Presentation material quality: 4.13 (3.94, 4.01)
Demos: 3.76 (3.82, 3.89)
Q&A: 3.77 (3.88, 3.91)

The last two are totally understandable as well. I did not actually have a live demo. That was for two reasons. I do not have a laptop at the moment and my tools setup is too complex for Sun's presentation machines. But also, and more importantly, I could have had a tiny demo or cover an additional theme. I chose to cover an additional theme (XML file processing). To my mind, that was a worthwhile tradeoff. I did built some demo-like slides into the presentation, but there is obviously nothing quite like seeing the speeker work the keyboard live.

With Q&A marks, people were generous. I totally run out of time for Q&A. Last time, they had to drag me off the platform, I was running so late. This year, I finished with 30 seconds to spare, so could evacuate before the dreaded bell. I stayed outside the room and chatted with anybody who was interested (I had about 10 people), but I really wish I had time for Q&A. Maybe next time, if I don't overpromise with the session topic again.

I also wish I had some better feedback on how to improve my talks. Sun may provide the copies of the hand-filled comments sections eventually, but if you were at the presentation and have a comment, please make one on the blog. Constructive negative comments are very welcome.

BlogicBlogger Over and Out

Nested Archive Toolkit

2006-05-21T21:14:00.000-04:00

IBM alphaWorks has released an interesting tool called Nested Archive Tookit that allows to look inside the nested zip/jar/ear/war files. It looks to be useful for modification of config files deep inside the structure (e.g. web.xml inside foo.war inside bar.ear). It can also provide XML file with the description of the archive to the full depth. This makes it useful for postprocessing with XSLT and/or visualization tools.

There is a big problem with the tool however and it is its license. As with most of other alphaWorks tools, it uses 90 days evaluation license. This means that production use is not really allowed and certainly not for long term. Given that the tool is not for everyday use, it is questionable whether good evaluation can be made in the time permitted.

I have spoken with alphaWorks representative about the license issues at JavaOne and he said that it is possible to request for the tool to be open sourced. So, check it out and if it matches your needs, ask for it to be open sourced via license form.

Oh and your IBM id is most probably an email (this one catches me every time).

Finally, if the license issue irks you as much as it does me, there is always TrueZIP with its Apache license. TrueZIP has some advanced functionality of its own, so it is worth a good look as well.

BlogicBlogger Over and Out

JavaOne day 0 - the issues Sun will do nothing about

2006-05-15T23:15:00.000-04:00

A lot of interesting questions were asked at alumni-only fireside chat with Sun team today. I am only going to mention the questions Sun did not have a good story for.

JVM on PDAs - Sun is pushing Java for Mobiles quite hard. Witness the support for SaveJe. This however is a difficult road especially in North America with vendor fragmentation and difficulties of implementing full multimedia/call-handling capabilities. A question was asked whether Sun would consider implementing JVM just a basic app environment for non-phone PDAs running Windows 2003/2005 and for phone PDAs but without full integration of call-capabilities (like IBM's j9, but better). The answer made it very clear that PDAs are not a target device by themselves and that phone integration is a goal, but only with full-functionality . I have a non-phone PDA and was hoping to have a good JVM for it, but it looks like I should just forget about it.
Multimedia (JMF) API - This one is well and trully dead. Sun is happy for somebody else to develop an alternative reference implementation, but they are not putting any money into it themselves. The basic reason is that they think it will take too much effort to get JMF to really useful state and they are not interested in making it less than perfect. Apparently, there is now a project to do an open-source alternative implementation; let's hope it works out.
Serial port/USB access - Serial port implementation is semi-dead and there is no USB implementation. Sun's response was that there is a JSR by IBM about USB access, but no further details or enthusiasm were forthcoming.
Java hosting - the cost of java hosting is 10-15 times higher than hosting PHP apps. I have asked whether they knew why that might be and whether this is something Sun should pay attention to. The reply I got was that this is how capitalist market works and that maybe I should email Jonathan Schwartz about it. I might have mangled the question as the meaning I was aiming for was to see whether it was known what made Java hosting an expensive business - whether it was memory requirements, complicated versioning/classpath issues or something else. If I figure out a good way to phrase it, I will send an email to Jonathan and see what happens.
JavaOne scheduler application suckage (my question again) - every year scheduler application gets better, but it never seems to be as good as current technologies make it possible. Turns out that Sun people don't use the scheduler themselves (duh!), so they don't actually know it sucks (or pretend that they don't). Well, they got crowd's feedback on that issue loud and clear (very loud). Hopefully (as suggested) the next year's scheduler will be a competition of some sort with the best version becoming an official one. James Gosling sort of volunteered for that one.
Allowing JavaDocs to be published and remixed on 3rd party websites. This was a reference to several websites that a while ago put JavaDocs for official JDK packages together with JavaDocs for various open source software and allowed remixing/commenting/other social manipulations across all those APIs. Sun leaned heavily to have their API removed and I wanted to know whether that negative stance changed and/or whether they thought the legal/copyright issues really were more important than potential benefits of API presentation remix. Graham Hamilton answered that legal issues were quite important, but that JavaDoc will be much better in Java 6 and beyond. An answer, but not quite to the original question. He did add that maybe that issue should be revisited, but - as with other 'to be revisited' issues - I don't really expect them to remember the vague promises with all the other issues and pressures occuring during JavaOne.

There were many other good questions and answers and I am sure somebody will blog about them. These are just the issues that I think are difficult to pin Sun people down on usually and it is worth writing them down for future references, while the memory is strong.

BlogicBlogger Over and Out

Link: Nice port and network services search engine

2006-05-05T08:32:00.000-04:00

Ports-Services is a useful service to identify what might be listening on a particular port or what port should particular (well-known) service listen to.

It has some java related ports (RMI, Jini).

It also has trojan listings for the ports and it is a bit scary to see the useful service entries completely overshadowed by the huge number of various malware.

BlogicBlogger Over and Out

My talk made the top 10 suggestions for JavaOne 2006 Tools

2006-05-01T20:12:00.000-04:00

Dana Nourie lists 10 presentations in tools space that she thinks are worth seeing at JavaOne 2006. My own presentation is there as well (number 7). Guess it struck the chord.

I also have 240+ people booked for my session as of now. No way to tell if that is many or not enough for this stage. But if any of those 240+ are reading this blog, are there any questions on what you would like to know about? While the slides are locked, the Q&A and closing comments are still flexible.

BlogicBlogger Over and Out

Ethereal (network protocol analyzer) 0.99 is out

2006-04-30T01:16:00.000-04:00

Ethereal is a tool I always use when I have any kind of network related problems. Be it a firewall dropping packets, a server that incorrectly sets cookies or even an overly clever application that tries to obscure the exact network resources it uses. Ethereal collects that network data all the way up and down the network stack and all the way into file formats, such as GIFs and XML. And it allows to filter on any of the fields it understand.

Usually Ethereal is updated every couple of months or so, but this time it was nearly 4 months. But the wait was worth it. They fixed a number of problems, made Windows experience just that tiny bit nicer and started to integrate a real scripting engine. They chose Lua.

This last one is a biggie. Ethereal has a lot of capabilities, but people always ask for just one more. Mine was to be able to extract all files of a particular mime-type out of the captured HTTP stream without having to click the payload packets one-by-one.

I have not looked in details yet, but from the example given it looks like it can plug into the menus and have a full interface presence as well as low-level procedure hooks.

Even with Lua being only an experimental addition, I really like the direction Ethereal team is taking. I only wish they actually explained the changed them made to the protocols. It is always nice to see more and more features in the HTTP dissector, but it would be nice not to have to actively hunt for them.

BlogicBlogger Over and Out

The Art of _Technical_ Customer Service

2006-04-14T11:45:00.000-04:00

Guy Kawasaki writes about the art of customer service. While all of the points are applicable, he did not take the one about integrating customer service into the mainstream far enough.

He talks that customer service people receiving accolades as much as sales, engineering and marketing. That's great. But what about actively helping them to do their jobs better. Sales have fancy tools to track customers' pipelines, marketing get party budgets, engineers get multiple-monitors and fancy tech toys. Support is lucky if they get headsets that do not suck.

So, what can be done to increase the productivity for technical support people?

Are all communications with customers logged with specific cases? Usually this is true. But does that case tracking system allows to have files/attachment linked to it? Does it have web view so that both customer and the support engineer have a common history to discuss if the issue gets hairy? Does the case tracking system records every email sent out to the customer or is it an extra burden to record what's happening?
Are there tools/training that would make common tasks easier? For example, what text editor is being used to look at all those log files? Notepad? Shareware textpad? Or has someone actually introduced a feature rich editor (I like Vim) and trained people in how to use it.
Is there a backchannel from support to engineering and/or customer service advocate? This one is extremely important, as there are many things engineers do not have time to think of when the product is rolled out or updated. Small things like visible versioning of release and patches, like locations and types of configuration and log files, like whether the log files produced by the application miss correlation information such as thread id. Similarly, is there a checklist from support to engineering, QA and doc groups on what kind of things the product needs to have/explain so the new functionality is immediately supportable.
Is there a communication forum (mailing list, IM group, wiki) for the support people, especially if they span multiple locations? Given enough pain, support groups will collect the knowledge themselves, but it will be delayed, fragmented and not universal.

These things taken together are a pie in the sky, but each one individually is actually within an easy reach with a strong return on investment. I am presenting at JavaONE this year on a similar topic, so if anybody is interested to chat with me about it, I would be happy to meet.

BlogicBlogger Over and Out

Thread: More data, more tools or more answers?

2006-04-06T11:38:00.000-04:00

One of my conversations was commented on by Dr Anton Chuvakin. I had replied to him in his comments, but unfortunately he keeps reposting the article to new services without bothering to address the reply, so I am putting it here as a central reference.

Basically, Anton (if I may call him this) thinks that the universe of choices around log file discussion is not just
1. More data
2. Better formatted data
3. More tools to mangle the data

But must also: include
4. More answers to their issues-of-the-moment

He favours the last point much stronger than others and says:

Thus, I think the focus should be on more intelligence and giving answers, not tools.

I happen to think that while his 4th point is great, it is not a concrete, actionable step forward and therefore does not really help the discussion.
Specifically (from my original comment):

...The whole point of tech support is that the answer is not easy to find. Certainly on the 2nd/3rd level of support it is so.

Therefore, what you instead get is a lot of data that _probably_ can lead one to the answer. Unfortunately, if the data is not coherent (parsable, timestamped, thread-stampted, etc), it is next to impossible to find the answer even if one is known to exist.

One of the real examples I had was trying to prove that a particular access did _not_ happen based on 2 Gigabytes of badly formatted, mismatched log files. I have managed to do that but only because I wrote some tools for the task. And even with the tools, it was still up to me to interpret the results correctly and explain them to the customer.

In summary, your fourth point is a goal, while first 3 are possible ways of getting there. Just asking for that 'give answers' solution is like looking for silver bullet.

I would love Anton to comment on this specifically. Maybe he can give a concrete support to his point, as I did before for the point 2).

BlogicBlogger Over and Out

I am now on JavaOne's EventConnect tool

2006-04-02T14:08:00.000-04:00

I am presenting at JavaOne again this year, My session is TS-1669 entitled " Unhappily ever after: support, maintenance and troubleshooting of Java applications in production environments "

It covers more topics than the one I did in 2004. It also seems to have been marked as more interesting by Sun. Last time I remember having about 300 people in the room. This time, the earmarked room is for 700 people. I will believe it when I see it. I think they might change it once they had a chance to compare all the presentations.

In any case, I will be happy to push the message out to however many people will show up. But if I catch the interest of any techie managers having to provide technical support for Java applications in the field, I will be very happy. I think there is not enough thought put into making technical aspects of support easy. There are some low-hanging fruits out there for the taking with the results better for both support and developer side of the problem solving proces.

If anybody is coming to JavaOne, I am now on the Event Connect tool and will welcome the introduction/meet requests.

BlogicBlogger Over and Out

Why I prefer Andrew C. Oliver to write 'drab' articles

2006-03-22T20:55:00.000-05:00

Andrew is confused why his 'drab' article on speeding up JBoss is much more popular than his repost of a deep voodoo article asking other people to implement someting that he finds interesting. I wonder!

I have linked to the speeding up article when I saw it, because it was something useful either immediately or later if I do have strange slow-downs/memory issues to troubleshoot. It (as I mentioned originally) was also useful beyond JBoss and I knew that because I had to explain the same issues to the customers using Weblogic.

His other article seems to requite much more thought and a deeper involvement. Maybe one day I will have free time and skills to understand that article as I am sure it deserves to be. But probably not soon.

P.s. I am not vain enough to think that my opinion really deserves a full blog entry (and I saw the article from JavaBlogs' pupular list anyway); it is just that he doesn't have the comments enabled.....

BlogicBlogger Over and Out

Oh Tomcat of the multiple conflicting ports!

2006-03-22T19:55:00.000-05:00

Ever tried running multiple Tomcat on the same machine and have that fail because of the port conflict. And not because of the HTTP listen port conflict - because you did know about that one and changed it. But, rather, because of the other ports that are open out of the box as well that even Tomcat's documentation does not mention.

So, turns out that Tomcat 5, has 3 ports open and what interesting ports they are:

8080 - that's the one they tell you about and it is where all the normal HTTP traffic goes to
8009 - that's an AJP connector that you need if you are behind a webserver like apache that will pass the requests to you. Why is it on by default, I don't know. You have to configure the webserver side anyway, how difficult would it be to uncomment it at the same time. And if - for whatever reason - you are running Tomcat on its own, you now have another obscure port to worry about as a management hassle or even a possible attack vector.
8005 - This one is interesting. It is binded to the localhost only and it is how you shutdown the tomcat when you run the shutdown script. And to shut it down, all you need to do is telnet to the port and say the magic word, which for tomcat 5.0 is hardcoded at SHUTDOWN and for 5.5 is helpfully kept in the open in the server.xml . You don't even need to be the same account to do this, just a user on the same system. This small issue has been acknowledged by the Tomcat's team.

So, to make this also a fishing lesson rather than a handout of seafood, here is a generic way to check those ports without having to page down the 19K of semi-commented-out XML.

The command should be all in one line:


...\xmlstarlet-1.0.1\xml  sel -T -t
 -m //*[.//@port]
   -m ancestor::* -o -+ -b
   -v local-name()
   -i @port -o : -v @port -b
   -n
 server.xml

The command line above means: for each element that has an attribute port or a child with such an attribute, print the element with offset based on it nesting depth; if this particular element does have the port attribute, print the port value as well.

The result for the default Tomcat's setup is:


Server:8005
-+Service
-+-+Connector:8080
-+-+Connector:8009

BlogicBlogger Over and Out

Review: Log4j Chainsaw v2

2006-03-20T20:53:00.000-05:00

Recently, I had been contacted by one of the Chainsaw's developers - Scott Deboy (no blog yet). He asked if I had any feedback on the tool's new iteration as it would seem to be a good match to my interests and blog's goal.

I had looked at the Chainsaw a while ago, but it was not quite up to scratch then. Looking at it again, the tool feels much better and if you have to look at the log files, I strongly recommend to check it out (and/or Splunk).

Initially, Chainsaw may feel a little forbidding due to a large feature set. But as with other strong tools, such as Ethereal and XMLStarlet, if you have to deal with the problem space repeatedly, these tools make it worth your while. And all of those features are actually useful and come from attacking real problems. This is quite a different feel to some of the vendors' tools which look shiny, but fail in real use.

I do however have some gripes that I feel would make troubleshooting much faster if resolved. Most of them are around (otherwise great) highlighting and expression facilities.

For the fields that take an expression, it would be good to see whether the expression typed in so far is correct. Ethereal does this quite nicely. Expression builder is good for beginners, but immediate correctness feedback would be useful for those that are more advanced.
It would be good to have the highlight rules available as a view/pane. In Vim (with OTF script installed) - I can highlight with as little as :1OTF XYZ (1-9 is a color index). So, the price of verifying an idea is very quick. I feel both Ethereal and Chainsaw are slightly behind in that respect. In addition, I can highlight multiple elements in a line with different colours. That really helps sometimes. Again, nothing else I have seen allows me to do that.
The expression window for Next/Previous navigation does not seem to have keyboard shortcuts equivalent to the buttons and requires switching hands from the keyboard to the mouse. Having keyboard mapping would be faster. Some other window also did not seem to respond to Enter key and required switching to the mouse, but I did not note which one.
Bookmarking a particular line and ability to come back to it quickly would be great. This feature however is more nice than important.

So in summary, Chainsaw is very nice, but still with some room to grow.

BlogicBlogger Over and Out

What's in the access log

2006-02-26T20:12:00.000-05:00

What is a server access log and how much information can one actually extract from it? If you don't know, read this good introductory article on the subject.

And if you find the article interesting, look for other logs your server/application produces. The easiest way to find out what log files you have is to run ProcessExplorer on Windows or lsof on *nix and see what .log files the program is keeping open and where. Doing this may also help you to discover where all the disk space is gone.....

BlogicBlogger Over and Out

More GraphViz goodness

2006-02-25T00:54:00.000-05:00

I wrote about GraphViz before, but so many new Java projects showed up, that I thought it was worth making an update.

Grand: Ant config file visualizer that even better than other options I wrote about before.
LightUML: UML generator from Java classes
Linguine Maps: Visualizer library for many types including WSDL, Ant, Hibernate and DTD
JarAnalyzer: Dependency visualization for jar files (with an independent review)

If you have anything where A and B point to C and C points to D, GraphViz is really worth checking out.
It is not in Java however; if you only look at Java products, check out Prefuse instead.

BlogicBlogger Over and Out

The good and bad of outsourcing to Russia

2006-02-17T15:26:00.000-05:00

Yakov Fain writes (from experience) about good and bad aspects of Russian programmers for outsourcing.

I left Russia well before American companies thought of trusting russian (or indian) programmers. But I remember the software and ingenious hacks written back then by my friends to work with limited computing resources available. I participated in programming competitions and summer camps, where school kids learned and applied concepts that are only taught on advanced level of university education in other countries.

To this day, I claim to have gone through two different schools of programming. One is Russian and one is Australian. And you shouldn't have to guess (after this article), which one I feel had given me a stronger grounding in what I do daily.

BlogicBlogger Over and Out

Good-bye ProcessExplorer - your license got too strict

2006-02-15T22:27:00.000-05:00

I used to rave to everybody about how good ProcessExplorer from Sysinternals was for technical troubleshooting. Oops, I guess I was too loud.

The new license terms state:

A commercial license is required to use the software in any way not covered above, including for example:

Use of the software for technical support on customer computers

This license does appear in ProcessExplorer 10.05. The license is not yet in my other favourite tool (FileMon), but I assume it will get rolled out with the next release.

This may still allow users to download and install the product themselves, but even that seems to violate the spirit if not the letter of the license.

I was going to present at JavaOne with very strong emphasis on the tools from Sysinternals, but now I may have to rewrite that presentation.

I can understand that the company wants to drive the customers towards their commercial offerings, but it is still sad to suddenly be able to do less rather than more with the new release.

BlogicBlogger Over and Out

Re: Guerrilla Debugging For Java

2006-02-14T22:07:00.000-05:00

Russ Olsen (via Michael Baum) writes about tools to use in production, when ones does not have access to tools. Specifically, when one does not have access to tools like Eclipse IDE.

He mentions a good list of tools, though he has a bit too much praise about Strace. To quote:

Ever wonder why your program can‘t open that one file? Use strace to find out precisely which file it is trying to open and the exact identity of the error. Does your program simply go into a trance and stop responding? There is a fair chance that you will find out why with strace. Clueless as to why your webapp is leaking memory? Strace may supply the clue.

In reality, there are much better tools to use for those task. At least on windows, FileMon is a much better tool to check which files are being opened and Handle is great for checking which process locks what files (lsof does that for Unix). Thread dumps analysis (my tool) is better for checking what's hanging. And Ethereal is a must for any network troubleshooting.

But the general point is strong. And the section about BeanShell was something I haven't thought of before. Very good summary article.

BlogicBlogger Over and Out

Quoted, but misunderstood: What's Missing from Production System Troubleshooting

2006-02-07T21:49:00.000-05:00

Michael Baum quotes my feedback on his survey article, but completely misses that we actually want the same thing. We just approach it from different angles.

To get to the (misunderstood) point:

The notion that IT people need even more data generated by developers kinda misses the point. Troubleshooting production applications is a whole lot different that debugging code in development or staging environments. Production systems involve many technologies and systems that just don't appear in pre-production environments.

I do not ask for more data, I ask that the data format currently being used is reviewed to see whether it is actually useful for troubleshooting/monitoring and that concerned effort is made to change the format where it proves not to be useful.

For example, any messages produced by multithreaded services must include thread/transaction id in them. Trying to extract sequence out of the logs that just intermingle their log entries is next to impossible. Same problem with having timestamps in a format that does not allow to correlate to other log types.

My point was that developer would not see this kind of issues, until they have to do the troubleshooting themselves. Then, they might be more amenable to pleas for better logging formats.

And yes, I did spend 3 years as technical support engineer for BEA looking at the multi-megabyte (sometimes multi-gigabyte) log files for people whose configuration I did not know 30 minutes earlier. So, I believe I did have to deal with the issues Michael have seen at Yahoo and at Splunk too. In fact, I will be delivering JavaOne presentation about this very issue this year.

Speaking of Splunk, it is a great idea and a step in a right direction (as I wrote was 9 months ago). I could see how it would have been useful to me when dealing with large data sets.

Unfortunately, it is only a first step; to replace my advanced troubleshooting environment (*cough* Vim *cough*), I would need to at least be able highlight several patterns at the same time in different colors (e.g. IP address, time sequence and URL type).

But I will be evaluating Splunk in more details and probably will be mentioning it in my presentation at JavaOne. Especially, if it will be downloadable as VMWare image to try under Windows environment.

(Update from Feb 14th: we are on the same page now)

BlogicBlogger Over and Out

Gmail and the periodic spam filtering failure

2006-02-07T01:21:00.000-05:00

Gmail is usually pretty good about spam filtering. But not at the moment. As of the last 2 or 3 days, the spam I know it caught before is now ending up in my mailbox. And I think I know why.

The same thing happened over Christmas. Suddenly all the spam appeared in my inbox. I had a theory then and it seems to be confirmed now.

I think Gmail team drops their filtering level around big celebrations (Valentine's day is coming up). Around these events, people send to each other emails that differ from their normal day to day activities (warning: humour). It is very possible that normal Gmail's algorithms would treat celebratory emails as spam and people would end up loosing messages.

So, to avoid classifying good emails as spam, Gmail team drop the filtering level. That in turn makes some spam emails look normal which annoys people in a different (more habitual and therefore less important) way.

I hope this is not the case and that I am just observing some fixable glitches, but this is the second time now and therefore is a coincidence (up from the chance). If it happens again around may (mother's day), I would think I have a full blown theory.

Oh and any Gmail engineer is more than welcome to comment. :-)

BlogicBlogger Over and Out

The techie way of liberating the podcast URL from iTunes

2006-02-03T12:48:00.000-05:00

Jon Udell is not happy about having to transcribe podcast URLs that iTunes displays, but does not allow to copy. While the general point about lock-in is good, here is a quick techie workaround in a meanwhile.

However much iTunes may want to hide the URLs, at some point it has to actually retrieve something from it, and do it using standard network protocol.

Enter Ethereal, open-source multi-platform network protocol analyser. Using it to get the URL is a serious overkill, but the tool itself comes useful over and over again in many situations.

The steps to snag the URL are:

With iTunes already open, start Ethereal up and run capture in non-promiscuous mode with 'tcp' as the filter.
Go to iTunes and do 'Update podcast' on the podcast you want.
Let the update happen and stop the Ethereal capture. You should now have a lot of packets in the Ethereal's view.
In the filter section (this is different filter from before) enter: http.request.uri contains "xml" .
This should show only the packets that are http requests for files with xml in their name. If nothing shows up, try http.request.method as filter. If there is still problem, try http and see if you have any traffic on the chosen interface at all.
Assuming you finally have the right request, you now do right-click-menu/Follow TCP Stream and copy the URI and host from there.
You are done. It sounds a bit long, but takes about 30 seconds after installation is done and correct interfaces are figured out.

For more advanced HTTP work, it is also worth going to the preferences for Ethereal and make sure that packet reassembly is enabled for all levels of the network stack (HTTP, TCP, IP).

BlogicBlogger Over and Out

Re: How Hard is it to Troubleshoot IT Anyway?

2006-02-02T09:44:00.000-05:00

Michael Baum reports on the survey of system administrators regarding their troubleshooting activities. It is an interesting summary, but something is missing.

There seem to be a lot of questions regarding how the problems are handled now with the predictable answers of base power tools like grep, perl and Ethereal. What I don't see is any questions on how to fix the problem going forward.

By now we pretty much established that until the developers themselves try to support/troubleshoot their own products in production (or get loud enough feedback), they will not understand how to make their products easier to manage post-deployment.

The surveys of the how do you deal with it now kind should always include questions on why commercial solutions are not suitable (usually due to installation/license difficulties) and also what the companies creating the products could do to make things easier in a long run.

I know some companies slowly do it on their own (e.g. dTrace from Sun), but I think, if backed by organisations such as LOPSA or NaSPA, the progress might have been faster. After all, by now we have pretty much established that the problems will not go away by themselves, but - if anything - will get worse.

And if the System Administrators want to join forces with other technical people running into the same problems, they should pay more attention to technical support people as well as to forensic analysts. Both of these groups also have to deal with finding a needle of important information in a mountain of obscure, disjointed, overwhelming data. The progress in the area of forensic analysis tools is especially fast these days, as it is driven by the very high profile security concerns (most of the February issue of Communications of the ACM is about this very topic).

BlogicBlogger Over and Out

I am presenting at JavaOne 2006

2006-01-31T21:12:00.000-05:00

This feels good. Last time I submitted and got accepted to talk at JavaOne (2004), I was working for one of the major sponsors. With all the rumors about too many presenters from Sun and sponsors, it makes one wonder how much of the reason was the presentation quality and how much was some of the other factors.

These days, I am definitely not working for a sponsor, so hopefully my presentation is due to my topic alone. And the topic is: Unhappily ever after: support, maintenance and troubleshooting of Java applications in production environments.

The preliminary outline I have submitted is extremely ambitious, but I hope I can fit it all in, even if not in details. I have talked about many of the issues in the blog already, so that could become useful for back-references.

I will probably be putting some snippets on the blog as I go along, but if any of you out there have strong opinions on what you would like to see in such a presentation, the comments are open.

BlogicBlogger Over and Out
Technorati tags: javaone

Re: I love when its in Word - you have it easy

2006-01-24T21:43:00.000-05:00

Andrew C. Oliver muses on the need for violence when getting Java stack traces embedded in the MS Word docs. I call his bluff.

A day, when your MS Word document arrives and you open it to discover an image of the DOS window where part of the stack trace is seen at 50% zoom, then you can contemplate violence.

Until then....

BlogicBlogger Over and Out

Support aspects of AJAX

2005-12-10T23:41:00.000-05:00

AJAX has grown up! People starting to wonder what impact AJAX style applications have as compared to the older logic. Some impacts are obviously positive, some not so.

From the support point of view, AJAX style application may cause more hidden problems due to the granularity of the communications. Specifically, where a single user previously might have made one new server request every 30 seconds after initial load, with AJAX, the same user might be generating a new request every couple of seconds.

This means that even a server for small userbase will have to be configured and tuned as if the userbase is much higher. I am talking higher workers counts, more potential for bottlenecks, higher log volume.

And if the userbase was already high or is growing rapidly, the servers may not even be able to deal with it well and may even require a different implementation style.

Greg Wilkins - who is the author of Jetty HTTP server - has started thinking of it already. His article about these issues is well worth a read even if you are using a different server.

BlogicBlogger Over and Out

Link: new troubleshooting blog

2005-12-10T23:37:00.000-05:00

Another troubleshooter decided to speak of his experience. This time it is Michael Baum with an IT Troubleshooter blog over at Infoworld.

The amount of content there is still small, but the ideas sound good to me. If you like what I write, check his blog out as well.

BlogicBlogger Over and Out

My take on "problem solving skills"

2005-09-19T19:02:00.000-04:00

cynicalman asks what the "problem solving skills" in a job description actually mean and then tries to discover and blog the answers.

I do not actually agree that his first discovered rule is actually "a skill" (I think it is more of a test procedure), but the question asked does require a thought.

I was frequently told that I have good problem solving skills. So, here is my quick take on what those skills are:

Being able to quickly deduce a structure and meaning from the given examples or behaviour: For example, my co-worker was having a problem with a Fortran program. I haven't seen Fortran before, but was able to figure out the control structure rules on the fly and see where the bug was. Another example would be to take a program with an unfamiliar menu structure (or a familiar Microsoft one) and quickly find where a particular option could be.
Knowing how to ask question - of yourself and others - that would partition the problem space: So, you say the server has crashed? is not a good question. How about When you say the server crashed, did the program exit completely? If not, was it rather just very slow and/or returned to normal a while later? It is really a couple of questions here, but the idea is to split the possible problem space into segments and decide which segment to keep working with. Similarly a good question would be: If this was caused by a network problem, how/where/when would it show up? Which log or trace file do I need to check to confirm or reject this?
Knowing how to convert data into information: Getting a 3 page long java thread dump is a (bad?) luck. Knowing how to read that thread dump to identify what the program was doing at the time and what it was _not_ doing is a problem solving skill.
Being able to manipulate large datasets and extract required individual entries: To me, this covers not being scared by 2GBytes access logs when I search for a problem behaviour pattern. It also covers knowing the tools to slide/dice/reconstruct the data sets.
Having a toolkit and knowing it well: At the moment, my toolkit is Vim with couple of plugins for editing and RegExp work, Ethereal for network, FileMon/RegMon/TcpMon from SysInternals for Windows troubleshooting, XMLStarlet for XML files, and, of course, Cygwin for its full toolchain. I have also written custom programs, mostly for complex data extraction. Finally, I use GraphViz for visualization of my extraction results. One of the things that frustrates me most is a person reading a log file in Notepad. What can they do with it except looking at it one line at a time? No highlighting, no place marks, no way to interactively clean it up by regexp deleting some unrelated sections.
And, above all, not giving up: Most of the situations can be troubleshooted successfully. Especially if you can reproduce it, insert trace statements or do other developer level modifications. If you think you hit a writer's block of troubleshooting, use the same techniques writers do (try anything, switch attention, define next action, etc.)

Comments, thoughts, references?

BlogicBlogger Over and Out

Weblogic 9: support aspects

2005-09-03T10:38:00.000-04:00

Vinny Carpenter reviews Weblogic 9. He mentions many features, but forgets what I think will be a very interesting supportability aspect: Diagnostic Service

Up to the 9.0 release, one of the big support headaches was that for many of the subsystems, the running state was not exportable for analysis; that a transaction running through the system would have different IDs in different subsystems with no way to associate them.

Basically, it was often impossible to tell what the system is actually doing without writing custom patches, which the customer may or may not be willing to install in the production system.

The Diagnostic Service was designed to solve exactly that. I don't think it provides for it all out of the box, but the hooks certainly are there and I am hoping to see a community form around this service within or outside of BEA.

Technorati Tags: WebLogic

BlogicBlogger Over and Out

Easier way to collect thread dumps

2005-08-16T21:54:00.000-04:00

When I worked at BEA support, we troubleshooted a lot of issues using the thread dumps. There were a couple of problems with them:

On Solaris/Linux/HP-UP, the thread dumps had to be extracted from the stdout files, which in some cases did not exist (never redirected) and in others were not rotated on restart and therefore had several months of logs (GBs). Either way finding the thread dumps was difficult.
Windows users often just had stdout going to the console and, when the thread dumps were screen-scraped, they got broken up in all sorts of strange locations making it harder to do any automated processing or even a decent pattern search.
When more than one thread dump was required (90% of the cases), it was hard to figure out how often the thread dumps were taken as they did not include a timestamp. At least SUN JVM did not. JRockit and IBM, both had it.

Therefore, I am glad to see that it is becoming easier to get good thread dumps. At least for JRockit with the jrcmd command. In fact, the command looks useful for a number of things.

Unfortunately, it is still somewhat hard to do the same with SUN's JVM. They have the jstack command, but it is still not available for Windows.

Of course, with dTrace, thread dumps might become obsolete, but yet again this is only a Solaris 10 option, though there are some rumours of dTrace for BSD or maybe even Debian.

In a short term though, if you find yourself in need of some serious thread dump taking, do take a look at JRockit. It is free after all. And it does have other cool troubleshooting tools.

BlogicBlogger Over and Out

Book review: 'Blog' by Hugh Hewitt

2005-08-14T20:34:00.000-04:00

I blog. Obviously! But I felt that an influential blogger like Hugh Hewitt might give a good overview and write a book worth reading. Boy, was I mistaken.

This book is squarely targeted at non-technical, right-wing, religious Americans who don't mind being sneered at. Anybody else should try to avoid it.

For that Anybody else group, here is couple of reasons from the book itself (emphasis mine).

An audience test that really should have been on the book's jacket and not on the page 88 of the book:
Most of America knows that elite journalism is staffed by people who are overwhelmingly way-left-of-center in their politics. If you don't believe that America believes that, or you want to argue over what 'way-left-of-center' means, you have purchased the wrong book. You are still living in the land of the lost and don't want to move, so go buy Al Franken's or Michael Moore's latest and miss the revolution.

Hugh seems to think that contempt is a good thing to demonstrate to prove your point:
Barbra Streisand is a blogger. A not-very-bright one, but a blogger nonetheless. (p 128)

I am also not so sure Hugh was the right person to advise on how to setup a blog. He freely admits to not knowing the technology, but the following quote shows that maybe the editor should have insisted harder on a technophilic co-writer.
I am still unsure what an RSS feed is and have trouble making the permalinks work - but it does not matter!
RSS is so important in having a blog read and indexed that sacrificing it for whatever easy to start a blog point Hugh had was in my mind a terrible choice. He has an RSS feed on his own website - four types actually - so he has denied others what might be contributing to his own success.

Nearly all of the examples are given based on political or religious blogs. Are the technical blogs not worth a mention? Or is the technical community is just not a target for the book because they have already figured all of this out? Either way, mentioning a blogging CEO like Jonathan Schwartz would have carried Hugh's point much further than the argument in the book did. In fact the only real mention of the technological company is Technorati, which Hugh is plugging quite often.

Even the practical advice does not seem to ring a bell for me. For example:
So how to go about making friends in the blogosphere? Here are some specific steps.

First, acquaint yourself with the form, and having done so, decide who genuinely does a good job, who meets your information needs. Then tell them. Via e-mail. Repeatedly.

The word SPAM comes to mind. I am not sure why. Maybe it is the Repeatedly bit that seems to imply rejection at first attempt.

The section on what to do when one is convinced to start blogging is interesting, but unfortunately, it is barely 20 pages long. Couple of good articles would have done it better.

Finally, there seems to be no mention of the long tail market which allows for so many people to blog about interesting things that are not politics or religion.

Those were the reasons that really made me to write this review. People with stronger political views and sharper tongues actually pick on different points.

Still if I haven't convinced you, maybe the book is for you. It is currently available at Amazon at 60% discount ($8 down from $20). This is quite telling all by itself. After the Amazon takes its own 20% cut - which is another $4 - there is not much margin there for the author to make any money of the book. To me that means that the real purpose of the book was to further increase Hugh's fame rather than to really educate the audience. Caveat emptor!

BlogicBlogger Over and Out

Re: More Readable Classpaths for Ant Builds

2005-08-01T21:59:00.000-04:00

Lance Hankins got annoyed with having to eyeball ant's output of the long classpaths and wrote a task to do the classpath output in a slightly more readable form.

Neat, but perhaps there is a more generic way to do it. After all, not only ant produces these long semi-column separated monstrosities.

So, I want to show how I do it with Vim (Why Vim...).

Let's use Lance's output as given.

Copy and paste it into Vim.
Notice that it is broken into multiple (23) lines. While he probably did it for a better web presentation, this also happens when somebody screen scrapes the classpath from the DOS window.
Let's combine all the lines back into one long one: :%j!.
This command means:
- in the command-line mode (:),
- for the whole buffer (%)
- join lines (j)
- without inserting or deleting any spaces on the join (!)
Now, let's split them on the semicolon: :s/;/^M/g.
In here, we are
- in the command mode again (:),
- working with the current line (no range indicated)
- doing a substitution (s/FROM/TO/)
- from semicolon (/;/)
- to a line break (^M, produced -at least on windows- with CTRL-Q CTRL-M key sequence),
- and repeating this substitution for the whole line (s/FROM/TO/g)
Ok, we are done in less than 20 keystrokes including cut and paste and carriage returns!
Let's see what else we can do. How about sorting and deduping the entries? (after cleaning up the first line to remove the [echo....] bit)
:%!sort -u. This will require Cygwin to be first on the path in the Windows environments.
How about highlighting all the sprint jars in one color and all hibernate jars in another?
:1OTF spring and :2OTF hibernate. This requires OTF Vim plugin.
As an added benefit, the command above immediately highlight spring's jar for hibernate as it is the only line with two colors on it. Could be useful to keep in mind.

I could go on for a while, but hopefully this will be a more useful and generic way to deal with long classpaths rather than having to write an ant task. KISS, DRY, etc.

BlogicBlogger Over and Out

Two takes on culture clash of Mainframe vs. Modern skill sets

2005-07-25T19:36:00.000-04:00

C|Net News has an article talking about Legacy (mostly mainframe) skills and how they contrast to the skills developed in the modern world of Linux, Web, Java and .Net.

It is a typical high level article where each word is important, but the whole passage is hard to read.

On the other hand, we have a Slashdot discussion on the same topic from only a couple of days earlier. In there, deep comments are mixed in with just as deep geek humor with the end result of being much more entertaining end educational at the same time.

But only by reading both articles, one gains a good understanding of what the issue is actually about.

BlogicBlogger Over and Out

Re: Open Source Applications for IT You've Never Heard Of

2005-07-24T15:27:00.000-04:00

Alan Williamson is planning to present about Open Source application that most of people don't know about.

He is also asking the community to contribute their lists. Well, I have three tools that I like enough to mention (again).

Ethereal: Whenever you have to troubleshoot an application with a lot of network traffic, Ethereal is a good tool to keep in mind. It allows to see the network traffic on both high and low level and is flexible enough to narrow down exactly to the information subset needed. And it keeps getting better.
XMLStarlet: If you are trying to extract some information from an XML file, XMLStarlet is a great way to quickly try several approaches without having to type a lot of XSLT. I have written about it before , but there were some bigger introduction articles written about it later here and here
GraphViz: This is a visualization tool that allows you to build complex graphs (directed or otherwise) from a very simple description language. I talked about it in depth before as well.

All of these tools have just enough of a learning curve, to need an effort to learn them. But once you get over that initial hump, the tools are suddenly very applicable nearly at the daily basis. From my experience, I really would recommend to have a look at them, so that later, when the actual needs arises, you would know that the powerful, open source tools exist to help you out.

BlogicBlogger Over and Out

Link: Users hate apache server

2005-07-22T21:32:00.000-04:00

Ok, so they don't. But there are some bits they do hate and Rich Bowen goes through big issues present in apache's configuration file. Warning, it is a PDF file.

What I find interesting is that we see the full circle feedback in action here. Because the people supporting apache are mostly the same people who are developing (or documenting) it, they have credibility and strength of getting things done. Sometimes, within days....

On the other hand, in a big company the support is done by a separate team, whose only channel of communication is via bug fix requests. From what I have seen, there are no easy channels for a support person to suggest a feature to the developer just because it drives customers crazy. It has to come forth from the developer himself/herself, from customer surveys created by marketing, or through osmosis process where each layer from support upwards feel enough pain to finally pass it on unofficially.

I would love to be proven wrong here. I really would. I have even tried checking all the Safari books on 'customer support practices' to see what they say about such suggestion channel. Big nothing. The support is at the end of the chain and their opinion is not worth listening to.

In the best case, support people can create their own documentation and FAQs to help customers, but nobody would look at it and see whether there is a simple developer level fix to stop a lot of annoyance level issues.

BlogicBlogger Over and Out

Link: Socket + XML -> hard to troubleshoot issue

2005-07-21T22:22:00.000-04:00

Kohsuke Kawaguchi (Sun engineer working with XML) writes about what happens when you try to read XML directly from a socket stream with expectations of actually writing back to that socket. Specifically, a big nothing. The connection either hangs or dies without sending the response.

Kohsuke gives a clear step by step explanation of why that happens and how to prevent it in the future. If you work with XML, read this article just in case you will run into this in the future. It will save you a lot of troubleshooting time.

Now - after reading the article - the question is how would one figure this out without knowing the XML code backwards?

Since the issue is network related, I would probably start with my favourite multi-tool Ethereal. With the tool running, doing the test case should show that the connection is closed too early by the server. From that point, I would start tracing which bit of code is calling InputStream.close(). Or I would instrument/substitute/wrap the InputStream implementation and do a stack trace dump when its close() method is called.

That's what I would do. I wonder though how Kohsuke did it himself.

Update: I posted this as a question in the original article's comments and Kohsuke answered it.

BlogicBlogger Over and Out

Some java technical support patterns

2005-07-14T22:34:00.000-04:00

I while ago I was asking whether there are any technical support patterns out there. After all we have architectural patterns and design patterns already.

Turned out that BEA - the company I used to work for - has put together a number of the technical support patterns related to its products. But since BEA products use a great number of base technologies also used elsewhere, the patterns are useful to more than just BEA customers.

Unfortunately, they had put it behind a really long and ugly URL (*cough* BEA Portal *cough*), so nobody knows about. If you ever have to troubleshoot complex java problems, check it out and you may well find it rewarding. There are even some viewlets, or as it is popular to call them now screencasts.

BlogicBlogger Over and Out

Link: To blog or not to blog, is there a danger?

2005-07-13T22:16:00.000-04:00

The Chronicle of Higher Education has published an article which argues that a blog is often a bad thing to put on your resume or even to have within a google reach of your name. As a proof, they show a lot of negative things they found out about interview-ready people whose lives turn out to be more complicated than expected.

Perhaps when the person in question is an education professor with plenty of publishing opportunities and the blogs are only for personal aspect of their life, they should not have it listed (or searchable by their name).
On the other hand, for technical people a blog is often the best way to show what we know, what we care about, the problems we solved that might be useful for other geeks to know. And we have enough strange people in our midst with or without blogs that even some technical bile does not change our opinions of people.

At the same time, the article has some strong points about political views, etc. Putting them into the name-identifiable blog is like tattooing your affiliation on a forehead, bound to cause issues. One's choice of course, but a much bigger choice than people seems to realise.

I have my political views, but they are not in the blog. Nor are my food preferences or pet love/hates. On the other hand, if somebody decides to judge my English from the blog, I might be in trouble..... :-)

BlogicBlogger Over and Out

Disk Fragmentation is becoming a bigger threat than viruses? You must be joking!

2005-07-13T21:19:00.000-04:00

In the latest eWEEK paper edition, there is an advertisement that starts with the following: Why disk fragmentation is poised to outpace the virus as the biggest threat to productivity.

And it tries to prove this point by saying that viruses make our computers slower and making users frustrated and that so does the disk fragmentation.

Furthermore manual defragmentation (as bundled in at least 3 versions of Windows) is not good enough because the damage is already done. While can't link to the eWeek's ad, I found an online copy of it.

Now, I don't know about other people but I hate viruses because they

Slow down my computer (ok, I may grant disk fragmentation that one)
Actively resist uninstall attempts (hmm, defragmentation is never 100% complete either)
Can destroy my files or even entire hard drive (disk fragmentation does not)
Try to attack other computers (nope)
Can steal my data and pass it on to the virus creators (nope, disk fragmentation does not cause this one either)

So basically, the Diskeeper Corporation/Executive Software (because that is who the ad is for) tries to scare people into buying full version of their software for something that is a minor annoyance. And with the ever-increasing hard drive sizes, the issue is becoming smaller and smaller.

I wonder if their sales are falling so low that they are starting to get desperate enough to use these scare tactics. Or is this just a normal part of their Scientology methodology?

Oh yeah, and if you expect any support from the company for the product, you may just find an unexpected complication. Worse than that, in the age of the automated web update, would you actually trust them? After all Germany does not.

BlogicBlogger Over and Out

Viva the solaris 10 supportability

2005-07-04T23:03:00.000-04:00

It looks like Sun is really serious about making Solaris 10 well supported. I am talking about DTrace facility that allows scripting what previously had to be done with truss/strace and a lot of strategic greps.

Better yet, DTrace can hook into Java and trace the function calls across Java and native code alike.

Adam Leventhal talks about this issue and has even presented at JavaONE 2005 with this information.

I am really looking forward to this technology maturing and hope it will be adopted by the Java support teams everywhere.

BlogicBlogger Over and Out

Links: How to get good customer support

2005-07-03T12:20:00.000-04:00

Couple of articles discussing the customer support and what both sides could do to make it better.

An article from the Mobile Magazine where they called a number of support lines and compared their processes and results from the user's point of view. What is good about this article is that the problems were known and the same for each call, so you could judge for yourself who went through a troubleshooting process and who only had a decision tree to rely on.

A blog entry from Jason Fried, who supports Basecamp, Backpack and Ta-da list. He writes how to get a good customer support by being nice to the support person and providing them with the relevant explanation. The entry also has more than 50 replies elaborating on the issue and providing additional links.

MySQL support page for technical level examples and for additional couple of links to other detailed descriptions.

Finally, if you want to make your own mark, there is a Wikipedia page with the basic information that is crying out to be fleshed out.

For myself, I am starting to wonder if the only people who really get what the suggestions mean are those who had to do some customer/technical support themselves. Until then, it is often hard to understand why you are being asked for the configuration file every time, even if you believe that nothing was changed. (Answer: Often people will change something to try fixing a problem and then change it back but not completely. Everything looks like before to them, but is not anymore.)

BlogicBlogger Over and Out

Discussion: How should an application's log work

2005-07-02T01:00:00.000-04:00

An oldie but goodie discussion on slashdot about what needs to be done to make sure that application logs are useful to actually troubleshoot problems.

At reading level 4 (as linked), the suggestions actually make sense.

Try a quick test. If you are developing a software which produces a log file, or better yet many log files, how long would it take to extract all events produced by user X between 2pm and 3pm of July 30th?

You need to quickly locate files (where is the location encoded?), extract a regexp defined subset of time (do your log messages split over multiple lines and therefore not greppable?), and narrow it down to a specific user (do you even record that in the logs?).

And that's a basic task. With Java, you also have to deal with multiple threads, which most applications don't bother identifying. If you don't log it, even the most clear event sequences under single thread condition start looking very confusing when multiple threads are running similar tasks at the same time.

BlogicBlogger Over and Out

Do you mind closing that support case?

2005-06-28T22:59:00.000-04:00

Andrew Wulf writes - with feeling - about the clueless operations group in his company and how they manage to damage anything they touch.

Great reading, but I have to take an exception with the following passage:

As usual they initially blamed the programmers, then blamed the Java VM, BEA Weblogic, and the LDAP classes from Sun. They even called BEA to open a ticket on why Java could delete people from LDAP even though no delete function was even being called. I bet the yucks on that one hasn't stopped yet.

So, you know it is not BEA's fault, and yet you allow the support case to drag on. I had my share of the prove the ldap code does not do that cases and they are painfull.

Would you be so kind as to make sure BEA knows that the case had been resolved. You don't even have to say that it was your fault. They would really appreciate it. Please!

BlogicBlogger Over and Out

Visualizing Weblogic's config.xml

2005-05-08T17:33:00.000-04:00

config.xml is a central piece of Weblogic's configuration. And for the problems spanning multiple configuration elements, it is often easier to look at the whole configuration file at once.

However easier does not mean easy. Reading and searching through XML files is hard, especially when element/attribute meanings need to be looked up.

To make it easier, Eugene Kuleshov has come up with a XSLT transformation that will lay out a config.xml in a nice HTML format and will also provide reference document link within one click.

The next step would be to allow elements to cross reference each other, as there is a lot of internal cross linking within config.xml. This would of course depend on meta-information about linking patterns which is different between different versions of Weblogic.

The transformation as provided should work for Weblogic 6.1 through 8.1. Diablo/WLS9, as far as I know, has a different format of config.xml.

BlogicBlogger Over and Out

Applying open source philosophy to the end of the world!

2005-04-08T19:19:00.000-04:00

I was reading a SF book, where due to the End of the World, shops were being looted to the extend where the shopkeeper decided to just give everything away. And that is where it hit me. He could still make money out of it.

Nobody's house is big enough to have everything. The shop owner could give away the goods, but sell the advice on what one needs to survive the longest in relative comfort (toilet paper, can opener, large UPS battery for the computer, etc). He may not make as much, but it certainly better than just giving it away with nothing returned. He may even be able to advise on what to take from other shops.

So, there we have it. People kept saying that the Open Source Philosophy of giving away the goods and selling the services would be applicable to many areas. I believe I have found an unexpected confirmation for it.

Oh, and the book was Messiah Node by Lyda Morehouse. It is 3rd in the series and the whole series is strongly recommended for a geek (only slightly less so for a strongly religious geek).

BlogicBlogger Over and Out

Re: Put down that decompiler - part two

2005-03-23T19:13:00.000-05:00

In my previous article on this topic, I had given an scenario which justified the use of the decompiler. I have now remembered a much better example.

A customer had problems after upgrading Weblogic 8.1sp2 to 8.1sp3. His JSP - which was rather large - suddenly stoped working with 'method exceeds 64K' messages. The JSP itself hasn't changed between the versions.

The obvious suggestion here is that BEA's JSP compiler started to generate larger code. We had one issue matching that, but reverting to the old behaviour via flag switch didn't help.

Unfortunately, due to the complex nature of the JSP, I could not replicate the issue in my lab (too many dependencies and propriatary taglibs). So, I get the customer to send me the class files generated by both versions of JVM (JRockit's equivalents of SUN's 1.4.1 and 1.4.2).

The class file sizes are different!

So, given the same input JSP, but different resulting class sizes what would you suspect? I figured that the JSP compiler must have other, not yet identified, changes and get the customer to give me java sources generated by the JSP compilers.

Now, reading the source generated by a JSP compiler in my eyes is just as much of a voodoo and 'under-the-covers' work as any other decompilation. But of course, it was our (BEA's) compiler, so I had to do it anyway. No way to shift the blame to the other vendor here.

So I slog through the source versions trying to identify the differences that are NOT comments. Eventually, I find out the sources are the same!

So, same JSP produces the same Java source, yet the class sizes are different. This is where I had to pull out a decompiler or to be exact the disassembler (javap).

Unfortunately, decompiling both classes produced so many differences on a bytecode level that I had to go back and try to create a replication.

To cut the long story short, it turned out that the difference in file sizes only appeared when there was a try/finally clause. From that point on, the simple test case showed that the content of the finally clause was duplicated into the try class. And - worse yet- with nested try/finally clauses, the outermost finally could get copied up to four times.

As you can probably imagine, BEA's JSP compiler had a lot of nested try/finally clauses in an attempt to recover from nearly any JSP failure possible. The resulting copies of large finally clauses were more than enough to blow the method limit out of the water.

So, I had a replication and I had an explanation of what happened. But I still did not know why. As the problem was happening in JRockit JVM, I escalated it to the JRockit engineers.

They came back and said that there was no JRockit change that could cause that. But, they spoke to their Sun counterparts and somebody mentioned an internal bug report that may have something to do with it. It took a while to find the internal partner's account number, not something they just give out to random support personnel.

Eventually, I got to read the bug and, sure enough, Sun changed the behaviour or javac to generate duplicate sections (Bug ID: 4494152). And the reason they did that was so trivial that there must have been other ways to fix it. In fact, the change causes some ugly cascading bugs (1, 2).

In summary, there would have been no way I were able to identify the problem without breaking out the javap. And no way would anybody else be able to identify a non-public Sun change without having a clear cut replication case.

Disassemblers, decompilers, tracers, etc are all effective tools of a developer. Refusing to ever use one due to the vendor's responsibility concept is not really a supportable stance. Even book publishers think that way (1, 2).

BlogicBlogger Over and Out

Podcast: Fight, Fight: Doctorow vs. Scoble on AutoLink

2005-03-22T20:03:00.000-05:00

You don't normally get to hear the near-famous people fighting. Usually, they have their responses polished or they agree to disagree. Not so in this recent podcast on Google's AutoLink from the ITConversations.

In this one, both Robert Scobble and Cory Doctorow lose their cool and start really throwing words around without listening to each other. Which is a pity, because it would have been nice to see them actually give in a point or two to the other side.

It would be interesting to see what the Great AutoLink FlameWar will end up with, though I have to say I am on Cory's side. I have just started using GreaseMonkey for the Firefox and think it is Good.

Either way, I strongly recommend the podcast for all its noise to signal ratio.

BlogicBlogger Over and Out

Re: Put down that decompiler

2005-03-22T08:29:00.000-05:00

Andrew Savory(?) writes that if you have to stoop to decompiling a product to figure out a problem, it is the vendor's fault because they chose not to supply you with the code and documentation. So, if the vendor does not provide the source and the particular functionality is not well explained, it is completely vendor's fault. I wish it were so simple.

I do agree that in the ideal world, every feature would be documented and/or working 100% correct. In reallity, it may be somewhat different.

To give an example, Weblogic server allows to add your own security providers by dropping a jar into a particular directory. So, one of BEA customers gets a patch for the security provider code and dutifully drops it into the directory as per detailed instructions.

And the fix does not work. Couple of support rounds happen where frontline support checks and rechecks that the customer has the right version of the file (yes), that it was put in the right directory (yes), that the fix does match the problem (again yes). All is correct, but nothing changes.

The case gets escalated and out of desperation, we (effectively) stop beliving the customer and ask to provide a listing of the security provider directory showing sizes and dates of all the files.

And then we realize that instead of a single jar (patched providers bundle), there is also another file in there called something like providers.jar.bak, an original copy of the bundle. Removing that file makes the problem goes away. Looking at the source code, it turns out that every file in that directory is loaded up as a jar with a provider bundle. And so, the old bundle kept being picked up.

Whoever wrote the documentation, asked the old provider to be backed up. And so they were. But nobody expected the backed up file to be in the same directory. A bug was filled and issue was fixed.

But all this took a while. Mostly because we did not have access to the customer's system and everything was being done from the arm's distance. If the customer was conversant with a decompiler, they could have reverse-engineered the code that loaded the jar and ether noticed the problem right away or put log statements in to print out when the jar was loaded (and see it ping twice, rather than expected once).

The lesson here is that, yes this was dodgy and should have been fixed (and was). But there is a range of situations where the problem lies in shades of meanings and reverse-engineering will often give a better and faster answer than jumping up and down and screaming for the vendor to fix 'something, anything, whatever it is'.

On a related note, it might have been possible to identify the same problem by running a FileMon and noticing that backup file was actually read in where it should not have. But in my eyes, FileMon is just one of the tools in reverse-engineering/decompiling toolkit, so
it would be out of Andrew's reach as well.

BlogicBlogger Over and Out

RE: MIke Clark's article on making software troubleshoot itself

2005-03-20T00:14:00.000-05:00

Mike Clark is looking at automatic first-line support by building some auto-support functionality straight into the program. It is an article worth reading.

One thing that the article does mention - in a somewhat negative way - is the 'secret' checklist the developer would go through to troubleshoot the application. He suggests that the 'secret' checklist can be automated by the tests. In my own experience (with BEA weblogic support), the checklist is usually more like a branching tree of decisions that would be fairly hard to encode as any simple set of JUnit tasks.

I wonder how real Mike's examples were and what percentage of the checklist he managed to automate?

To be perfectly clear, I do not think his article is wrong. Far from it, I think it is great direction to explore in further details. I just don't want people to accept the support checklists as 'secret' in general. I think the support techniques should be paid attention to and built into the great knowledge web that anybody can learn from without having to rely on an expensive specialist at the other end of the line.

I know that there is a lot of FAQ style websites dealing with the specific problems and they are good. But we also need some meta level discussions on the open forums as well.

Perhaps a support-wiki or Wikiversity course would be the way?

BlogicBlogger Over and Out

Technical support patterns, where are you?

2005-02-28T22:21:00.000-05:00

We have Design patterns, Analysis patterns and even Maintenance patterns. But where are the Technical support patterns ?

Oh, we have plenty of horror stories, but nobody seems to be drawing lessons out of them.

The nearest I found was the Maintainability pattern language, which is worth a read. But even that document is too generic to be useful in the trenches.

I am still looking. If I don't find anything, I will try to codify my own experience, but surely somebody else has done at least some of it already.

BlogicBlogger Over and Out

Re: IBM's HeapAnalyzer

2005-02-18T23:15:00.000-05:00

IBM has refreshed the HeapAnalyzer software. Goodness of their heart? Not so fast.

Apparently, HeapAnalyzer used to be an internal IBM support tool. Paid customers who had problems with IBM JVM would take the memory snapshots and IBM support would use the tool to do the analysis and recomendations.

Now - with the tool available to the public - IBM support will no longer analyse customers' memory leaks. Apparently the tool is so intuitive and produces so good results that the customer can figure out the answers themselves after playing with the tool for a short while.

So, when a big customer of both BEA and IBM products was having production memory leaks within IBM's JVM, we (BEA support) had to help them to troubleshoot it because IBM refused. Something not quite right here.

Oh yeah, and even after spending more than an hour on it, the tool's output and interface was still confusing to me. And with Vim as my primary editor, I should be fairly familiar to obscure interfaces.

BlogicBlogger Over and Out

Thoughts on: Google Video Search

2005-01-26T18:00:00.000-05:00

So, now Google can search TV programs based on the subtitles. While there is still a lot of improvements, I can see one interesting use of it straight away (well, within reach anyway).

Connect Google Video Search, Google alerts and Tivo (or more probably a custom coded PVR) and you can chose shows to record in slack time based on how often those shows mention your favourite keywords. Make the interface real-time (probably not very likely) and you might be able to trigger PVR to start recording a program when a keyword is mentioned.

It might be possible to do this now in PVR software by making it watch the captions keywords, but from what I understand you need to actively monitor specific channel for that. The Google method would allow to collect the statistical information and fine-tune the results over time.

Actually, there is a business idea here for a paid service. Somebody could setup several capture cards in parallel and have subscribed keyword watches with real time ping triggering user's PVR. Let me know if you make a business out of this idea or want more details on how that could work. It is quite possibly not a new idea in its basic form, but with more people building PVRs it might be the time to reexamine it.

If you have any feedback on the Google's new feature, you can always comment on Google's discussion group that is setup specifically for this.

BlogicBlogger Over and Out

Link: The BileBlog, without the Bile

2004-12-06T22:45:00.000-05:00

I read Hani's BileBlog incidentally. I used to have him in my RSS reader, but the language makes it hard to concentrate on the arguments (valid as they often are). I still get him through search or topic aggregators occasionally.

So, when I saw Greg Luck's deBilefication and further discussion of the Hani's entries, I read them very carefully and empathised much more with the topic than when I read the original articles.

Perhaps Graig could setup translating service for those of us who do not require our content covered with a colourful language gravy.

I would also love to know where did the 100 types of tests chart came from. I wouldn't mind having one myself. Was it from the linked book?

BlogicBlogger Over and Out

Link: A revenge Zombie is still a Zombie

2004-12-02T21:53:00.000-05:00

Frank Carver is not impressed with anti-spam screensaver. I think there is an additional danger beyond just handling the control over to the corporation.
And that is a danger of somebody hacking the communication/update protocol and substituting the list of hosts. And since the screensaver was installed on purpose, user will ensure that all the firewalls are cleared and will ignore any warnings regarding the program.

On the other hand, the download website already seems down. Did someone perhaps repointed their program back at them? Or was it the Slashdot Effect?

BlogicBlogger Over and Out

ReRe: support's view on customers

2004-12-01T07:25:00.000-05:00

The internet could be a very small place. One of my clued-up customers is aparently also my subscriber (and vice-versa).

He was wondering why questions from 'intelligent customers' do not go straight to Backline (tier-3) support.

There are several reasons.

First one is that Backline (BL) and Frontline (FL) support (as BEA names tier-3 and tier-1 correspondingly) actually have different strength.
Frontline support is very good on collecting initial information, checking documentation and internal support databases. They may seem a bit information greedy, but often missing a configuration file at the start may lose a lot of time later. Essentially, FL DREs are very good at connecting the 'known' dots.

Backline support engineers on the other hand are much better at the hard problems and at problems with hidden variables. We are also the ones familiar with non-BEA technologies and other developer land tricks.

When I (a BL support) had to do a Frontline engineer's job, I sometimes find out that I miss basic information nugget that would have been collected by a FL support and matched against a known issue. I would get to the same information via replication or source code search which is more general technique, but may takes longer.

In most of the cases, leaving the case to FL DREs until they are ready to escalate is really the best way. Especially, since FL will often bring in a BL engineer behind the scenes to just get a read of a case and confirm the direction taken. That way, the customer still deals with one person, but gets the expertise of two (or sometimes even three) engineers.

The second reason is to do with a field of expertise. A JMS specialist may not know much about certificate signing; an EJB guru is fairly unused to the network issues cases by firewall for remote clients. Support engineers (FL and BL) are trained to be jacks of all trades and even FL engineer is more than enough support presence for a customer working out of his/her depth.

The final reason is simply financial. Higher priced support offerings allow a faster access to senior DREs (FL or BL). Some offerings even have dedicated DREs (at least in their own timezone). Also, the more money goes into the support. the more company is willing to hire additional people to do it.

Finally, just to comment on the support question Dim was writing about, I looked it up. If I found the correct case (from mid 2003), it was actually resolved by me via replication and turned out to be the correct - if not obvious - behaviour with the given configuration.
Customer was frustrated because it took a while to figure the problem out. Why did it take so long? Well, it turned out that I was missing customer's configuration file (config.xml). In absence of it, I followed written down configuration instructions, which turned out to be incorrect on one tiny, but crucial point of JMS file store configuration. A frontline support engineer would have ensured that the config.xml was collected as a compulsory first step.

In conclusion, I am sure that there are good and bad cases. BEA's near future goal is to actually concentrate on making the support better, faster and more proactive. In fact, I have some visibility and influence into the process. If you are a BEA customer and have a real suggestion on how BEA support could be better, put it in the comments.

BlogicBlogger Over and Out

Link: What a difference .1 makes: HTTP 1.1 Persistent Connections

2004-12-01T01:38:00.000-05:00

Subbu Allamaraju has written a very good description of the tricks HTTP 1.1 goes to utilise connections more efficiently.

Reading his article makes very clear that persistent connections are quite tricky and developing new code dealing with the topic requires careful thought.

And, of course, troubleshooting persistent connections is quite hard. Ethereal comes quite handy in here, but even that does not know how to extract individual parts of conversations. Does anybody know free tools that work in a network sniffing mode (not proxy mode)?

BlogicBlogger Over and Out

Slashdot: Should vendors have root access to customer systems?

2004-11-30T01:44:00.000-05:00

Interesting - and at comments threshold 4, meaninful - discussion on slashdot whether vendors need root access to customer's equipment/OS. Eventually, it diverges into dumb vendors vs. dumb customers.

At BEA, we don't actually need root access to the system. In fact, you have to go out of your way to run at port 80 or with other root related issues. Comes with a Java territory, I guess. In fact, we prefer not to login into the customer system at all, but rather collect logs and analyse them on our own machines.

On the other hand, I do feel the discussion of who is smarter to be close to my heart.

I think that after years of working as Backline (Tier 3) support at BEA, I find myself liking two extremes of the customers.

I like the customers who know what they are doing, how their system or network is setup and where their logs go. I like the customers who I can ask to run the software under nohup and redirect both stdout and stderr into the same file.
I also find that I don't mind customer with not much clue, but aware that they are out of their depth. There was a number of times when an engineer has to fix the system they weren't even aware of until that evening. And it usually is evening and - lo and behold - they have a whole night to sort it out.
These customers will follow my direction to the letter, they will tell me what they see, and they will read back the error messages with a precision. Yes, it takes a bit longer to go through the steps and the support engineer need to really think on their feet, but it is still ok. And when you fix their problems, these customers do know that you helped them.

The customers I find myself not enjoying are those in the middle. They think they know what the problem is, therefore they only send partial logs, snip out unrelated error messages and think the problem is obvious from a terse general description. (try Server crashed today. Please send patch.) For them, I find I have to explain every step in very clear written down details.

As an example, we often request an access log and server log in parallel. A customer like this would send access log for last 3 months and a server log from the last 3 hour run for the problem that happened 2 day prior. I have to re-request the log and in process explain everything about log roll-overs, checking the log entires' timestamps and general log locations. Not something you would expect to need to do for a sysadmin of a large organisation.

In terms of case types, the most hated situation around the office here is a customer that creates a new case (again, Server crashed, please fix) and requests an engineer to join the bridge where 30 people have been discussing this issue for 8-30 hours already.

Very often, we are not useful without seeing exact exception and/or log files in a context. Also, BEA product family is very large and - even though we support all of it - we have our own individual expertise areas. My hanging server skills are much higher than my JMS message lost skills (which my coworker excels in), so just asking somebody to join the bridge (and stay on it until resolution) will actually benefit less to the customer than spending extra 10 minutes on writing down the description.

For myself, I find it survivable to land in one of those bridge calls as long as the customer has a clue.

We have two very large USA telcos that like to do bridge calls. One has engineers with a very strong clue (even their CEO does), another seems to provide no training OR resources to their so-called sysadmins. Guess which one I hate to work with? Guess which one I will NOT consider as my provider, as I can see how bad their problem-solving skills are.

Anyway, enough rant. If anybody is interested in specific aspects of support life, ask in the comments.

BlogicBlogger Over and Out

Link: Weblogic and Active Directory Authentication

2004-11-29T18:12:00.000-05:00

Luke Dewavrin is writing about what it takes to get Weblogic use Microsoft Active Directory as an Authentication Provider.

He also mentions couple of issues that people get burned by the first time they use WLS Security Providers architecture.

Specifically, he talks about the need to set JAAS flags to "Sufficient" or "Optional". Let me reinforce that; the flag need to be changed before the provider setup is saved. Otherwise, you most probably will not be able to restart WLS instance again.
Fortunately, WLS 8.1 stores the information in the config.xml and it can be edited fairly easy. As a history note, WLS 7.0 stored the information in the binary form outside of config.xml and to edit it, one would have to follow recovery export/import procedures.

For myself, I always found Active Directory hard to troubleshoot, mostly because of the inability to get good logs. As a comparison, iPlanet Directory Server has very detailed and helpful access logs that I have used many times to isolate issues like incorrect configuration, slow authentication and even firewall timeout. I tried to find an equivalent set of logs for AD, but could not. Maybe they call it something else entirely.

BlogicBlogger Over and Out

Business model for ITConversations

2004-11-28T21:14:00.000-05:00

Doug Kaye is asking how to make his very reputable ITConversations podcast website actually bring in money.

For me, ITConversations is the only podcast that I listen to reliably as I listen for content, not entertainment. Pop!Tech series were absolutely riveting, especially WorldChanging.com and BioMimicry.

So, following is my idea which I was thinking about for a while and can see having worth beyond one site's commercial success.

The idea is based on having paid subscribers who then get to vote on the show they want to hear.
Following is a sample use case:

I join ITConversations and pay $X deposit (say $20).
At some point later, I find that I really want to hear (for example) Tim O'Reilly's views on podcasting. Or maybe I listened to a Memory Lane and want a follow-up on specific issues. Or I may want an interview on the state of the art of Machinima, which is just entering global consciousness.
I create a request item and commit $Y of my deposit to it.
The request goes to a vote with the other paid subscribers. They can discuss it and hopefully commit some of their own funds to it as a vote.
Once the total amount reaches the threshold, site owner (Doug) accepts the request and effectively makes a sale for future podcast.
When the show is produced, it is available to the people who committed funds to it.
The sponsored show may also enter general availability some time later (or be available at nominal cost to the other subscribers)

There are several advantages to this model:

There is no need for micro-transactions for each vote, as they are pre-aggregated by the deposit and can be managed via top-ups mechanism
Since the money is already paid, the perceptual barrier of entry (for vote) is easier.
Voting means that the topics selected will be user-driven and more timely than trend spotting.
If the sponsored show is only available to subscribers (even for free), it makes it very enticing to join, which requires some deposit. And once the deposit is made, it is easier to vote. This creates a positive feedback loop with more cash throughput.
Community around voting and discussion (which could be made public) will keep people coming back over and over again.
Advertisers will love seeing the vote results and perhaps target the ads at the specific subjects being discussed.
Money paid as deposit are transferred before being used up and therefore earn the percentage from early on.
Finally, once the system is implemented, it can be used for features requests and other website work that requires a confirmed commitment rather than bluesky ideas.

There are obviously some difficulties to resolve, such as amount thresholds, clear topic selection and satisfaction of paid content. But they are all solvable, at least in a long run.

I would be happy to answer further questions on this idea, if anything is not clear.

BlogicBlogger Over and Out

Suport Story: A tale of twenty cookies

2004-11-25T21:53:00.000-05:00

Very interesting story of what happens when built in defaults have silent limits. In this story, each ADF application modules (whatever they are) needed a cookie with default configuration. This silently runs into spec limitations.

Fortunately, Weblogic does not generate multiple cookies by default, though you could certainly assign one for each of the webapp.

For me as a support person, it is fascinating to read how other support people solve customer's problems. I wish I could read the internal case log for that particular support case.

If it were me, I would have asked customer to do the requests with ieHttpHeaders, enabled Webogic proxy debug log (if available) or put Ethereal on the wire.

BlogicBlogger Over and Out

Microrebooting paper's reasoning holes

2004-11-25T18:56:00.000-05:00

CIO Today noticed microrebooting research paper. While I agree that the paper is very interesting, I think there are some holes in it that were not explored (or at least not explained).

Specifically:

Early in the article, memory leak and resource leaks are named as the problems solved by rebooting.
From what I can see, microrebooting will only solve the issues where those resources/objects are held by instances that are cleared on microreboot. It will do nothing for pooled resources leaked because close/release method was not called. It will also do nothing if the memory leak is through creating too many classses at runtime (e.g. Stubs) or if resources are held in the static parts of the classes. This is because the classloader is explicitly not recreated.
Recommended rebooting sequence of the application server in the paper is as follows:
1. microreboot of the component transitive closure
2. then kill -9 the server
3. reboot the O/S.
It seems that something is missing between 1) and 2). Where is attempting to shut down the server via normal shutdown command? That would allow to synchronise the buffers, finish writing out logs, etc. Going straight to kill -9 is really an emergency exit and is highly not recommended. In the same vein, I am not sure how much good rebooting the O/S is going to do for Java AppServer. Of course, the proper shutdown takes time, but is faster reboot worth getting your transactions into the doubt state due to the corrupted JTA in-flight store?
Threads. The paper proposes killing all threads associated with the resources. I would be very interested in how they are proposing to do that well, given that Thread.stop is deprecated beyond belief and Thread.interrupt() does nothing for method synchronization deadlocks/bottlenecks.
Memory recovery via microreboot. The paper suggests microrebooting the components to free some space. A more effective suggestion in my thinking would be to have hooks into the component caches and to request to drop all cached content down to the working set or to the start counts.
So, a typical production system will start with 5 JDBC pool connections, but may go up to 100. Asking it to drop back down to 5 (or working set, whichever is larger) will free up a lot of cached result sets. Same goes for cached Entity beans, etc.
Finally, the main requirement for all this to work is a component storing its state in external transacted storage. Have they calculated whether the cost of shifting to external storage instead of faster in-memory/in-place is worth the time benefits of microrebooting in the long run. I am not so sure, especially with the larger HttpSession data sets I have observed in a real world.
The exact question here is: how often does one need to microreboot instead of full reboot to save the time lost on satisfying the additional constraints? My own sort-of answer here would be that perhaps extra half-second per response is acceptible replacement, but I would like to know what the researches themselves think.

Still, all the nitpicking aside, Weblogic does provide some things already that resonate with the idea if not the suggested implementation.

Component redeploy will try to remove all instances (and classes) on the EAR/EJB/WAR level and reload them in. In fact I am having troubles figuring out how the micro-reboot is better than this already available functionality.
Node managers will monitor your system and restart the node if any of the subsystems goes into the 'warning' state
JRockit allows to monitor memory usage and fire code triggers on crossing the thresholds. You can do what you like at that point.
Weblogic 8.1sp3 monitors JDBC pool entries and can timeout inactive connection and consider it leaked. It also provides connection leak profiling where non-closed connection will scream when it's finilized method is hit. It also provides resource retry for JDBC similar in spirit to what the paper talked about.
JMX and SNMP also allow to define all sorts of threshold with notifications triggers.
Thread subsystem will monitor request processing time and will log a message when a single requests takes longer then a threshold value to process. In the next version of WLS, it will also print the stack trace of the stuck thread. Notice that it will not kill the thread.

But I do welcome any research that makes support job easier. Check out the other interesting papers at the Microrecovery and Microreboot center

BlogicBlogger Over and Out

Link: Do's and Don't's of Logging

2004-11-25T18:25:00.000-05:00

Jeff Mesnil is a developer who had to play 'support' for a bit. Specifically, he had to read logs produced by others. He immediately discovers a lot of lessons that are not obvious when you only read your own code.

To his list I want to add a couple of things from my own support experience:

Make sure you have some way to identify which log entries belong together. Use TransactionId if you have one, thread name if you don't. Weblogic logs put thread name for each message. If this is expensive, use thread ID number instead and correlate them later with Thread dumps.
Always put timestamps in. Again, if this is expensive, put System.currentTimeMillis() instead of (new Date()).toString() and put a log entry at the start (or periodically) that shows both (to clarify timezones and allow correlation to other dated logs)
If you have a long operation, try to put a log message at the start and end of it. That way, when you later want to discover what your real life processing time is, you can parse those entries. Again, this requires ID and timestamps. One of the frequent problem we have is customers running into default timeout values being too small, but not knowing what the real value should be for them either.
Remember that logging is expensive and (at least for log4j) is gated at the write method. So, if you have 100 threads all logging away into the same file, at some point they will all bottleneck on a write method. And if the write method needs to rollover the file or build a complex log string, they will all just wait. I have support case scars to prove this point.
Try to grep your own logs instead of eyeballing them. If you have difficulty getting a good set of log entries, the person supporting your application will have even more problems.

BlogicBlogger Over and Out

EPIC is coming. Want to become its editor?

2004-11-25T17:07:00.000-05:00

A very strong (and quite dark) futuristic look at personalised news and media consolidation (8 minutes). Via Doc Searls. Really worth watching for anybody in the blogosphere.

The idea as such has been covered in Science Fiction before, but the anonymous author of this piece has used the real names and very realistic sounding dates.

After watching the piece come back here and think about this. If EPIC is coming, do you want to be its editor? I do!

So, given 10 years, what skills do I need to cultivate to become one? Any suggestions?

BlogicBlogger Over and Out
P.s. I hope it will get put to archive.org or on P2P fast, otherwise it will kill the website very fast.

Technorati response to the Bloggercon Overload Session

2004-11-25T01:20:00.000-05:00

I wrote about my impressions from Bloggercon's Overload session before. Now, Tantek (who works for Technorati) details specific technologies that could help to implement the wish list items mentioned.

BlogicBlogger Over and Out

Please don't make me support this WSDL 2 spec

2004-11-22T21:57:00.000-05:00

Rich Salz writes about major problem with WSDL 2.I really hope it gets fixed before BEA decides to implement it. Otherwise, it will be completely unsupportable.

We already have problems with handling the customers bitten by vague specs, but this looks nightmarish.

BlogicBlogger Over and Out

On Bloggercon III Overload session

2004-11-19T04:41:00.000-05:00

It was very interesting listening to the Overload session recording from ITConversations.

Unfortunately, there are no transcripts of the recording available. Having to listen to the record second time just to pick up points worth of commenting on is somewhat annoying.

One thing however I do want to say. Robert Scoble was the moderator, due to his 1000-blogs-a-day reading routine. Unfortunately, he did not contribute enough of his own methods to provide the enlightment. If anything, it was the opposite. He does not like categories, he does not spend much time thinking what goes into linkblog, nor did he seem to care for metadata.

However, I am sure he does have some techniques that would be very valuable to know.

So Robert, if you are reading this, how difficult would it be for you to video yourself while you are doing the scanning and reading process. The way your eyes move, the pauses, the hands will surely tell us what is hard to describe in words. And if we could see what the screen actually shows at the same time (in a side by side view with a camera), it would be yet better (more than twice as good actually).

Perhaps one of the HCI labs equipment could do something like this.

I enjoy your videos of other people. I would enjoy one of you doing what we can only hope one day to achieve.

BlogicBlogger Over and Out

Re: Code Generation: good or bad?

2004-11-15T23:26:00.000-05:00

David Rupp writes - eloquently - that he does not understand why people are reluctant to use software that auto-generates code (e.g. Hibernate and AOP/AspectJ).

He is of course right. Code generation on the fly is in exactly the same league as JSPs, EJBs and dynamic proxies in terms of how the code one writes does not corresponds to the code that actually runs.

But he is also wrong. What we have here is not a Good/Bad breakdown, but rather a spectrum. And the measure of that spectrum is how easy it is to trace the problem with the generated/compiled class back to something one can change.

In the case of JSPs, it is fairly easy to trace back as most of the application servers allow to keep the generated java file and - at least Weblogic - will put JSPs original line numbers as comments in the generated Java code.

The same applies to any offline precompiler. However complex they are, there is always a class file produced in the end that can be used as a reference.

Finally, a third party library can - in desperate situation - be decompiled and the decompiled source will include the line numbers (unless the class had been stripped or obfuscated).

On the other hand, AFAIK code generators do not usually bother with putting in source line numbers, nor do they save anywhere the code blocks that become classes.

I have a case to the point. A week ago, I had a support case which had a ClassCast exception in the dynamically generated wrapper class that Weblogic puts around Oracle driver's proprietary methods. The top of the exception stack was:
at weblogic.jdbc.wrapper.Blob_oracle_sql_BLOB.getBinaryStream(Unknown Source)

Of course, I did not know at first that it was a generated class. I could have searched through the whole WLS source code base forever without finding it. The only hint was the class name. Seeing wrapper has triggered the memories of other parts of Weblogic where we use similar tecniques (RMI, EJB, etc).

Eventually, I traced this to the part of code which generated the class on the fly. And when I did, I found - as expected - that the class is just generated in memory and is loaded from bytearray without any export or debug functionality.

Basically, we were generating a weblogic class at the runtime that will implement all Oracle's proprietary methods, but will add pre and post processing to them. A not-so-dynamic proxy, in other words.

I had to build a custom patch to dump the file out and then replicate the basic logic to trigger the class generation. Once that was done, I finally had class file to decompile and analyse.

However, this was comparatively simple problem in that it was easy to reproduce and the code generated was conditional on classpath (Oracle drivers), but not on any runtime conditions. Were it not the case, there would have been no way for me to reliably confirm the code behaviour.

The same issue will apply to woven-in Aspects once they become very popular. What is a boon for a developer is often a woe for the support engineer.

I think the lesson here is that any code generation framework must ensure that there is an easy way to get class bytecodes, whether through debug flags, explorable ClassLoaders or any other option.

BlogicBlogger Over and Out

Koders: Another search mini-app

2004-11-08T16:32:00.000-05:00

It looks like search is starting to gain momentum, both desktop and service based. In the last couple of weeks, we had:

Desktop Search

Google Desktop
renewed interest in Lookout (Lucene .Net based)
nascent Lucene Desktop is generating a lot of interest, though it is hard to say how it will deferentiate itself from Zilverline
And many more. See comparison on FileHand's pages

Web based search miniapps

JarHoo will help you to locate a Jar where a class is defined
Koders will allow you to find which open source project contains code you may need and will show some interesting stats about it (e.g. Projected reimplementation time). It may or may not be based on Lucene for the search component.
I have to say though that I did not feel as a target audience for Koders. I wanted to get in, type the class name , find the useful project that had it and go to that project's home. I could do all, but the last one. Instead, when I clicked on the project's name, I got drawn deeper into the Koders site and was offered to look at statistics and source code. Oh well, Google is always a click away.

I think in the future we will see yet more of search miniapps, possibly very specialized. I can think of several that would be useful for my BEA support work.

Just as an example, I would like a small application that would index all descriptors for both standard and Vendor specific deployment descriptors and then would allow me to search by name substring. That way, I could search for timeout and get a general overview of what timeouts are configurable, in which file and exactly where in deployment descriptor's hierarchy.

Any takers?

The biggest problem of course is who will pay for traffic. JarHoo was running google ads, but it still needed to be sponsored by a hosting company. Such good will may not last long.

BlogicBlogger Over and Out

I always was curious what writing for O'Reilly was like

2004-11-07T21:35:00.000-05:00

Guillaume Laforge writes about the 'New Author' treatment.
I hope he will continue writing about Life as an O'Reilly Author.

I also wish somebody with a different publisher would write about their experiences.

BlogicBlogger Over and Out

What type of a Framework Fashionista are you?

2004-11-04T19:17:00.000-05:00

Lara D'Abreo decided to classify developers according to their attitude to Java Frameworks.

I feel I can claim to be Pragmatic, but the truth will most probably reveal at least some of Oblivious. I also seem to remember Dogmatic period of my career, but I have (hopefully) overgrown it a while ago.

BlogicBlogger Over and Out

Is gmail planning to introduce new domain names?

2004-11-02T17:42:00.000-05:00

I have a gmail account. Today, when I signed in and then signed out, I suddenly noticed that in the 'username' field it is now showing my full email address. This is instead of just username. So, 'user@gmail.com' instead of just 'user'.

If I am not imagining things and that was the change, then there is only one reason I can guess for that. Gmail is planning to introduce additional domains (as mail.com did) and automatically substituting full address for login is a transition step.

Any guesses on the additional names? Gbrowser anyone?
BlogicBlogger Over and Out

Goodbye Dev2Dev, Hello CodeShare

2004-11-02T07:21:00.000-05:00

A while ago BEA has introduced Dev2Dev, a way to reach out to the BEA community. The website has gone through at least 3 changes, but was still not overly popular. Finally, BEA has realised that a 3rd party will do a better job out of it.

Meet CodeShare, a new project developed by BEA with great help from O'Reilly Media and CollabNet.

The biggest problem with Dev2Dev in my opinion was that you could publish something to it, but it was hard to update. Some of the older sections (like utilities)were also hard to find after a website redesign.

Hopefully, CodeShare - which in many ways seems modelled after SourceForge - will fix that. I, for one, am looking forward to using the new site.

BlogicBlogger Over and Out

Standalone XSLT/XQuery tools from Altova

2004-11-01T19:02:00.000-05:00

Altova, the maker of XMLSpy has released several standalone engines that previously were only available as part of the editor itself. They have XSLT 1.0, XSLT 2.0 and XQuery engines. XSLT 1.0 implementation looks very complete, XSLT 2.0 is less so.

Of course, my own preference is XMLStarlet. I call it The find of XML world because I like to do a lot of ad-hoc queries on XML files. Also XMLStarlet is available on multiple platforms (Altova is for Windows world only).

Still, the more engines are out there, the better it is. We can always do a consolidation later, once the needed feature set is fully understood.

BlogicBlogger Over and Out

Jarhoo is a good step in a right direction

2004-11-01T00:12:00.000-05:00

If you often wonder which jar a particular class comes from, Jarhoo has indexed a number of common distributions of java applications and libraries and allows to do a quick lookup by a name substring or a jar file substring.

Among other applications, it includes BEA Weblogic, so it is sometimes useful for me in troubleshooting classpath issues.

There are some growing pains still:

It does not indicate which Service Pack of the product is uses. One assumes the latest
It does not allow to add or remove product from a search list. It would have been useful to speed up the search by adding - for example - +product:WLS81
There seem to be some sort of double escape of < sign in Jar File Locations entries

An interesting thing is that just from looking at the interface, I can tell it is running Lucene under the hood. The '2 seconds' response time especially is a telling sign.

I wish the project well and I hope that, based on its success, many more mini apps will appear. It is certainly the right direction.

BlogicBlogger Over and Out

Can't afford Weblogic AppServer. Check out the discussion of the free ones.

2004-10-29T01:00:00.000-04:00

A discussion on JavaLobby is trying to compare the AppServer features and ease of development. Only free AppServers need to apply.

BlogicBlogger Over and Out

Using rsync to synchronize with remote Weblogic

2004-10-27T21:41:00.000-04:00

TheLastBlogger showed us that not every vendor limitation need to be solved by the vendor's fix.

Hurray for flexible utilities and solutions! I hope TheLastBlogger will keep recording the interesting solutions discovered during development/migration.

BlogicBlogger Over and Out

Perfect multicast storm

2004-10-21T09:51:00.000-04:00

This is a story of a well meaning default causing more problems than a randomly picked value. Read it if you are running BEA Weblogic cluster on a switched network, especially with a CISCO switch. Read it even if you do not run WLS cluster, but are interested in TCP voodoo.

Weblogic server is a combination of many technologies. Quite a number of these technologies used to require a dedicated professional to configure and maintain. I am talking about such subsystems as SSL, transactional JDBC and - case to the point - multicast. Back then, the user interfaces and procedures were arcane and nobody was surprised that a specialist was required to make sure things were working.

Nowadays, people seem to believe that the default settings for a sub-system they are not familiar with will fullfil their need - whatever that need may be. And most of the time, the defaults do seem to work, especially when tested in an environment that does not match production setup in throughput, domain, or network layout (e.g. cluster). However, when the code is promoted to production, unexpected happens.

For this example, let's talk about multicasting. By itself, multicasting is a fairly heavy voodoo and applications that require multicast are usually setup and configured by gurus/vendors who ensure that all the i's are dotted and t's are crossed.

But BEA Weblogic ships with and uses multicast implementation as part of its clustering technology. As it is used in a limited scenario (single network, no switch cross-over, etc), multicast administration is suddenly simplified to a simple set of properties: primarily multicast address/port (others are Send Delay, TTL and buffer size). Look at the configuration page, all values are preset to defaults. In fact, you don't even need to see that page when you create your cluster; you have to manually choose it if anything needs to be changed. One way or another, the values are usually left at the defaults even for very large deployments.

Now, Weblogic's default multicast address is 237.0.0.1. If you cringed at this point, do not continue reading. You know way too much about TCP, MAC address and multicast special cases. You are a guru.

For the rest of us, following is an explanation why this default is a bad one.

If you read multicast RFCs - which are many in number (3170, 2236 , 1112, 2365, etc) you will find that not all multicast addresses are equal. There is a magic address 224.0.0.1 on which all IP hosts will listen and this includes gateways (which covers CISCO routers and switches).

Now, 224.0.0.1 is NOT 237.0.0.1, so there should be no issue. And on TCP layer, there is none. The problem happens bellow TCP layer. OSI Layer 2 switches (of which CISCO switch is an example), do not actually listen to TCP directly. Instead they listen for the multicast MAC address. There is a direct mapping between multicast IP address and multicast MAC address, but it is not unique. In fact, there can be 32 different IP addresses corresponding to the same MAC address (as described in this book excerpt, section 1.6.3).

So, to follow the specification, CISCO switch has to listen for the multicast address 224.0.0.1, which is MAC address 01-00-5e-00-00-01. But that is the MAC address for 237.0.0.1 as well. Therefore, every time a packet passes by for 237.0.0.1, CISCO switch firmware triggers a match.

Once the match is triggered, software part of the switch examines packet's full IP address and figures out that it was not a 224.0.0.1, but just one of the other 31 co-sharing IPs. The packet is discarded and nobody is hurt.

Or so it seems. Turns out that these interrupts are relatively expensive. When a packet is supposed to be sent to a different network, this cost is part of a business. But when it happens every time for a fairly frequent internal cross-server chit-chat (e.g. JNDI tree updates), the switch gets interrupted too often. And a switch that is interrupted unexpectedly often, may be sluggish to respond to legitimate requests.

This, under high enough load, can cause unexpected network timeouts that are nearly impossible to detect. After all, how often do you check your switch's CPU utilisation, when your DNS misbehaves?

So, what's the solution. Easy! Switch to a different multicast address that does not map to the same MAC as the 224.0.0.1. Weblogic will do exactly that for the next release.

The proposed new address is 239.192.0.0. If this makes you cringe, please let BEA know NOW! :-)

BlogicBlogger Over and Out

I want my Scopo

2004-10-16T00:37:00.000-04:00

Scopo - a wearable display from Mitsubishi - is not going to be here until at least next year. I am really looking forward to it being available for public. It would go long way towards making wearable computing a reality.

They do not say what resolution it is, but the projection is equivalent to 10-inch screen. I wonder if it is good enough to actually write programs. Surely, I could fit a VI session into it, even if it is at 24x80 screen size.

I can barely wait.
BlogicBlogger Over and Out

Vegetable or Mineral: support's point of view

2004-10-15T18:30:00.000-04:00

The article on Hacknot talks about testers and the grief developers face from them due to incorrect problem reporting practices. Of course developers do exactly the same things when they are the ones reporting problems.

I am talking about Technical Support requests. When a developer (or an operations personnel, or - worse yet - an operations manager) calls with a support case, they do exactly the same things that Hacknot described for the testers.

As a very common example, the support case description will read Weblogic crashed without reason. After extensive probing, following things emerge:

The Weblogic process hasn't actualy exited, but is still around. It just stopped responding. This means Weblogic hanged, rather than crashed. Equally annoying, yet caused by completely different set of problems.
No reason turns out to be a new code drop that, in their opinion, changed nothing important and therefore was not worth mentioning.
Doing thread dumps and log/config analysis often shows a bottle neck and/or incorrect configuration. At this point, it usually emerges that the server load at the hang time was higher than ever before and no, it was not load tested in the same configuration (e.g. cluster with proxy) anywhere else.

And of course, if it is not an overly general description, it is 'Diagnosing Instead of Reporting' exactly as described in the article.

I have to admit that when I was a developer, I commited all those mistakes and more myself. I feel that work in BEA support has opened my eyes and makes the code I produce now more user friendly with better reporting.

As a result, my thought on the QA/Dev separation is to actually fold developers into the testing team for short rotational periods.

The developer will then discover whether there are real causes to reporting problems (insufficient logs, misleading dialogs or just unclear logic) that will only be acknowledged as problems when a developer walks a mile in a tester's shoes.

The final benefit is that developers and testers have different ways to do similar things and exchange of techniques is always beneficial.

BlogicBlogger Over and Out

Thinking of WebSphere 6 supportability features

2004-10-12T09:12:00.000-04:00

The news of IBM releasing the next version of WebSphere real soon now are propagating through the websites. As a Weblogic support person, it was interesting for me to see what new supportability features were included by IBM. Unfortunately, the details are very vague.

From the general description (failover, failure detection and recovery, etc), it sounded like the features we already had in Weblogic for a while. Some we even had for a very long while (e.g. WLS 6.1 proxy plugin that supports smart failover).

However, I did find one article that provided more serious details.

Specifically,

ClassLoaderViewer
Hang Detection code in 5.03, 5.1.1 and 6.0
LeakBot for memory leak detection
Session Data Crossover detecton facility in 5.03, 5.1.1 and 6.0
Connection Leak detection
ThreadAnalyzer for deadlock debugging
Event Alerter for notification

Some of the items above Weblogic already has. JDBC leak detection was there for a long time; deadlock detector of sorts is built into latest JRockit and there are several free standing tools; event alerter is possible via JMX, SNMP or custom loggers, etc.
Others, we probably should look into.

I will not open any secrets by saying that BEA has recently started to pay serious attention to supportability of its products with Dev2Dev utils, support patterns and new support seminars. Even more interesting announcements in this area should come out soon.

However, seeing IBM to put some real effort into supportability area may spur yet more effort from BEA side and will not do any harm to either companies The real benefits will of course be reaped by system administrators and operations staff.

Long live the healthy competition. As long as we win. :-)

BlogicBlogger Over and Out

Podcasting loop: evolution continues

2004-10-09T23:26:00.000-04:00

Sometimes the only good outcome of a post is to provoke somebody else to do one better. Seemed that I had caused exactly that.

About a week ago, I wrote about extending podcasting loop using Windows Scripting Host and IPod's COM interface. In a comment, Scyro had written about disliking my approach and explained how to achieve the same effect using the smart playlists instead.

I have read his article and actually like his approach more than my own. It is more automatic, does not require to run any scripts and is cross-platform. I still prefer explicit rating system as a feedback rather than playcount, because I switch away from some tracks before the end or come back after the finish. It is easy to modify the solution either way.

Of course without my post, he may not have been motivated to write his. Between us, I believe, we explain how to make iPodding easier.

So, is my script still useful? Probably not for its original purpose. But if you want to delete the old tracks or do any other complex processing, you may still find it helpful as a base.

BlogicBlogger Over and Out

Who is blogging about Weblogic

2004-10-04T08:06:00.000-04:00

There are some people out there blogging about Weblogic. Here is the short list in hopes this article will make them a bit more noticeable:

Cid Danis. Weblogic Senior support engineer
Unknown, but thorough in documenting test problems
Vinny Carpenter, who is using Weblogic and blogs good links and articles
News from BEA Dev2Dev. From the horse's mouth so to speak
Moazam Rajas. He is actually SUN Senior support engineer, but if you run Weblogic - or any other J2EE servers - you will do well to read his articles

All of the blogs above have RSS feeds.

There is also a couple of interesting individual links:

Unofficial WebLogic FAQ
Cached copies of Weblogic Developer's Journal

If you use Weblogic, check these out.
BlogicBlogger Over and Out

The power of GraphViz

2004-10-03T07:40:00.000-04:00

Ever felt the need to extract some relations from the configuration or data and present it in a visual form nicely layed out. Ever given that up as too hard due to the hard problem of laying out the elements? If you did, then check out GraphViz.

While GraphViz by itself is not Java, it is cross-platform and is fairly easy to setup and invoke from Java. It is also open-source, if that is important to you.

So, what does GraphViz do? Well, it does fancy graph layouts. Sounds boring until you realize that graphs - or data that could become graphs - are actually all over the Java world.

Let me give you a few examples:

Ant task dependancies are graphs. If your ant file is rather large, check out ant2dot, Vizant or AntGraph.
Spring dependencies are graphs. If your Spring project is getting complex, get an overview of it with SpringViz.
If your code is already deployed, but you experience hangs or slowdowns, Graphviz will help to visualize the thread dumps.
If you are not happy with GIF/SVG/PNG/PS output that Graphviz is capable of generating, you could always try a more interactive approach with ZGRViewer (a bit raw, but very promising project)
Many other uses exist, some of which are described at the GraphViz resource page
Finally, if you are able to generate text file that says a->c; b->c; c->d, you can generate GraphViz input (dot file) and let the program itself do the magic of layout. And once you master a simple dot file, you can start adding such bells and whistles as colors, clusters and structures.

If you have an itch to scratch, let GraphViz help you with it.

BlogicBlogger Over and Out

Extending the podcasting loop

2004-10-02T03:24:00.000-04:00

So, you already use iPodder to listen to your ITConversations on the iPod with a 1-click ease.

All is well, except that it is starting to get difficult to remember which shows you already listened to and which ones are still new. You can of course go to iTunes, find the already-listened-to track and manually delete it or move it to the 'archived' section. But that is very much against 1-clickspirit.

It would be very nice to be able to mark from inside the iPod which tracks you are no longer interested in. Except that iPod is a read-only device.

Except that it is not! You can rate a track with 0-5 stars and that information will get synchronized to the iTunes on the next connection.

Introducing ArchiveTrack. Inspired by one of the iPodder impementations, it uses Windows Scripting Host to archive the tracks (in a given genre) using rating value as a trigger.

The script is given here in all entirety. Feel free to modify it or incorporate it into other setups as long as this article is attributed.


/*

    Run this with  'cscript ArchiveTracks.js'



    The script goes through the songs in sourceGenre and

        moves/archives the ones that have ratings under the threshold

        into targetGenre.



    Usage scenario:

        1) Download podcasting content using iPodder (1 - click)

        2) Listen to it on your iPod

        3) While you are listening, rate the track 1/2 stars (default script threshold)

        4) After the next synchronization, run this script with iPod still connected



    Version: 1.0b1 (Oct 2, 2004)

    Author: Blogic Blogger

    License: Use in any way desired, but mention the source and author

*/



try

{

    var sourceGenre = 'Speech'; // The genre used by ITConversations

    var targetGenre = 'Speech-Archive'; //The genre to collect all the listened content in

    var archiveThreshold = 50; // 1 or 2 stars

    var progressNoticeRepeatCount = 250;



    WScript.Echo('Connecting to iTunes');

    var    iTunesApp = WScript.CreateObject("iTunes.Application");

    var    mainLibrary = iTunesApp.LibraryPlaylist;



    var tracks = mainLibrary.Tracks;

    var songCount = tracks.count;

    WScript.Echo('Total Song Count: ' + songCount);



    for (i=1; i<=songCount; i++)

    {

        if ((i%progressNoticeRepeatCount) == 0)

        {

            WScript.Echo('Checked ' + i + ' items');

        };



        var aTrack = tracks.Item(i);

        if (aTrack.Genre==sourceGenre)

        {

            if ((aTrack.Rating > 0) && (aTrack.Rating < archiveThreshold))

            {

                WScript.Echo();

                WScript.Echo('    Archiving: ' + aTrack.Name + ' by ' + aTrack.Artist + ' based on rating ' + aTrack.Rating);

                WScript.Echo();

                aTrack.Genre = targetGenre; //use 'aTrack.Delete();' to delete instead of archiving

            }

            else

            {

                WScript.Echo('    Keeping: ' + aTrack.Name + ' by ' + aTrack.Artist + ' based on rating ' + aTrack.Rating);

            }

        }

    };

}

catch (e)

{

    WScript.Echo('Problem archiving: ' + e);

};



try

{

    WScript.Echo('Requesting iPod update');

    iTunesApp.updateIPod();

    WScript.Echo('Successful iPod update request');

}

catch (e)

{

    WScript.Echo('Problem updating iPod ' + e);

}

WScript.Echo('Done!');

BlogicBlogger Over and Out

Connecting the Microsoft Dots

2004-10-02T01:25:00.000-04:00

When Win98 uptime was revealed to be never more than 49.7 days, Slashdot laughed. They could not imagine anybody actually wanting - or managing - to keep their Windows machine up for that long. I wonder if they would laugh now.

I wonder what other funny Microsoft problems are going to bite us well after their discovery.

BlogicBlogger Over and Out

Troubleshooting ClassNotFoundException with Filemon (on Windows)

2004-09-30T07:49:00.000-04:00

Have you ever run into classpath issues where you could have sworn the class was in a jar, the jar was on a classpath, yet you still got NoClassDefFoundError or ClassNotFoundException. Or perhaps you define several Classloaders in your application each with its own classpath and are not quite sure which one gets the priority in a specific situation.

On Windows, FileMon from SysInternals is a great free tool to resolve exactly those kind of questions.

Imagine you have a very simple java class:



public class Test

{

   public static void main(String[] args)  throws Exception

   {

              Class c = Class.forName("somepackage.SomeClass");

   }

}

Basically, we are trying to load somepackage.SomeClass into the system. For our test purposes that class does not exist.

Compile this class and run. It comes back with java.lang.ClassNotFoundException: somepackage.SomeClass. As it should.

The question however is what Classpath the JVM was looking for the class on. If your application is a J2EE deployment or anything with custom classloaders, this question may become a hard one to answer.

This is where the FileMon shines. FileMon will show you every file operation on the machine performed by any - or all - of the processes.

Start it up and set the filter to java.exe . If that causes problems you can reset filter to * and check what name your JVM is generating the log entries under.

With FileMon capture enabled, rerun your test or scenario. After you see the ClassNotFoundException, stop the capture. Now search from the capture start for the classname you got in the exception, in this case SomeClass. You should see a sequence of the lines similar to the following (some columns had been cut):

C:\temp\filemontest	SUCCESS
C:\somepackage\SomeClass.class	PATH NOT FOUND
C:\temp\filemontest	SUCCESS
C:\Utils\somepackage\SomeClass.class	PATH NOT FOUND
C:\temp\filemontest	SUCCESS
C:\temp\somepackage\SomeClass.class	PATH NOT FOUND
C:\temp\filemontest	SUCCESS
C:\temp\filemontest\somepackage\SomeClass.class	PATH NOT FOUND

From here, we can figure out that the classpath checked was: C:\;C:\Utils;C:\temp;C:\temp\filemontest.
Now, you can check that against your expectations and figure out how to fix the discrepancy.

If you are on Solaris/Linux machine instead, look into truss/strace for a way to achieve very similar results.

BlogicBlogger Over and Out

XMLStarlet: A gentle introduction into XSLT

2004-09-29T17:21:00.000-04:00

In my work as a BEA Tech Support engineer, I have to spend a lot of time dealing with XML content. In the past I tried to grep through the files, but had found that element order and random newlines make it quite hard. I have since found a solution.

Everybody who works with XML probably knows that XSLT/XPATH is supposed to be a good way to manipulate XML files. Unfortunately, XSLT is very verbose and sometimes quite obscure, especially when it comes to plain text output. I have tried using it a couple of times and never got beyond a simple HelloWorld.

Then I discovered XMLStarlet. It is built on libxml2 and libxslt and is provided for several platforms as a all-in-one executable.

XMLStarlet has a number of options; among them are transformation, edit, canonicalization and many others. The one I find most useful is select. With select, XMLStarlet works something like an in-file unix find. You provide command line switches which describe what you are searching for and what you want to display.

And the best part of all is that the proprietary syntax translates into 100% correct XSLT, so when you have outgrown the tool or need more specific tweaking, you take the generated XSLT and start editing that directly.

Let me give you an example. I often have to work with Weblogic's config.xml file which defines the configuration for a whole domain. It describes servers, clusters, JMSQueues, SecurityRealms and many more aspects. And, as config.xml was designed for the software's convinience rather than support engineer's, it groups elements together in a less than intuitive fashion.

Following, is a very small part of config.xml structure (derived from a sample config.xml file using XMLStartlet's xml el -a config.xml command.


Domain

Domain/@Name

....

Domain/Server

Domain/Server/@ListenAddress

Domain/Server/@ListenPort

Domain/Server/@Name

Domain/Server/@Cluster

...

Domain/Server/WebServer

...

A task I need to do often is to check how many servers a client's config.xml has, what their names are and how are they distributed between clusters. Manually, it means searching for <Server tag and then eyeballing for Name attribute.

With XMLStarlet it becomes:
xml sel -t -m //Server -v @Name -n config.xml
with the output of



nhgprdadmin

nhgapp03a

nhgapp03b

nhgapp01a

nhgapp04a

nhgapp01b

nhgapp04b

nhgapp02a

nhgapp02b

In here, -m is match and -v is display. -n is a newline, not something that is obvious in XSLT itself. The parameters are XPATH values.

Do this 5 times and it pays back the time investment involved in learning the command syntax.

Now, let's try to do something more complex. Specifically, I want to check which cluster each server belongs to. And to cut this post short(er), let's say I want it sorted by the cluster.

The command would be:
xml sel -t -m //Server -s ATL @Cluster -v @Cluster -o " - " -v @Name -n config.xml
with the output of



- nhgprdadmin

cehs - nhgapp03a

cehs - nhgapp03b

cehs - nhgapp04a

cehs - nhgapp04b

his1 - nhgapp01a

his1 - nhgapp02a

his2 - nhgapp01b

his2 - nhgapp02b

We can immediately see that there is one non-clustered (admin) server, 4 servers in one cluster and 2 servers each in the other two clusters. In the command above, -s is sort by Cluster attribute and -o is verbose output used for spacing. Instead of -v -o -v, I could use -v "concat(XXX,yyy,ZZZ)", but that required a greater knowledge of XSLT than I had the first time I used the tool.

And just to show you the XSLT that the above command really corresponds to, it is (after removing long header and footer):



<xsl:template match="/">

 <xsl:call-template name="t1"/>

</xsl:template>

<xsl:template name="t1">

 <xsl:for-each select="//Server">

   <xsl:sort order="ascending" data-type="text" case-order="upper-first" select="@Cluster"/>

   <xsl:value-of select="@Cluster"/>

   <xsl:value-of select="' - '"/>

   <xsl:value-of select="@Name"/>

   <xsl:value-of select="'&#10;'"/>

 </xsl:for-each>

</xsl:template>

As you probably can imagine, it takes quite a bit longer for an XSLT beginner to write the code above from scratch.

I hope this was useful and will convince you to have a look at the XMLStarlet.

BloggicBlogger Over and Out

A fitting first post: I love Vim

2004-09-28T02:18:00.000-04:00

One's first post should be interesting. And what could make a techie's post more interesting then the Vi vs. Emacs battle. So in my very first post allow me to declare my allegiance to Vi. Or to be more specific Vim.

But it would not be interesting to just announce my preferences and be done with that. Let me instead explain a little what I like in Vim and what I have done to it in making it useful for me.

Following are the basic plugins I have added to my setup:

OTF: On-The-Fly coloring of patterns: This plugin is indispensable if you are trying to track several items happening in a file at once. For example, if you are trying to read a piece of java code and need to keep track of all log messages, just highlight the specific calls with one of OTF's colours and you are done. You can of course do it using search pattern, but then you java lost the search ability. And OTF allows you to do multiple colours, so you can keep track of multiple patterns at the same time.
F: An unfortunately named Find plugin: You may not use it too often, but if you have to look at something that has a line of useful data, couple of lines of junk, another line of useful data, this tool allows you to hide away the junk by folding it away. In some instances, it is better than doing :v/pattern/d because you may actually want to examine the particular context in details. With F, you just open the fold above/bellow.
JavaDecompiler: A Vim frontend for JAD: If you decompile class files for research purposes, you might be annoyed with having to run class file through decompiler, save the result into a second file, open that file in the favourite java editor and then - when you are finished - still having to delete the decompiled file. If Vim is your favourite editor, this instead becomes a true one-click exercise. Find the file, open it and voila. No manual conversion, no deletion. This plugin will do it all for you. Just make sure your .class files are associated with Vim.

And this is my basic setup. I have additional configuration settings and may blog about them another time if there is any interest.

And just as a disclamer, you can do most of those things with Emacs and I have done so in the past. But Vim is the one I kept coming back to and don't think I will change away from it any time soon.

BlogicBlogger Over and Out