Archive

Author Archive

Managing and Building version-controlled Maven Repos using Git, Gradle and Nexus Server

I currently work in a VERY OLD code base that uses a thirdparty directory as a version-controlled “library” directory, dated from the era before Maven. Some colleagues decided it was time to adopt Maven for building new components and that was exciting… However, you can imagine having a build system composed of “dino” ANT scripts to manage really old stuff, and then the introduction of Maven pom.xml.

I personally chose to use Gradle for the projects that I had started and integrated it well to publish generated jars to the “thirdparty” directory…

http://stackoverflow.com/questions/7826652/how-to-upload-an-existing-collection-of-3rd-party-jars-to-a-maven-server-in-grad

Gradle for Maven Dependencies

The most updated version of this script is shown below. Note that this approach differs from the one in the stackoverflow in that this version DOES NOT generate the Maven Metadata files (pom.xml, *sha…), but simply copies the generated versioned jars into the “thirdparty” directory (scm cached dir).

Then, I used Gradle’s dependencies mechanism to use the same thirdparty directory as a dependency…

http://gradle.1045684.n5.nabble.com/Javac-Java-Annotation-Processor-Maven-classpath-dependencies-in-Gradle-tp4626751p4633029.html

This week I received a surprising email from a developer complaining about not being able to build one of the components. The problem was simply because the Maven repository server was COMPLETELY REBUILT and all the dependencies were wiped-out. As a result, developers maintaining different projects needed to upload the projects again. That triggered something the question where the Maven artifacts should be stored. As the old approach of using the “thirdparty” directory might become a hybrid approach in a future, I thought I could use GIT to store the versions of the maven versions on a version-controlled directory of my projects after I stumbled upon the following blog

http://cemerick.com/2010/08/24/hosting-maven-repos-on-github/

I agree with the pros/cons about that approach, and that happens to be a very similar situation I faced today. In this way, I decided to create something similar using BitBucket private’s GIT project to simulate my same environment.

Reusable Gradle Properties

First the initial maven configuration is managed by the following:

This gives me the following capabilities:

  • Using the property “-Prelease” as a switch to not use the -SNAPSHOT prefix in the generated jars.
  • Lots of properties saved in the project, making it easier to create scripts that depending on those properties from other build scripts.
  • Could also provide an option to generate jars to the “thidparty” directory WITHOUT the Maven’s Metadata files.

For instance, given a project “Maceio Finance” on a project repository on BitBucket, I have created the following build.gradle file to build, generate the jars, declare dependencies to the local scm repository, and able to upload new versions to that.

The simplicity of Gradle uploadArchives

Running the task “uploadArtifacts” with the different switches result in the same expected behavior as described by the blog on http://cemerick.com/2010/08/24/hosting-maven-repos-on-github/, as shown in the output below.

The generated directories with the Maven Metadata files is shown below:

Now, if you just want the same Jars to a given version-controlled directory WITHOUT the Metadata file, you can use a task to “installThirdparty” described above.

Finally, the other support needed is a Maven server. We use the open-source Nexus server. Here’s the example of uploading the same contents from the configuration. Note that it uses the previous definitions of the properties from mavenProperties.gradle.

Gotta love gradle properties

Just as a reference, the output of the command “gradle properties” is a good reference to see ALL the projects variables at runtime. Here’s the output of that command. Take a look at the property “projectMvn” just as a reference.

Cheers!

Running EMMA Code / Test Coverage with JAVA 7 and Gradle 1.0M9

Had a huge road block while trying to integrate EMMA Code Coverage with the latest version of Gradle today… I found this blog post that helped starting reaching a solution…

http://www.breskeby.com/2010/04/add-emma-code-coverage-reporting-to-your-gradle-build/

It has been 2 years since the last update (although some users have commented other solutions), but as Gradle DSL and Java are still evolving, that version did not work. One common problem related to Java 7’s new bytecode instruction. EMMA, Cobertura and others are facing the same problems discussed at http://stackoverflow.com/questions/7010665/testng-emma-cobertura-coverage-and-jdk-7-result-in-classformaterror-and-verif. Just adding the jvmArg “-XX:-UseSplitVerifier” solved the problem for me with EMMA.

Well, after spending a few hours trying to get the previous patch working, I got a working solution running Test Coverage with EMMA on the latest Grails with Java 7.

First, add the emma configuration and its dependency.

configurations{
  emma
}

dependencies {
  // EMMS Code Coverage
  emma "emma:emma:2.1.5320"
  emma "emma:emma_ant:2.1.5320"
  ...
  testCompile group: 'junit', name: 'junit', version: '4.9'
}

Then, update the test task by adding the doFirst{} and doLast{} closures below.

test {
    // add EMMA related JVM args to our tests
    jvmArgs "-XX:-UseSplitVerifier", "-Demma.coverage.out.file=$buildDir/tmp/emma/metadata.emma", "-Demma.coverage.out.merge=true"

    doFirst {
       println "Instrumenting the classes at " + sourceSets.main.output.classesDir.absolutePath
       // define the custom EMMA ant tasks
       ant.taskdef( resource:"emma_ant.properties", classpath: configurations.emma.asPath)

       ant.path(id:"run.classpath") {
          pathelement(location:sourceSets.main.output.classesDir.absolutePath)
       }
       def emmaInstDir = new File(sourceSets.main.output.classesDir.parentFile.parentFile, "tmp/emma/instr")
       emmaInstDir.mkdirs()
       println "Creating $emmaInstDir to instrument from " +       sourceSets.main.output.classesDir.absolutePath
       // instruct our compiled classes and store them at $buildDir/tmp/emma/instr
       ant.emma(enabled: 'true', verbosity:'info'){
          instr(merge:"true", destdir: emmaInstDir.absolutePath, instrpathref:"run.classpath",
                metadatafile: new File(emmaInstDir, '/metadata.emma').absolutePath) {
             instrpath {
             fileset(dir:sourceSets.main.output.classesDir.absolutePath, includes:"**/*.class")
             }
          }
       }
       setClasspath(files("$buildDir/tmp/emma/instr") + configurations.emma +    getClasspath())
    }

    // The report should be generated directly after the tests are done.
    // We create three types (txt, html, xml) of reports here. Running your build script now should
    // result in output like that:
    doLast {
       def srcDir = sourceSets.main.java.srcDirs.toArray()[0]
       println "Creating test coverage reports for classes " + srcDir
       def emmaInstDir = new File(sourceSets.main.output.classesDir.parentFile.parentFile, "tmp/emma")
       ant.emma(enabled:"true"){
          new File("$buildDir/reports/emma").mkdirs()
          report(sourcepath: srcDir){
             fileset(dir: emmaInstDir.absolutePath){
                include(name:"**/*.emma")
             }
             txt(outfile:"$buildDir/reports/emma/coverage.txt")
             html(outfile:"$buildDir/reports/emma/coverage.html")
             xml(outfile:"$buildDir/reports/emma/coverage.xml")
          }
       }
       println "Test coverage reports available at $buildDir/reports/emma."
       println "txt: $buildDir/reports/emma/coverage.txt"
       println "Test $buildDir/reports/emma/coverage.html"
       println "Test $buildDir/reports/emma/coverage.xml"
    }
}

You can run the updates from your gradle as follows:

marcello@hawaii:/u1/development/workspaces/open-source/interviews/vmware$ gradle test
:compileJava
:processResources UP-TO-DATE
:classes
:compileTestJava
:processTestResources UP-TO-DATE
:testClasses
:test
Instrumenting the classes at /u1/development/workspaces/open-source/interviews/vmware/build/classes/main
Creating /u1/development/workspaces/open-source/interviews/vmware/build/tmp/emma/instr to instrument from /u1/development/workspaces/open-source/interviews/vmware/build/classes/main
Creating test coverage reports for classes /u1/development/workspaces/open-source/interviews/vmware/src/main/java
Test coverage reports available at /u1/development/workspaces/open-source/interviews/vmware/build/reports/emma.
txt: /u1/development/workspaces/open-source/interviews/vmware/build/reports/emma/coverage.txt
Test /u1/development/workspaces/open-source/interviews/vmware/build/reports/emma/coverage.html
Test /u1/development/workspaces/open-source/interviews/vmware/build/reports/emma/coverage.xml

BUILD SUCCESSFUL

Building a Developer’s Social Network…

I’ve been interested in blogging for a long time as I’ve created this blog to publish my personal stuff… I had focused 2 years of my undergraduate Computer Science studies (2000-2002) developing Graw as a research tool to support collaborative learning environment for scholars. This was triggered when I started thinking about how could I take advantage of twitter and/or FaceBook to reach out other developers. In addition, I was personally looking for ways to organize my Computer Science life from school and open-source contributions I had developed in the past and others I like to work, etc.

Today I see a developer’s Social Network as his/her image and presence in a given community of interest. For instance, BlackBeltFactory, the old JavaBlackBelt, is a community for Java developers looking for certification-level training in Java skills, among others. Similarly, the developers at GitHub and Ohloh are focused in bringing developers together from a social coding perspective, or open-source contributions, etc. Finally, I see LinkedIn as the professional visibility of developer/engineer/researcher/scientist as they provide groups of interest, etc. As I started reading more and more about this subject, I started seeing this as a way to help a developer use the power of the Internet to grow his/her social network among the professionals of a given community.

Tools

Here are a list of tools I found to help developers build their social network using their own website. WordPress is great for starting your own blog. You can download an open-source version of WordPress and maintain it yourself. Other tools that can be used to help a developer spread his/her news are:

  • Blogging: Blogging is by far the simplest way to start. Start by writing solutions to common problems, or a solution to a problem you haven’t found online. When you have more time, do a “Code Garage Sale” and have code snippets from previous projects developed in the past, say in grad school or another open-source project. That can be used as a future reference to yourself and others who face or will face the same problem. Blooging helps developers to write long emails to answer regular questions from other users, in addition to helping him/her to develop his/her writing skills. Consider not only writing blog posts, but also commenting on the ones you are interested about. You can later learn about your audience by using Google Statistics and Feedburn for statistical analysis of your audience.
  • Social Bookmarks: delicious.com give developers an account to add bookmarks based on tags. While working in a project, I tend to add links that solves a given problem or a reference to something I have done. In my opinion, other users can take advantage of what you have tagged and see the links you were working to solve a given problem. I think this can be used to find other developers based on the links you share in a social bookmark, while other developers or peers in a project can have a view of the references you’ve used. Just think about integrating a new team member to a project and share all the research references. For scientific papers, I use MyACM for research papers at on ACM and I constantly bookmark references for a given paper I’m writing.
  • Open-source Identities: for those who contribute with Open-Source projects, you probably have heard about Ohloh.net. It maintains a cache of popular open-source repositories and help users identify themselves based on the commit messages. This is a great tool to find who are the developers behind open-source projects based on the commits to the repository.  GitHub and GoogleCode provide a feature that allows you to create a link to a specific code line, allowing developers to give more specific and direct examples. Why not subscribing to someone’s commits?
  • Question-Answer Communities: what do developers do when they need to fix a given problem? What if you need an answer to a given problem that is specific to the domain of knowledge of the problem? Yes we can google for specific key words, but it is more effective to make a presence at StackOverflow. A blog can seen as a developer’s point of view about a solution to a problem, taking control over the comments, etc. On the other hand, others may prefer Wiki pages, whose content is managed by a community of users. StackOverflow’s model revolves around the idea of community supported answers sorted by correctness and votes, where the “most optimum answer” is displayed on top. It uses the Digg’s approach to “Up” and “Down” a given answer. This type of community can  help building one’s reputation in the community since it is a human desire. Have you heard of “Jon Skeet“? 

  • Mobile Support: put a developer on a train ride and a mobile device will be his/her best friend. I’ve been using my Android phone more often to scan QR image for a link to a website, a Google Map, Contact Information, etc on a webpage. I usually use the Google Chart API to generate my QR images for a given purpose. Use you barcode reader and you will automatically open this post’s webpage for future reading.
  • Broadcast Yourself: Use twitter to allow others to see what you are working at or connect with a given community. This is a very useful way to reach out to a great number of developers at the same time. As we see today, blogs allow us to tweet blog posts and that’s just the beginning. 

  • Syndicate your blog: Allow RSS Feeds so that people can subscribe to a given developer’s blog.
  • How can social networking sites make you a better developer?

Some more information on social networking for developers in the links below.

http://channel9.msdn.com/blogs/glucose/hanselminutes-on-9-social-networking-for-developers-part-2-make-your-blog-suck-less
http://channel9.msdn.com/Blogs/Glucose/Hanselminutes-on-9-Social-Networking-for-Developers-Part-1-Every-Developer-Needs-a-Blog

Categories: Open-source
%d bloggers like this: