The nightmares of getting images from the Mac OS X clipboard using Java

Posted by Dan on Feb 8, 2008 @ 4:55 PM

One thing I rarely ever do in Java is program anything that requires a desktop UI. If I'm doing any Java programming, it's generally server-side related. Recently that changed when I began working on a Java applet that allows users to upload images (screenshots) in their clipboards directly to the server.

Building the basic applet was pretty straightforward. Even signing the applet (so you can actually query the operating systems clipboard) is pretty straightforward—made even easier once I created an ANT build script.

The applet was progressing nicely, well that's until I tested things on the Mac.

The idea of write-once, run-many is a great idea but that's the one big hype Sun pushed in the early Java days that never came to be. Oh, don't get me wrong; there are some really great cross-platform Java applications, but if you're doing anything beyond the basics you start having to branch code or do some OS testing to get things to work properly.

When you grab data from the clipboard in Java, the object is a type of DataFlavor. Essentially this is a way to determine what type of data is in the clipboard—image, plain text, rich text, etc. To test to see if something is an image, you should be able to test to see if it's a DataFlavor.imageFlavor.

This works great on the PC, but the Mac OS X rolls up it's own DataFlavor. The PC DataFlavor looks like this:

java.awt.datatransfer.DataFlavor[mimetype=image/x-java-image;representationclass=java.awt.Image]

While the Mac image DataFlavor is:

java.awt.datatransfer.DataFlavor[mimetype=image/x-pict;representationclass=java.io.InputStream]

No big deal I figured, there's got to be an easy way to convert this to something I can actually use in Java. However, after much research I could only find one decent method which involves using QuickTime for Java.

The problem is QT for Java is sort of a mess. From doing some reading, it may or may not be installed correctly on the Mac. When installing QuickTime on the Mac OS it's supposed to install the QT for Java library, but on the Mac OS X 10.3 machine I had handy, I actually had to reinstall QuickTime 7 to get things to work.

There's another problem with using the QT for Java libraries. In order to get things to compile and execute correctly on machines without QT for Java, you need to use reflection to create instances and run methods on the QT classes. That leaves you with some pretty pretty ugly code. Fortunately, someone much smarter than me already wrote the code up in a getImageFromPictStream() method.

I was finally making some progress on the applet. I could now grab the contents of the Mac OS clipboard and I could display in my PreviewPane (which extends the Canvas object.) I figured now that I could display it, converting the image to PNG or JPG ByteArrayOutputStream would be pretty straightforward.

Once again, I was wrong.

Turns out the QT for Java classes generate an object of apple.awt.OSXImage. It also turns out this object appears to be completely undocumented. There's almost no information on that class that I could find anywhere. The only thing I could find were other people complaining about the same problem I was having.

For the past couple of days, I've been mucking around trying to find a way to convert this apple.awt.OSXImage to some derivative of the java.awt.Image class. Because I'm not used to programming GUI elements in Java, I think it's the reason I kept overlooking the obvious solution of drawing it on a Graphics2D object. I kept trying to find ways to convert that apple.awt.OSXImage to a BufferedImage using introspection and reflection. I was really making it way more complicated than it really is.

Finally the light bulb went off this afternoon when I thought "Hey, since I can display on the screen, can't I just convert that into a usable image?"

After banging my head trying to do something with this applet.awt.OSXImage object off and on for a couple of days, the solution was quite simple:

public static BufferedImage getBufferedImage(Image img){
    if( img == null ) return null;
    int w = img.getWidth(null);
    int h = img.getHeight(null);
    // draw original image to thumbnail image object and
    // scale it to the new size on-the-fly
    BufferedImage bufimg = new BufferedImage(w, h, BufferedImage.TYPE_INT_RGB);
    Graphics2D g2 = bufimg.createGraphics();
    g2.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BILINEAR);
    g2.drawImage(img, 0, 0, w, h, null);
    g2.dispose();
    return bufimg;
}

This very basic method will take any generic Image object and return a BufferedImage object. It's amazing how simple the solution turned out to be. I hate when I make things way more complicated than they need to be!

Categories: Java

10 Comments

  • Alan Mesut's Gravatar
    Alan Mesut
    "... Mac OS X rolls up it's own DataFlavor. .."

    ORLY????
  • Dan G. Switzer, II's Gravatar
    Dan G. Switzer, II
    @Alan:

    Well it's not overly surprising that they have their own DataFlavor for images, it *would* be nice if Apple would document all these custom objects. I could almost nothing on Apple's sun regarding their Java classes.
  • George Ulyanov's Gravatar
    George Ulyanov
    Hello,

    This last method 'getBufferedImage' works bad with the great-sized images. I tried to convert some JPEGs with the sizes about 2000x2000 with this method and it failed :(
  • Dan G. Switzer, II's Gravatar
    Dan G. Switzer, II
    @George:

    As with anything in Java, you need to be aware of memory management. When you're manipulating large images, it's really easy to fill up memory alloted to Java--depending on your settings. Certainly one of the issues I had when dealing with images on the Mac is I needed to do a lot of manipulation to the images, which in turned caused me at times to have multiple objects that were copies of the image in memory.

    Certainly, if you figure out a less memory intensive method for dealing with the images, make sure to share the solution.
  • Terry Riegel's Gravatar
    Terry Riegel
    Hello,

    I am struggling with the same thing. I am not a java programmer but was able to find and get working a java applet that will take the clipboard and "paste" it to a server. I am having trouble with certain types of clipboard data generated on the mac. I am pretty sure they are png images. But everytime I attempt to use they java states there is no image on the clipboard.

    import java.awt.datatransfer.Clipboard;
    import java.awt.datatransfer.DataFlavor;
    import java.awt.Toolkit;
    import java.awt.Color;
    import java.awt.image.BufferedImage;
    import java.awt.image.ImageObserver;
    import javax.swing.JApplet;
    import javax.swing.JOptionPane;
    import javax.swing.ImageIcon;
    import java.io.ByteArrayOutputStream;
    import com.sun.image.codec.jpeg.JPEGImageEncoder;
    import com.sun.image.codec.jpeg.JPEGCodec;
    import java.net.URL;
    import java.net.URLConnection;
    import java.io.InputStream;
    import javax.swing.JLabel;
    import java.net.URLEncoder;

    public class PasteImageApplet extends JApplet{
      Clipboard clipboard;
      Toolkit toolkit;
        JLabel lbl;
      public String getClipboardImageURL(String server,String ipath,String ci_imagepath){
            lbl.setText("pasting image");
        String url = "";
        try{
          DataFlavor dataFlavor = DataFlavor.imageFlavor;
          System.out.println(dataFlavor.getDefaultRepresentationClass());
          Object object = null;
          try{
            object = clipboard.getContents(null).getTransferData(dataFlavor);
          }catch (Exception e){
            JOptionPane.showMessageDialog(null, "Clipboard Doesn't seem to have an Image in it.");
            return "";
          }
          BufferedImage img = (BufferedImage) object;
          BufferedImage bimg = null;
          int w = img.getWidth(null);
          int h = img.getHeight(null);
          bimg = new BufferedImage(w,h,BufferedImage.TYPE_INT_RGB);
          ImageIcon ii = new ImageIcon(img);
          ImageObserver is = ii.getImageObserver();
          bimg.getGraphics().setColor(new Color(255, 255, 255));
          bimg.getGraphics().fillRect(0, 0, w, h);
          bimg.getGraphics().drawImage(ii.getImage(), 0, 0, is);
          ByteArrayOutputStream stream = new ByteArrayOutputStream();
          JPEGImageEncoder jpeg = JPEGCodec.createJPEGEncoder(stream);
          jpeg.encode(bimg);
          URL u = new URL(server);
          URLConnection con = u.openConnection();
          String boundary = "-----------------------------7d637a1aa100de";
          con.setRequestProperty("Content-Type", "multipart/form-data; boundary="+boundary);
          con.setDoOutput(true);
                String data =
                         "--"+boundary+"\r\n"+
                         "Content-Disposition: form-data; name=\"ci_uploaded\"\r\n\r\n"+
                         "JAVA\r\n"+
                         "--"+boundary+"\r\n"+
                         "Content-Disposition: form-data; name=\"ci_imagepath\"\r\n\r\n"+
                         ci_imagepath+"\r\n"+
                         "--"+boundary+"\r\n"+
                         "Content-Disposition: form-data; name=\""+ipath+"\"; filename=\"clipboard.jpg\"\r\n"+
                         "Content-Type: image/jpeg\r\n\r\n"+
                         stream.toString()+
                         "\r\n--"+boundary+"--\r\n";
                con.getOutputStream().write( data.getBytes() );
          con.connect();
          InputStream inputStream = con.getInputStream();
          byte [] urlBytes = new byte [inputStream.available()];
          inputStream.read(urlBytes);
          url = new String(urlBytes);
          System.out.print(url);
            lbl.setText("image pasted");
        } catch (Exception exc){
            lbl.setText("an error occurred: " + exc.getMessage());
        }
        return url;
      }
      public void init() {
            lbl = new JLabel("");
            lbl.setText("applet started");
            add(lbl);
        toolkit = Toolkit.getDefaultToolkit();
        clipboard = toolkit.getSystemClipboard();
      }
    }
  • Dan G. Switzer, II's Gravatar
    Dan G. Switzer, II
    @Terry:

    It's been a while since I've touched the applet, but as this entry talks about the Mac passes clipboard images in it's own x-pict DataFlavor. Images in the clipboard aren't necessarily related to the original image format.

    You should be able to use that getImageFromPictStream() method I talked about to get the image from the clipboard on the Mac:

    http://voxel.jouy.inra.fr/darcsweb/darcsweb.cgi?r=...

    This blog entry hold pretty much all knowledge I gained from the project and being so far removed from it, the details outside what I wrote are blurry.
  • javaguru's Gravatar
    javaguru
    you could just say this:
    public static BufferedImage getBufferedImage(Image img)
    {
      return ((OSXImage) img).getBufferedImage();
    }
  • javaguru's Gravatar
    javaguru
    also, typo in last paragraph, its apple.awt.OSXImage, not applet.awt.OSXImage.
  • Dan G. Switzer, II's Gravatar
    Dan G. Switzer, II
    @javaguru:

    It's been 3+ years since I worked on this code, but I seem to recall casting to OSXImage did not work in Java 1.4--which was the target JVM at the time. I may be wrong though. Also, the reason I'm not casting directly to an OSXImage is I needed a cross platform solution. Our main audience is Windows, but wanted to make sure things worked on the Mac.
  • JC's Gravatar
    JC
    You just saved me a ton of work! I was sitting very stuck...

Comments for this entry have been disabled.