Read pubsub message publish time using Apache beam with Google dataflow

By -
This is a simple message to read publishTime of a message when reading using PubSub IO.

@ProcessElement
    public void processElement(DoFn<String, MessageDTO>.ProcessContext c) throws Exception {
       
        MessageDTO dataPacket = c.element();
       
        Instant time = c.timestamp();
      
        dto.setPubsubTimestamp(time.toDateTime());

        c.output(dto);
       }

Comments

Post a Comment

Popular posts from this blog

Struts 2 : Warning :No configuration found for the specified action: 'Login.action' in namespace: '/'

By -
No configuration found for the specified action: 'Login.action' in namespace: '/'''. Form action defaulting to 'action' attribute's literal value. Solution: To remove this warning you need to change following... See your struts.xml file what name you have specified... If you are using namespace / and your action name is 'Login' then  replace Login.action to Login as given below... <s:form action=" Login.action " method="post" >   Use below with namespace  1) <s:form action=" Login " method="post" namespace="/">   Or   2) <s:form action=" /Login " method="post" >   Now you will not get this warning.
Read more

Upload csv file at Google Cloud Storage using java

By -
First do all required pre requiste as per google doc https://cloud.google.com/appengine/docs/java/googlestorage/ Now see the sample code for a csv file uploader using latest api public String uploadFileUsingGCSClient(String csvDate, String fileName, String dirName,String bucketName) throws IOException {  // init the bucket access     GcsService gcsService = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());     GcsFilename filename = new GcsFilename(bucketName+"/"+dirName, fileName);     GcsFileOptions fileOptions = new GcsFileOptions.Builder()    .mimeType("application/CSV")       .acl("public-read")    //.addUserMetadata("myfield1", "my field value")    .build();     GcsOutputChannel outputChannel = gcsService.createOrReplace(filename, fileOptions);          // write file using this st...
Read more

Read Images from a xlsx file using Apache POI

By -
This article is all about how to read and download images inside a file in xlsx format. We will use Apache POI using java. 1. Maven dependencies commons-logging-1.2.jar  commons-collections4-4.1.jar openxml4j-1.0-betapoi-3.16.jar poi-ooxml-3.16.jar poi-ooxml-schemas-3.16.jar xmlbeans-2.6.0.jar servlet-api 2. Sharing java code import java.io.File; import java.io.FileInputStream; import java.io.FileNotFoundException; import java.io.FileOutputStream; import java.io.IOException; import java.io.InputStream; import java.util.ArrayList; import java.util.Iterator; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import org.apache.poi.ss.usermodel.PictureData; import org.apache.poi.xssf.usermodel.XSSFCell; import org.apache.poi.xssf.usermodel.XSSFDrawing; import org.apache.poi.xssf.usermodel.XSSFPicture; import org.apache.poi.xssf.usermodel.XSSFRow; import org.apache.poi.xssf.usermodel.XSSFShape; import org.apache.poi.xssf.usermodel.XSSFSheet; imp...
Read more