grails,hadoop,amazon-web-services,amazon-redshift , Amazon Redshift: query execution hangs

Amazon Redshift: query execution hangs


Tag: grails,hadoop,amazon-web-services,amazon-redshift

I use amazon redshift and sometimes the query execution hangs without any error messages e.g. this query will execute:

select extract(year from date), extract(week from date),count(*) from some_table  where date>'2015-01-01 00:00:00' and date<'2015-12-31 23:59:59' group by extract(year from date), extract(week from date)

and this not:

select extract(year from date), extract(week from date),count(*) from some_table  where date>'2014-01-01 00:00:00' and date<'2014-12-27 23:59:59' group by extract(year from date), extract(week from date)

But it happens only when I deploy project to server and on my local machine all queries executed without any problems.

I already set in my code autoCommit=true for connection. Also all things listed above I do with grails using this library

        compile 'com.amazonaws:aws-java-sdk-redshift:1.9.39'

Any ideas?


This might not be an exact answer, but it's too long for a comment.

You may want to check mtu setting on the server where you performing the execution.

Redshift want's to operate on 1500 bytes frame and all EC2 instances are set with jumbo frame by default (9000)

In order for you to run queries without problems you need to have the same mtu setting.

To check what you currently have, run this command:

ip addr show eth0

an example output would be like this:

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP qlen 1000

in this case mtu is 9001 so you need to change it to 1500 by running:

/sbin/ifconfig eth0 mtu 1500 up


Grails logging auto inject

I use grails 3.0.2 and logback. How can I use this logger in my code, can I somehow autoinject it, like log.debug() in grails 2.x which was using org.apache.commons.logging.Log?

SQL Server 2012 & Polybase - 'Hadoop Connectivity' configuration option missing

As described in the title, I am using SQL Server 2012 Parallel Data Warehouse with Polybase feature to try to access a HDInisght Hadoop cluster. As a starting point for every connection to Hadoop from SQL Server, I find to execute the command sp_configure @configname = 'hadoop connectivity', @configvalue =...

hadoop complains about attempting to overwrite nonempty destination directory

I'm following Rasesh Mori's instructions to install Hadoop on a multinode cluster, and have gotten to the point where jps shows the various nodes are up and running. I can copy files into hdfs; I did so with $HADOOP_HOME/bin/hdfs dfs -put ~/in /in and then tried to run the wordcount...

Save flume output to hive table with Hive Sink

I am trying to configure flume with Hive to save flume output to hive table with Hive Sink type. I have single node cluster. I use mapr hadoop distribution. Here is my flume.conf agent1.sources = source1 agent1.channels = channel1 agent1.sinks = sink1 agent1.sources.source1.type = exec agent1.sources.source1.command = cat /home/andrey/ agent1.sinks.sink1.type...

Create an external Hive table from an existing external table

I have a set of CSV files in a HDFS path and I created an external Hive table, let's say table_A, from these files. Since some of the entries are redundant, I tried creating another Hive table based on table_A, say table_B, which has distinct records. I was able to...

Grails: Carry forward params on g:actionSubmit is clicked

How to carry forward the parameters when g:actionSubmit button is clicked? Here is my gsp code: <g:checkBox name="msgCheck" checked="" value="${userInstance.emailId}"></g:checkBox> ... <g:actionSubmit class="update" action="delete" value="Delete" params="${params}"></g:actionSubmit> Here is my controller code: def delete() { try { def user_list = params.msgCheck //deleting the user //successful. redirect(action: "list", params: params) } catch...

issue monitoring hadoop response

I am using ganglia to monitor Hadoop. gmond and gmetad are running fine. When I telnet on gmond port (8649) and when I telnet gmetad on its xml answer port, I get no hadoop data. How can it be ? cluster { name = "my cluster" owner = "Master" latlong...

How to insert and Update simultaneously to PostgreSQL with sqoop command

I am trying to insert into postgreSQL DB with sqoop command. sqoop export --connect jdbc:postgresql:// --table table1 --username user1 --password pass1--export-dir /hivetables/table/ --fields-terminated-by '|' --lines-terminated-by '\n' -- --schema schema It is working fine if there is not primary key constrain. I want to insert new records and update old records...

jets3t cannot upload file to s3

I'm trying to upload files from local to s3 using hadoop fs and jets3t, but I'm getting the following error Caused by: java.util.concurrent.ExecutionException: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: Request Error. HEAD '/project%2Ftest%2Fsome_event%2Fdt%3D2015-06-17%2FsomeFile' on Host '' @ 'Thu, 18 Jun 2015 23:33:01 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: AVDFJKLDFJ3242, HostId: D+sdfjlakdsadf\asdfkpagjafdjsafdj I'm...

Render a controller into a String

I can render a template or a view into a String, but how can I render a controller into a String ? I would like to perform this kind of operation: def myAction = { ... def html = renderToString(controller: 'myController', action: 'myAction', params: [what:'ever']) render modify(html) ... } ...

Hadoop map reduce Extract specific columns from csv file in csv format

I am new to hadoop and working on a big data project where I have to clean and filter given csv file. like if given csv file has 200 columns then I need to select only 20 specific columns (so called data filtering) as a output for further operation. also...

Sqoop Export with Missing Data

I am trying to use Sqoop to export data from HDFS into Postgresql. However, I receive an error partially through the export that it can't parse the input. I manually went into the file I was exporting and saw that this row had two columns missing. I have tried a...

Hadoop append data to hdfs file and ignore duplicate entries

How can I append data to HDFS files and ignore duplicate values? I have a huge HDFS file (MainFile) and I have 2 other new files from different sources and I want to append data from this files to the MainFile. Main File and the other files has same structure....

Why we are configuring mapred.job.tracker in YARN?

What I know is YARN is introduced and it replaced JobTracker and TaskTracker. I have seen is some Hadoop 2.6.0/2.7.0 installation tutorials and they are configuring as yarn and mapred.job.tracker property as local or host:port. The description for mapred.job.tracker property is "The host and port that the MapReduce job...

unable to resolve class in grails

I am developing a module in which some file need to be uploaded on a ftp server for that I have added the "commons-net-3.3.jar" in my grails app & while using it import import import import import It is throwing the error unable to resolve...

Grails 3.0 Searchable plugin

I've tried to use the searchable plugin in grails 3.0, but I am not sure wheather it is compatible or not. I've added compile "org.grails.plugins:searchable" to the dependencies. The build completes but yields a NotSearializableException: > Configuring > 1/1 projects > Resolving dependencies ':agent' > Resolving dependencies ': BUILD SUCCESSFUL...

Hive external table not reading entirety of string from CSV source

Relatively new to the Hadoop world so apologies if this is a no-brainer but I haven't found anything on this on SO or elsewhere. In short, I have an external table created in Hive that reads data from a folder of CSV files in HDFS. The issue is that while...

Importtsv command gives : Container exited with a non-zero exit code 1 error

I am trying to load a tsv file into an existing hbase table. I am using the following command: /usr/local/hbase/bin$ hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.columns=HBASE_ROW_KEY,cf:value '-Dtable_name.separator=\t' Table-name /hdfs-path-to-input-file But when I execute the above command, I get the following error Container id: container_1434304449478_0018_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(

Grails 2.4.4 spring security role doesn't apply to user

I have controller: class AdminController { def springSecurityService @Secured(['ROLE_ADMIN', 'ROLE_USER']) def index() { render "test"; } And user with role ROLE_ADMIN in the table: But, when I use: springSecurityService.getPrincipal().getAuthorities() There is only one role: ROLE_NO_ROLES Why? def loggedInUser = springSecurityService.currentUser; returns correct user. Config: ... grails.plugin.springsecurity.userLookup.userDomainClassName = '...' grails.plugin.springsecurity.userLookup.authorityJoinClassName =...

Log Grails Pre 3.0 startup time

In Grails 2.x.x is it somehow possible to log the startup time? We are currently trying to optimize the application startup and having the startup time logged would help tremendously.

Groovy - timestamp from minutes

I have an array or times/values coming back to be in an array like: [0, 60] Which are times in minutes, 0 = 12:00 a.m, 60 = 1:00 a.m. I am wanting to store these in an oracle database as timestamps. How do I convert minutes into timestamps in groovy?...

grails DataSource.groovy refer bean for decoding password

Referring, we need to specify a class with static methods to decode the encrypted password in DataSource config. For passwordEncryptionCodec property, is it possible to refer to a bean defined in resources.xml or any other grails bean definition files? For ex: having a decrypted Properties bean in resoures.xml or...

Intercepting login calls with Spring-Security-Rest plugin in Grails

I am using the spring security rest plugin for Grails to provide a login mechanism for an AngularJS app. Login works fine, but I can't figure out how to intercept login calls, in order to store additional statistics on (invalid/valid) login attempts. As I am quite new to Spring Security...

Flink error - org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4

I am trying to run a flink job using a file from HDFS. I have created a dataset as following - DataSource<Tuple2<LongWritable, Text>> visits = env.readHadoopFile(new TextInputFormat(), LongWritable.class,Text.class, Config.pathToVisits()); I am using flink's latest version - 0.9.0-milestone-1-hadoop1 (I have also tried with 0.9.0-milestone-1) whereas my Hadoop version is 2.6.0 But,...

Input of the reduce phase is not what I expect in Hadoop (Java)

I'm working on a very simple graph analysis tool in Hadoop using MapReduce. I have a graph that looks like the following (each row represents and edge - in fact, this is a triangle graph): 1 3 3 1 3 2 2 3 Now, I want to use MapReduce to...

how to drop partition metadata from hive, when partition is drop by using alter drop command

I have dropped the all the partitions in the hive table by using the alter command alter table emp drop partition (hiredate>'0'); After droping partitions still I can see the partitions metadata.How to delete this partition metadata? Can I use the same table for new partitions? ...

Grails 3.0.2 missing generate-views

I'm new to grails and I'm trying to get my first HelloWorld from it. I can generate it with the controller, but I can't create a view because it isn't listed in the Grails (3.0.2) list of command. Because any IDE support grails 3.*, I can't create it by that...

Apache Spark: Error while starting PySpark

On a Centos machine, Python v2.6.6 and Apache Spark v1.2.1 Getting the following error when trying to run ./pyspark Seems some issue with python but not able to figure out 15/06/18 08:11:16 INFO spark.SparkContext: Successfully stopped SparkContext Traceback (most recent call last): File "/usr/lib/spark_1.2.1/spark-1.2.1-bin-hadoop2.4/python/pyspark/", line 45, in <module> sc =...

File upload with grails and jquery

I have a form with multiple inputs plus a doc or pdf file upload. //Multiple input fields are here then upload -> <div class="controls"> <div class="fileupload ${ person?.attachment ? 'fileupload-exists' : 'fileupload-new' }" id="attachment" data-provides="fileupload" data-name="attachment"> <span class="btn btn-file"> <span class="fileupload-new">Add</span> <span class="fileupload-exists">Change</span> <input type="file" /> </span> <span class="fileupload-preview"></span>...

From Hadoop logs how can I find intermediate output byte sizes & reduce output bytes sizes?

From hadoop logs, How can I estimate the size of total intermediate outputs of Mappers(in Bytes) and the size of total outputs of Reducers(in Bytes)? My mappers and reducers use LZO compression, and I want to know the size of mapper/reducer outputs after compression. 15/06/06 17:19:15 INFO mapred.JobClient: map 100%...

Encryption of strings using AES 128 in Java/grails

I would like to encrypt 3 strings using AES 128 in Java / Grails, and using the code below, but i get the error "An error occurred when encrypting", can someone tell me what is wrong with my code, how to fix it. thanks in advance and to Stackoverflow. String...

Vertica: Input record 1 has been rejected (Too few columns found)

I am trying to copy file from Hadoop to a Vertica table and get the an error. The problem is same copy sometimes pass and some times fails,any idea? The Error: Caused by: java.sql.SQLException: [Vertica]VJDBC ERROR: COPY: Input record 1 has been rejected (Too few columns found) at com.vertica.util.ServerErrorData.buildException(Unknown Source)...

Grails JAX-RS Calling a class in src/groovy giving error - Message: No signature of method: is applicable for argument types

While developing the rest api using jaxrs plugin I need to create some common class which I have created in "src/groovy". Below is the class class ValidateToken { String validate(String token){ println(token) return "test" } //... In resource file(jaxrs) this is what I am doing def instance=ValidateToken.validate("test") This throws error...

How to run hadoop appliaction automatically?

I know that a MapReduce program can be ran using the command line "hadoop jar *.jar" for a time. But now the program is required to be ran a time for every hour in background. Are there any methods to make the MR program be hourly submitted to hadoop automatically?...

Decode base64 image in Grails [duplicate]

This question already has an answer here: Convert base64 string to image 3 answers I have a post api where I am sending a json string which contain the base64 encoded image.Below is the json string { "imageData":"base64encoded string", "status":"1" } where base64encode string is iVBORw0KGgoAAAANSUhEUgAAAHgAAACgCAIAAABIaz/HAAAAAXNSR0IArs4c6QAA\r\nABxpRE9UAAAAAgAAAAAAAABQAAAAKAAAAFAAAABQAABWL3xrAqoAAEAASURBVHgB\r\nlL2Fe1t7mueZme6uewNGMUu2LNkyySSjDJKZmZkSO8zM7CTmmJnZYbxUVbdgsKp7\r\nqqdrdp I cant post...

Grails: Do addTo* and removeFrom* require a call to save?

In the docs some examples have a call to save whereas some others do not. I'm assuming that addTo* needs the call to save whereas removeFrom* doesn't. Am I mistaken?

ERROR jdbc.HiveConnection: Error opening session Hive

i try to run JBDC code for Hive2 get error. i have hive 1.2.0 version hadoop 1.2.1 version. but in command line hive and beeline works fine without any problem.but with jdbc getting error. import java.sql.SQLException; import java.sql.Connection; import java.sql.ResultSet; import java.sql.Statement; import java.sql.DriverManager; public class HiveJdbcClient { private static...

alert grails pagination current offset value

I am using Grails 2.4.2. I have a list which is using pagination. When I mouse hover a number or next page icon it shows current offset number as tool tip. Can I alert this offset number or get my list's current page offset number?

passing backbone collection to view

I'm just starting out with backbone / grails and i've been struggling to figure out how to get everything to work. I'm building a pricing configurator where a user selects a product type from radio group A and radio group B containing the quantity / pricing / discount data will...

HIVE: apply delimiter until a specified column

I am trying to move data from a file into a hive table. The data in the file looks something like this:- StringA StringB StringC StringD StringE where each string is separated by a space. The problem is that i want separate columns for StringA, StringB and StringC and one...

[B cannot be cast to java.sql.Blob

This is my domain class class TimesheetSubmission { Date submissionDate=new Date() String foreman String shift String jobId Date date byte[] xmlSubmission String xmlResponse static constraints = { submissionDate nullable: false foreman nullable: false shift nullable: false jobId nullable: false date nullable: false xmlSubmission nullable: true xmlResponse nullable: false } static...

grails 3.0.1 scaffolded view does not show domain relationship

I am following the example in the book "Grails a Quick-Start Guide" using Grails 3.0.1 with following domains package com.tekdays class TekEvent { static constraints = { name() city() description maxSize:5000 organizer() venue() startDate() endDate() } String city String name TekUser organizer String venue Date startDate Date endDate String description...

How to declare javascript asset in the view to be rendered within the footer using Grails Asset Pipeline

Since there is no deferred option for: <asset:javascript src="custom_view_script.js"/> What else can be used, outside of resource plugin, to place view specific script right before the closing body tag and without declaring it globally in layout? I do know about: <asset:deferredScripts/> but that only handles on page script and not...

Oozie on YARN - oozie is not allowed to impersonate hadoop

I'm trying to use Oozie from Java to start a job on a Hadoop cluster. I have very limited experience with Oozie on Hadoop 1 and now I'm struggling trying out the same thing on YARN. I'm given a machine that doesn't belong to the cluster, so when I try...

Add PARTITION after creating TABLE in hive

i have created a non partitioned table and load data into the table,now i want to add a PARTITION on the basis of department into that table,can I do this? If I do: ALTER TABLE Student ADD PARTITION (dept='CSE') location '/test'; It gives me error: FAILED: SemanticException table is not...

JMH Benchmark on Hadoop YARN

I have written a JMH benchmark for my MapReduce job. If I run my app in local mode, it works, but when I run it with the yarn script on my hadoop cluster, then I get the following error: [[email protected] Desktop]$ ./ # JMH 1.10 (released 5 days ago) #...

Error when using angular with Grails

Here is my index.gsp <!DOCTYPE html> <html ng-app="myApp"> <head> <title>my app</title> </head> <body> <input type="text" data-ng-model="test"/> {{test}} </body> <script src="/js/angular.min.js"></script> </html> When I build an app, There is an error like this, Uncaught Error: [$injector:modulerr]$injector/modulerr?p0=myApp&p1=Error%3A%…20at%20d%20(http%3A%2F%2Flocalhost%3A8000%2Fjs%2Fangular.min.js%3A17%3A381)...

Spark on yarn jar upload problems

I am trying to run a simple Map/Reduce java program using spark over yarn (Cloudera Hadoop 5.2 on CentOS). I have tried this 2 different ways. The first way is the following: YARN_CONF_DIR=/usr/lib/hadoop-yarn/etc/hadoop/; /var/tmp/spark/spark-1.4.0-bin-hadoop2.4/bin/spark-submit --class MRContainer --master yarn-cluster --jars /var/tmp/spark/spark-1.4.0-bin-hadoop2.4/lib/spark-assembly-1.4.0-hadoop2.4.0.jar simplemr.jar This method gives the following error: diagnostics: Application application_1434177111261_0007...

Best way to store relational data in hdfs

I've been reading a lot on hadoop lately and I can say that I understand the general concept of it, but there is still (at least)one piece of the puzzle that I can't get my head around. What is the best way to store relationnal data in hdfs. First of...

Merging two columns into a single column and formatting the content to form an accurate date-time format in Hive?

these are the 2 columns(month,year). I want to create a single column out of them having an accurate date-time format('YYYY-MM-DD HH:MM:SS') and add as new column in the table. Month year 12/ 3 2013 at 8:40pm 12/ 3 2013 at 8:39pm 12/ 3 2013 at 8:39pm 12/ 3 2013 at...