spring,elasticsearch,spring-data-elasticsearch , Spring data elasticSearch returns null with findOne


Spring data elasticSearch returns null with findOne

Question:

Tag: spring,elasticsearch,spring-data-elasticsearch

I'm testing Spring data with elasticSearch. The ES server is running on a remote server in tha same room.

I have one index created a day, under an alias. I'm trying to find a simple tweet. But when I try a findOne(), it doesn't seem to work because it returns always null.

Also, findAll(ids) doesn't work because I'm using the alias, but I can't find in the documentation how to handle this.

What do I want to achieve ?

For the moment, simply retrieve a tweet with a given id_str.
The count method works, the findOne doesn't

Here are my questions
What should I do to make findOne() to work ?
Which way should I use to search on multiple indexes in this alias ?

Here is how the datas looks like in ES

{
    "id_str" : "135131315100051",
    "..." : "...",
    "user" : {
        "id_str" : "15843643228"
        "..." : "..."
    }
}

My model

import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;

@Document(indexName = "alias", type = "tweets")
public class Tweet
{
    @Id
    @Field(type = FieldType.String)
    private String idStr;

    public String getIdStr()
    {
        return idStr;
    }

    public void setIdStr(final String idStr)
    {
        this.idStr = idStr;
    }

    @Override
    public String toString()
    {
        return "{ id_str : " + idStr + " }";
    }
}

Alias is alias, and indexes are alias_dd-mm-yyyy

My repository

import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;

import com.thales.communications.osintlab.bigdata.webservices.models.Tweet;

public interface EsTweetRepository extends ElasticsearchRepository<Tweet, String>
{
    Tweet findByIdStr(String idStr);
}

My test

    @Test
    public void shouldReturnATweet()
    {

        //lets try to search same record in elasticsearch
        final Tweet tweet1 = tweetRepository.findOne("593768150975512576");
        //final Tweet tweet = tweetRepository.findByIdStr("593897683661824000");
        System.out.println("Count is " + tweet1);
        //System.out.println("Count is " + tweetRepository.count());
        //      System.out.println(tweet.toString());
    }

Of course, the tweet with the tested Id exists :). And the count() is working fine.

Thanks for your help


EDIT

Here is a sample application of what I have : https://github.com/ogdabou/es-stackoverflow-sample

It seems that spring-data-elasticsearch is look for the field "_id" and not the field "id_str". Maybe because of method parsing (look there). I'm looking for a way to bind my json "id_str" attribute to my idStr java model.


Answer:

What was the real issue

We set the _id field of our tweet in Elasticsearch with the id field given by twitter. But it saves it in another format ( eg 132 becomes 1.32E2)
When I'm going a findOne() it is searching for a match with the Elasticsearch _id field and not the id_str I needed.

Solution

There, you have 2 commits, the first is the issue, the second the solution.

New repository

public interface EsTweetRepository extends ElasticsearchRepository<Tweet, String>
{
    @Query("{\"bool\" : {\"must\" : {\"term\" : {\"id_str\" : \"?0\"}}}}")
    Tweet findByIdStr(String idStr);
}

The model

@Document(indexName = "my_index_01", type = "tweets")
public class Tweet
{
    // Elasticsearch object internal id. Look at field "_id"
    @Id
    private String id;

    // Twitter internal id, saved under the "id_str" field
    @Field(type = FieldType.String)
    private String id_str;

    @Field(type = FieldType.String)
    private String text;

    public String getId_str()
    {
        return id_str;
    }

    public void setId_str(final String id_str)
    {
        this.id_str = id_str;
    }

    public String getText()
    {
        return text;
    }

    public void setText(final String text)
    {
        this.text = text;
    }

    public String getId()
    {
        return id;
    }

    public void setId(final String id)
    {
        this.id = id;
    }

    @Override
    public String toString()
    {
        return "{ _id : " + id + ", id_str :  " + id_str + ", text : " + text + " }";
    }
}

Related:


Elasticsearch geospatial search, problems with index setup


elasticsearch,geospatial
I'm trying to search for documents previously added to an index, which has been configured to allow geospatial queries (or so I think). My elasticsearch instance is hosted on qbox.io. This is the code I wrote to create an index from the command line curl -XPOST username:[email protected]/events -d '{ "settings"...

Spring: @NestedConfigurationProperty List in @ConfigurationProperties


spring,properties,configuration,spring-boot
Hi I am trying to get the following configuration up and running. @ConfigurationProperties(prefix="my") public class Config { @NestedConfigurationProperty private List<ServerConfiguration> servers = new ArrayList<ServerConfiguration>(); public List<ServerConfiguration> getServers() { return this.servers; } } @ConfigurationProperties(prefix = "server") public class ServerConfiguration { private String name; private String description; } So, I want to...

Elasticsearch NumberFormatException when running two consecutive java tests


java,date,elasticsearch,numberformatexception,spring-data-elasticsearch
I have two test in a class, each of them containing the following query: SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(matchAllQuery()).withFilter(rangeFilter("publishDate").lt(date)).build(); In one of the tests, the number of the results elasticsearchTemplate.count(searchQuery, Article.class), in the other one the returned values are verified elasticsearchTemplate.queryForPage(searchQuery,Article.class) If I run any of these two tests separately,...

How to compute the scores based on field data in elasticsearch


elasticsearch
I have the following fields in documents { name: "Pearl", age : 43, weight: 54, bodyWeight : 103, height : 1.8 } Now i want to get scores for the documents based on the bodyWeight to height ratio of the documents. How to do that?...

Logging operations in lightadmin


java,spring,logging,lightadmin
I have a Java web application which use Spring and Hibernate and I plan to use lightadmin to provide an administration interface. However, I found very little information about the logging part of lightadmin : if I have such an adminsitration interface, I would like that any operation made to...

Default Spring bean when profile is not present


java,spring,spring-mvc,spring-profiles
I'm using Spring MVC controllers, and I'd like to make some extra request-mapped methods available during development. I can do this using Spring profiles: @Controller @Profile("!dev") public class DefaultController { } @Controller @Profile("dev") public class DevController extends DefaultController { } But ideally, I'd rather not pollute the production classes' code...

@RestController throws HTTP Status 406


java,spring,rest,maven
I am working on a basic Hello World program using Spring and Restful webservices. But when I try to call my service I am getting below error message: HTTP Status 406 - description - The resource identified by this request is only capable of generating responses with characteristics not acceptable...

How can implement long running process in spring hibernate?


java,spring,hibernate
as we know the struts interceptor execute and wait will take care of long running process by not getting the request to timeout and destroy it sends wait and at last the desired response i want to implement the same for long running process in spring and hibernate. Thanks....

Hibernate : Stale state exception


java,spring,hibernate,spring-mvc,transactions
I am working on a Spring-MVC application in which I am trying to delete an object from the database. Some days back, this error all of a sudden started, and now I am unable to delete. I checked on net, but I cannot find what am I doing wrong and...

Error while setting targetConnectionFactory in UserCredentialsConnectionFactoryAdapter Spring 4


java,spring,jms,spring-boot,spring-jms
We are receiving a compilation error in the following code, while trying to set the targetConnectionFactory. According to the example given in Spring 4 documentation for UserCredentialsConnectionFactoryAdapter, the setTargetConnectionFactory() method takes the object of JndiObjectFactoryBean as an argument. But, looks like the JndiObjectFactoryBean hasn't implemented the 'ConnectionFactory' interface, which is...

Why is my Spring Boot autowired JPA Repository failing JUnit test?


java,spring,junit,spring-boot
My JUnit test is failing with the following error: "java.lang.IllegalArgumentException: Could not find field [userRepository] of type [null] on target [[email protected]]" Here is the Test class: @RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration(classes = App.class) @WebAppConfiguration @IntegrationTest public class UserResourceTest { @Autowired private UserRepository userRepository; private MockMvc restUserMockMvc; @Before public void setup() { UserResource userResource...

JdbcTemplate Mockito ClassCastException


spring,mockito,classcastexception
I am trying to mock a method of the Spring framework's JdbcTemplate class. The method has the following signature: public <T> T queryForObject(String sql, Class<T> requiredType) throws DataAccessException {...} The mocking is being done as mentioned below: when(jdbcTemplate.queryForObject(anyString(), eq(String.class))).thenReturn("data"); However, this call throws the following exception java.lang.ClassCastException:...

How to apply HandlerInterceptor to Spring Boot Actuator endpoints?


java,spring,spring-mvc,spring-boot
I'm working with Java 8 and Spring 1.2.3 to build an application running inside a Tomcat 7 container. I do intercept every single call to my web application using a very simple HandlerInterceptor, that logs the overall time taken to create a response and the return code for every request....

How to write search queries in kibana using Query DSL for Elasticsearch aggregation


elasticsearch,querydsl,kibana-4
I am working on ELK stack to process Apache access logs. Spent a lot of time understanding Query DSL format so that more complex queries can be written. Currently am facing issues with running the queries in kibana interface but the same queries work just fine when posted using curl...

Using Spring Security, what's the right way to manage authorization to query responses?


java,spring,spring-security
Using Spring Security (v. 3.2.5), what's the right way to manage authorization to query responses? I have configured Spring Security to regulate access to different parts of the system based on user roles. But in many places, the principal is querying for data (e.g. looking up previously persisted details "owned"...

Spring OAuth2 not giving refresh token


java,spring,spring-security,spring-security-oauth2
I am running a OAuth Provider using Spring and "password" grant type. Running this (provider is on port 8080): curl -u "app:appclientsecret" "http://localhost:8080/oauth/token" --data "grant_type=password&username=marissa&password=koala" returns: {"access_token":"56da4d2b-7e66-483e-b88d-c1a58ee5a453","token_type":"bearer","expires_in":43199,"scope":"read"} For some reason there is no refresh token. I know according to the spec, the refresh token is optional; is there some way...

Spring Resttemplate login fails


java,spring,spring-mvc,spring-security,csrf
after an update to spring-4.1.6 i'am not able to login to my rest services any more. I looked on different sides, but couldn't solve the problem... so i ask for help. Here is my my web.xml: <?xml version="1.0" encoding="ISO-8859-1"?> <web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" version="2.5"> <!-- The definition of the...

ElasticSearch - Configuration to Analyse a document on Indexing


elasticsearch
In a single request, I want to retrieve documents from a SOR, store them in ElasticSearch, and then search those documents using the ES search API. There seems to be some lag from the time the document is indexed and the time it is analyzed and ready to be searched....

How to use arrays in lambda expressions?


c#,elasticsearch,nest
I am writing a program with NEST library of ElasticSearch. I want to write a lambda expression for a function with this argument: HighlighDescriptor<parentdocument> HighlighDescriptor.onFields (param Action<HighlightFieldDescriptor<ParentDocument>>[] fieldHighlighters) I don't know what is the array in the function argument?...

GenericApplicationContext cannot be cast to WebApplicationContext : Spring Web Flow


java,spring,spring-mvc,classcastexception,spring-webflow-2
I'm trying to set up Spring Web Flow using only Java annotations in a Spring environment that also uses only Java annotations. However when I attempt to access my flow I get the following exception SEVERE: Servlet.service() for servlet [dispatcher] in context with path [/forms] threw exception [Request processing failed;...

Spring Data JPA user posts


java,spring,jpa,data
I have user login and profile view, I would like the users to have posts. Can someone guide me in the right direction? I have a user entity: @Entity @Table(name = "usr", indexes = { @Index(columnList = "email", unique = true) }) // using usr because in may conflict with...

ElasticSearch asynchronous post


database,post,asynchronous,elasticsearch,get
I'm posting data on my ElasticSearch database. I've noticed that data is not immediately available, it requires some milliseconds to show up in a GET request. I can live with that (after all, the calls are asynchronous so this behavior is expected) but in my test code I need to...

Spring Data Rest: Return Resources of User


java,spring,spring-security,spring-data-rest
I'm creating a simple CRUD-App (a shoppinglist) with Spring Boot and Spring Data Rest. I have a resource: ShoppingItem. Is there an easy way to only return the resources which belong to the user who send the request? (Multiple User support) So a User only get his own ShoppingItems and...

Re-index object with new fields


elasticsearch,nest
It seems like as long as the id field is maintained, its super easy to re-index a document by simply calling Index(), but is there a way to given an object was updated and new fields were added, to have it include these new fields in the index? I'm not...

viewResolver with more folders inside of WEB-INF/jsp is not working in spring


java,spring,jsp,spring-mvc
<bean id="viewResolver" class="org.springframework.web.servlet.view.InternalResourceViewResolver"> <property name="viewClass" value="org.springframework.web.servlet.view.JstlView" /> <!-- avoid '???' --> <property name="prefix" value="/WEB-INF/jsp/" /> <property name="suffix" value=".jsp"/> </bean> if i create other subfolders under jsp , for instance /WEB-INF/jsp/reports , /WEB-INF/jsp/insertions how should i configure now the viewResolver to can resolve these new sub folders??...

Dynamic fields thymeleaf list iteration


spring,spring-mvc,spring-boot,thymeleaf
I'm getting a really weird error ! while iterating on list thymeleaf identified index as a property of my bean and not an index value ! <div th:each="phoneStat : *{phones}"> <select th:field="*{phones[__${phoneStat.index}__].variety}" > <option> </option> </select> <div class=" input-field col s4" > <input th:field="*{phones[__${phoneStat.index}__].number}" th:id="${'phonenumber-'+ phones[__${phoneStat.index}__]}" type="text" class="validate" /> <label...

How to implement customer subdomain in Spring framework


spring,saas,multi-tenancy
In many of the SaaS web applications (ex, Atlassian JIRA), a user can have dedicated subdomain. For example, if my user name is helloworld, then after I log in to the web application, I am redirected to helloworld.atlassian.net How to implement this in Spring Framework? Do I have to have...

Cron expression must consist of 6 fields (found 1 in “#{systemEnvironment['db_cron']}”)


spring,groovy,cron,spring-el
I'm trying to set a cron Scheduled annotation as follows: @Scheduled(cron = "#{systemEnvironment['db_cron']}") def void schedule() { } Next set the environment variable as: export db_cron="0 19 21 * * *" However, I get the runtime error: Cron expression must consist of 6 fields (found 1 in "#{systemEnvironment['db_cron']}") What can...

Get document on some condition in elastic search java API


java,elasticsearch,elasticsearch-plugin
As I know we can parse document in elastic search, And when we search for a keyword, It will return the document using this code of java API:- org.elasticsearch.action.search.SearchResponse searchHits = node.client() .prepareSearch() .setIndices("indices") .setQuery(qb) .setFrom(0).setSize(1000) .addHighlightedField("file.filename") .addHighlightedField("content") .addHighlightedField("meta.title") .setHighlighterPreTags("<span class='badge badge-info'>") .setHighlighterPostTags("</span>") .addFields("*", "_source")...

Can't Override Equals method on Java when calling contains


java,spring,hibernate
I have the following funtion that checks if a List of codigos contains a single codigos object: if (!concorrente.getJcodigoses().contains(cod)) { return "redirect:" + referrer; } I read that i need to Override the equals method like so: @Override public boolean equals(Object object) { boolean isEqual= false; if (object != null...

Override Spring form error messages


java,spring,default,custom-errors
In Spring how do I override default form error massages ? I'm using a Validator and a properties file to add my own error messages, but how do I override messages that get printed on conversion/encoding error for example ? They seem to be generated automatically and I don't think...

How to set property using “tasklet ref” tag


spring,spring-batch
I have a tasklet ValidarSituacaoTasklet that has an property situacao. This tasklet is used in 2 steps in distinct values for situacao. I declared steps as like: and the bean: <bean id="validarSituacaoTasklet" class="my.package.tasklet.ValidarSituacaoTasklet" scope="step"> </bean> I have to pass 'situacao' to tasklet . I tried: <step id="validaSituacaoStep"> <tasklet ref="validarSituacaoTasklet ">...

MultiMatch query with Nest and Field Suffix


c#,elasticsearch,nest
Using Elasticsearch I have a field with a suffix - string field with a .english suffix with an english analyser on it as shown in the following mapping ... "valueString": { "type": "string", "fields": { "english": { "type": "string", "analyzer": "english" } } } ... The following query snippet won't...

Cannot create bean when start the application


java,spring,javabeans,sftp,job-scheduling
I am facing some issue while creating the bean it gives error. Can someone please assist me. I will retrieve the files from sftp resource and will read those file with a spring task scheduler job. App-Config.xml <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context" xmlns:task="http://www.springframework.org/schema/task"...

Not able to access Kibana running in a Docker container on port 5601


elasticsearch,docker,dockerfile,kibana-4
I have built a docker image with the following Docker file. # gunicorn-flask FROM devdb/kibana MAINTAINER John Doe <[email protected]> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update RUN apt-get install -y python python-pip python-virtualenv gunicorn # Setup flask application RUN mkdir -p /deploy/app COPY gunicorn_config.py /deploy/gunicorn_config.py COPY app /deploy/app RUN pip install...

Spring Boot - How to set the default schema for PostgreSQL?


java,spring,hibernate,postgresql
What I am currently doing in my application.properties file is: spring.datasource.url=jdbc:postgresql://localhost:5432/myDB?currentSchema=mySchema Isn't there another property for this? Since it looks hacky and according to a post (still finding the link sorry, will update later), it is only applicable to PostgreSQL 9.4....

elastic search sort in aggs by column


sorting,elasticsearch,group-by,order
I am trying to sort in elastic search in aggs, equivalent in mysql "ORDER BY Title ASC/DESC". Here is the index structure: 'body' => array( 'mappings' => array( 'test_type' => array( '_source' => array( 'enabled' => true ), 'properties' => array( 'ProductId' => array( 'type' => 'integer', 'index' => 'not_analyzed'...

shall I use Spring framework for a performance-critical proxy application? [closed]


java,spring,authentication,servlets
I've created a servlet (Tomcat) application which has these functions: It performs HTTP Basic Authentication. It connects to a user and role database. It works as "security facade" for some geodata servers behind It forwards requests after doing some authorization tests In case the response contains XML data, it performs...

Parsing Google Custom Search API for Elasticsearch Documents


json,python-2.7,elasticsearch,google-search-api
After retrieving results from the Google Custom Search API and writing it to JSON, I want to parse that JSON to make valid Elasticsearch documents. You can configure a parent - child relationship for nested results. However, this relationship seems to not be inferred by the data structure itself. I've...

Importing Spring MVC Project Manually to Eclipse


java,eclipse,spring,spring-mvc
I have a little problem. I had googled for this problem and I haven't found any solution related. My problem is, my office internet using proxy that banned me to download zip file. I need Spring MVC Project template in Spring Tool Suite (STS). So I downloaded the zip file...

How to read data in logs using logstash?


elasticsearch,logstash
I have just started log stash, i have log files in that log file whole object is printed in the logs, Since my object is huge i cant write the grok patterns to the whole object and also i expecting only two values out of those object. Can you please...

Is it possible to construct an object partially with Spring DI and partially with Jersey DI?


spring,dependency-injection,jersey,jersey-2.0,hk2
I have a class JerseyWebService which uses Jersey DI to inject dependency @Path("/baskets") public class JerseyWebService { @Inject ExternalApiServiceInterface api; ... } The dependency is specified in the binder public class CustomBinder extends AbstractBinder { @Override protected void configure() { bind(ExternalApiService.class).to(ExternalApiServiceInterface.class); ... } But the problem here is that ExternalApiService...

Trouble with Login using Spring Boot and JDBC Security


spring,spring-security,spring-boot
My build.gradle file: task wrapper(type: Wrapper) { gradleVersion = '2.4' } buildscript { repositories { mavenCentral() maven { url "http://repo.spring.io/release" } } dependencies { classpath("org.springframework.boot:spring-boot-gradle-plugin:1.2.3.RELEASE") } } apply plugin: 'java' apply plugin: 'spring-boot' repositories { mavenCentral() maven { url "http://repo.spring.io/release" } } dependencies { compile("org.springframework.boot:spring-boot-starter-web") compile("org.springframework.boot:spring-boot-starter-jdbc")...

Operator '??' cannot be applied to operands of type IQueryContainer and lambda expression


c#,elasticsearch,nest
I am trying to create a method to process a certain query. I follow an example posted on the Nest repository (line 60), but still the MatchAll is not recognized by the compiler and if I try to build the solution, the error that shows is: Operator '??' cannot be...

Spring: Response time


java,spring,spring-ws
I have a SOAP webservice(spring ws)and need to save the response time of every request that I receive. I could create a servlet filter that measures time difference between HTTP Request and HTTP Response. But, I need to log the response time together with some values that I read from...

Transaction error in Spring


java,spring,jpa
I have a project https://github.com/ivartanian/onlinecontrol in Spring MVC + JPA, initially worked with the Spring configuration through XML. Now I am trying to set up the project through JAVA Config (without XML). The project starts, but when I create a new user(entity User), an error transaction. But if I create...

NEST ElasticSearch.NET Escape Special Characters


c#,elasticsearch,nest
I have been experimenting with the use of the NEST client for Elastic Search, but seem to have hit a barrier when filtering on a term which contains special/reserved characters such as '/' Below is a JSON representation of my model.. "categories": { "count": 1, "default": "root/Hello/World/Category", } When submitting...

NEST - Using GET instead of POST/PUT for searching


c#,elasticsearch,nest
Is there a way to tell NEST to use GET instead of POST when performing searches? Similar to how the ElasticSearch documentation shows CURL using GET I'd like to use GET when using NEST instead of using POST as it currently does.

Docker container http requests limit


http,elasticsearch,docker
I'm new to Docker so, most likely, I'm missing something. I'm running a container with Elasticsearch, using this image. I'm able to setup everyhing correctly. After that I was a using a script developed by a collegue in order to insert some data, basically querying a MySQL database and making...

Bad scoring due to different maxDocs of IDF


elasticsearch
I have two documents with a field title of: News New Website If I search for the term new website the score for the News document is much higher than the other one which is obviously not what I want. I wrapped an explain around it and got: 'hits': [{'_explanation':...