lucene,cluster-computing,liferay-6,ehcache,jgroups , Liferay 6.2 clustering issue with multicast

Liferay 6.2 clustering issue with multicast


Tag: lucene,cluster-computing,liferay-6,ehcache,jgroups

I am trying to cluster ehcache and lucene with Liferay 6.2 EE sp2 bundle on 2 servers with mutlicast enabled. WE have Apache HTTPD servers fronting tomcat servers using reverse proxy. A valid 6.2 license is deployed on both the nodes.

We user the following properties in the

# Since we are using SSL on the frontend

# set this to any server that is visible to both the nodes

#ports and ips we know work in our environment for multicast["cluster-link-control"]=ip["cluster-link-control"]=port1["cluster-link-udp"]=ip["cluster-link-udp"]=port2["cluster-link-mping"]=ip["cluster-link-mping"]=port3["hibernate"]=ip["hibernate"]=port4["multi-vm"]=ip["multi-vm"]=port5

We are running into issues with the ehcache and lucene clustering not working. The following tests fail :

  1. Moving a portlet on node 1, does not show up on node 2

There are no errors except for a startup error with lucene.

14:19:35,771 ERROR [CLUSTER_EXECUTOR_CALLBACK_THREAD_POOL-1][LuceneHelperImpl:1186] Unable to load index for company 10157 com.liferay.portal.kernel.exception.SystemException: Connection refused at at$LoadIndexClusterResponseCallback.callback( at com.liferay.portal.cluster.ClusterExecutorImpl$ at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$WorkerTask._runTask( at com.liferay.portal.kernel.concurrent.ThreadPoolExecutor$ at Caused by: Connection refused at Method) at at at at at at at at at at at at

We verified that the jgroups multicast works outside of liferay by running the following commands and using a downloaded copy of the jgroups.jar and replacing with the 5 multicast ips and ports.

Testing with JGROUPS

1) McastReceiver -

java -cp ./jgroups.jar org.jgroups.tests.McastReceiverTest -mcast_addr -port 5555

ex. java -cp jgroups-final.jar org.jgroups.tests.McastReceiverTest -mcast_addr -port 5555

2) McastSender -

java -cp ./jgroups.jar org.jgroups.tests.McastSenderTest -mcast_addr -port 5555

ex. java -cp jgroups-final.jar org.jgroups.tests.McastSenderTest -mcast_addr -port 5555

From there, typing things into the McastSender will result in the Receiver printing it out.



After a lot of troubleshooting and help from various folks in my team and at liferay support, we switched to using unicast and it worked a lot better.

Here is what we did:

All of the above tests worked.

So we shutdown servers and changed the tcp.xml to use jdbc rather than the tcpping so we don't have to specify node names manually.

Step for the jdbc config:

  1. Create the table in the liferay database manually.

    CREATE TABLE JGROUPSPING (own_addr varchar(200) not null, cluster_name varchar(200) not null, ping_data blob default null, primary key (own_addr, cluster_name))

  2. change tcp.xml and remove the tcpping section and add the following:

  3. Save and push the file manually to both the nodes.

Start the servers and repeat tests above.

It should work seemlessly.

It was invaluable to have the debug logging on for jgroups mentioned in the following the post:

tomcat home/webapps/ROOT/WEB-INF/classes/META-INF/portal-log4j-ext.xml file I used to triage various issues on bootup related to clustering.

<?xml version="1.0"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<log4j:configuration xmlns:log4j="">

<category name="com.liferay.portal.cluster">
<priority value="TRACE" />

<category name="com.liferay.portal.license">
<priority value="TRACE" />

We also found that the lucene cluster replication startup errors were fixed in a fix pack and are getting a patch for it.

We added the following portal instance properties for lucene replication to work better between the 2 nodes:

portal.instance.http.port=port that the app servers listen on ex. 8080

Hope this helps someone.


Memsql, Is it possible to add more leaf nodes on the same host machine?

I installed a functional test with Memsql on a machine with an aggregation node on port 3307 and a leaf node 3306. As queries being ran, there might be additional cpu available. Hence, is it possible to add more leaf nodes on the same machine to take advantages of available...

How to deploy three config server instances for sharding in mongodb?

I am a newbie in MongoDB. Also I have not much knowledge in networking and servers. I am trying to deploy sharded cluster in mongodb using this article. It says I need to deploy three configuration server instances and create data directories for each. I can create a data directory...

Matching && and || special character

I am writing a Lucene application to match && (double ampersand) and || (OR or double pipe). I want to write a regex to match any presence of && and || in the input text. If I write something like below, it only matches for the presence or absence of...

issue monitoring hadoop response

I am using ganglia to monitor Hadoop. gmond and gmetad are running fine. When I telnet on gmond port (8649) and when I telnet gmetad on its xml answer port, I get no hadoop data. How can it be ? cluster { name = "my cluster" owner = "Master" latlong...

Do 'reduce' with results from Cloudant search?

In Cloudant is it possible to do something like a reduce on a set of results from a search index (as opposed to a view)? In my case, I'd like to find all documents that have a title value that includes 'foo', then for each of these sum the total...

Hibernate Search not indexing items from database

I'm trying to integrate Hibernate Search in my application. Rough summary of what needs to be done: Spring Batch reads out an XML file and persists the objects to database. This is done with de JDBCBatchItemWriter. Not the HibernateItemWriter because of slow performance. After all items are inserted I would...

Lucene vs Solr, indexning speed for sampe data

I have worked upon Lucene before and now moving towards Solr. The problem is that I am not able to do Indexing on Solr as fast as Lucene can do. My Lucene Code: public class LuceneIndexer { public static void main(String[] args) { String indexDir = "/home/demo/indexes/index1/"; IndexWriterConfig indexWriterConfig =...

Normalization in Java (dataset of doubles)

I'm working on a clustering program, and have a dataset of doubles that I need to normalize in order to make sure that every double (variable) has the same influence. I would like to use min-max normalization where for every variable the min and max value are determined, but I'm...

Lucene get all non deleted document from index file

I am trying to get all documents from Lucene Index (which is already not deleted ). I heard that if I delete something from Lucene Index, Lucene will not delete immediately from file. So I wanted to get the documents from Index file which is not deleted....

Recycling app pool each time something is published

I'm working on an Umbraco site where I have custom sections, and therefore use the application.config and trees.config files. I have a problem where every time I publish something, the app pool recycles with the following message: w3wp.exe Information: 0 : _shutDownMessage=CONFIG change HostingEnvironment initiated shutdown CONFIG change HostingEnvironment caused...

Fuzzy search not working with dismax query parser

There is a field in my schema 'fullText' which is of the 'text_en' type, and multivalued. The term 'tests' is in the fullText field in one document. In solr, when I try to search using the word 'test', with the standard lucene parser with minimal distance 1, its returning the...

Understanding Apache Lucene's scoring algorithm

I'm working with Hibernate Search for months now, but still I'm not able to digest the relevance it brings. I'm overall satisfied with the results it returns, but even simplest test does not satisfy my expectation. First test was using the term frequency(tf). Data: word word word word word word...

Combining regex of characters and Strings in JAVA

I have a set of special characters, for ex. like ?. ^,! etc and also special string like && and | |, using these special characters and strings I have to write a regex that will escape all these special characters and string. Output must be something like this: \^\!\?\&\&\|\|....

bash: /usr/bin/hydra_pmi_proxy: No such file or directory

I am struggling to set up an MPI cluster, following the Setting Up an MPICH2 Cluster in Ubuntu tutorial. I have something running and my machine file is this: pythagoras:2 # this will spawn 2 processes on pythagoras geomcomp # this will spawn 1 process on geomcomp The tutorial states:...

How to use all the cores of Solr in solrj

I have downloaded solr 5.2.0 and have started using $solr_home/bin/solr start The Logs stated: Waiting to see Solr listening on port 8983 [/] Started Solr server on port 8983 (pid=17330). Happy searching! Then I visited http://localhost:8983/solr and created a new core using Core Admin / new Core as Core1 (...

Liferay 6.2 clustering issue with multicast

I am trying to cluster ehcache and lucene with Liferay 6.2 EE sp2 bundle on 2 servers with mutlicast enabled. WE have Apache HTTPD servers fronting tomcat servers using reverse proxy. A valid 6.2 license is deployed on both the nodes. We user the following properties in the

Linkage Error in Tomcat when 2 webapp instances load lucene classes

I'm running a tomcat 8 container with 2 different webapps, 1 prod and 1 sandbox. All the classes/libs and compilation is the same with just some minor differences in the config parameters. I'm using lucene core 4.10.4 (via hibernate search). Both apps startup just fine, now after startup if I...

How to add multiple suggesters definition in solr search components

I am using solr 5.1. I am trying to configure multiple suggester definition in Solr search component according to Apache solr wiki. I have configured single suggester perfectly and it works perfect but whenever I try to configure multiple suggester it gives me following errors java.lang.NullPointerException at org.apache.solr.handler.component.SearchHandler.handleRequestBody( at org.apache.solr.handler.RequestHandlerBase.handleRequest(

LUCENE_40 cannot be resolved or is not a field

StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_40); I am runing lucene search code and I am getting an error in above line saying LUCENE_40 cannot be resolved or is not a field I am using lucene 5.1.0 version. I have removed the version.LUCENE_40 from the standard analyzer parameters, so now there's no...

Elasticsearch filter only if no matches to first filter

My use case is for searching UK addresses where there is a well defined postal code system however my users may still make mistakes in the postcode. I want to use a filter as in most cases the user will get the postcode right and I do not want to...

Lucene Analyzer tokenizer for substring search

I need a Lucene Tokenizer that can do the following. Given the string "wines bottle caps", the following queries should succeed wine bott cap ottl aps wine bottl Here is what I have so far. How might I modify it to work? No query less than three characters should work....

Scripted Fields for if/else condition in Kibana 4

I have some numeric fields in elasticsearch, I have to implement some logic for which I need to create some scripted fields. I am new to kibana 4's scripted fields feature, so I need some help regarding a basic format that could be used for writing a basic if else...

jdbc backed channel on a cluster mode

I want to use the jdbc backed message channel. Reading the doc says that it is supported but after that I found a thread in the forum that explicitly says don't. Could you please clarify me if it's supported? In this case, could you guide me to do so? thanks...

Barrier after MPI non-blocking call, without bookkeeping?

I'm doing a bunch of MPI_Iallreduce non-blocking communications. I've added these Iallreduce calls to several different places in my code. Every so often, I want to pause and wait for all the Iallreduce calls to finish. Version 1 with MPI_Request bookkeeping -- this works: MPI_Request requests[]; MPI_Iallreduce(..., requests[0]); ... MPI_Iallreduce(...,...

Elastica Client with cluster

I have an Elasticsearch cluster with but I can't connect with Elastica Client. This is my configuration : $elasticaClient = new \Elastica\Client(array( 'servers' => array( array('host' => 'https://myusername:[email protected]', 'port' => 10050), array('host' => 'https://myusername:[email protected]', 'port' => 10062) ) )); $elasticaIndex = $elasticaClient->getIndex('test'); I got this error : Couldn't resolve...

How to combine a search phrase with a wildcard using Lucene.Net?

I am passing a search query to the Lucene QueryParser.Parse(string query) method, and then passing the result to Searcher.Search(Query query, int n). A string of: "system cleaner" returns 1 hit. A string of: "system clean*" or: "system clean\*" returns 0 hits. How can I provide a search query that uses...

Searching a TextField and IntField together seperated by an AND condition In Lucene

I have indexed my documents as: doc.add(new IntField("ID", id, Field.Store.YES)); doc.add(new TextField("First_Name", First_Name, Field.Store.YES)); doc.add(new TextField("Last_Name", Last_Name, Field.Store.YES)); doc.add(new TextField("Address", add, Field.Store.YES)); doc.add(new TextField("City", city, Field.Store.YES)); doc.add(new TextField("State", state, Field.Store.YES)); doc.add(new IntField("Zip_Code", zip, Field.Store.YES)); Where id, FirstName, city, add, state, zip are variables that store the values to be indexed....

How to wisely combine shingles and edgeNgram to provide flexible full text search?

We have an OData-compliant API that delegates some of its full text search needs to an Elasticsearch cluster. Since OData expressions can get quite complex, we decided to simply translate them into their equivalent Lucene query syntax and feed it into a query_string query. We do support some text-related OData...

Fast indexing using multiple ES nodes?

All I read and understand about running multiple ES nodes is to enable index replication and scaling. I was wondering if it could help us to make indexing faster for large number of files. I have two questions and they are as follows: Question 1: Would it be accurate to...

Pylucene 4.9.0 Ubuntu 14.04 Installation ImportError

I've been trying to install Pylucene on my Mac for a little over a week, and have given up on that in favor of trying to install it with Ubuntu through a virtual machine. I thought the installation process had gone well, so I fired up Python in the terminal...

In ActiveMQ clustering, how to find the consumer who picked a particular message or set of messages?

We have set of queues and camel routes have been written to consume messages from the queue - queues live in an ActiveMQ server and camel routes in Apache Karaf. We have put up a cluster with multiple karaf nodes; problem is that these messages get read by camel routes...

How to combine neo4j and elasticsearch

I am developing a Question answering application and for that I need to use neo4j and elasticsearch in the same maven project. I am using elasticsearch to make my application more robust. As we know that neo4j and elasticsearch works on different version of lucene, so whichever version I include...

Lucene doesn't search number fields

I'm trying to index and then search integer field with lucene. But it doesn't find anything (Text fields search well). Document doc = new Document(); //UserType = 1 doc.add(new IntField("userType", user.getType().getId(), Field.Store.YES)); FSDirectory dir =; IndexWriterConfig config = new IndexWriterConfig(new StandardAnalyzer()); writer = new IndexWriter(dir, config); writer.addDocument(doc); For search...

What's a Cluster / Bucket in couchbase Server

I'm new to Couchbase and NoSql technologies in general, but I'm working on a web chat application running on node js using express and some other modules. I've chosen to work with NoSql to store sessions and all needed data on server-side. But I don't really understand some important features...

Minimum number of config servers needed for Mongo Cluster

I am trying to test sharding in MongoDB cluster. I am folllowing MongoDB documentation . They suggest to use 3 config servers. Can I set up a cluster with 1 or 2 servers? Or I need 3 or more config servers?...

How to check if document exists in lucene index?

I have an index of news articles, where i save title,link,description of news.. sometimes its possible that same news from same link is published with different titles by different news sources. it don't want exactly same description articles to be added to find if document already exists?

Search for nodes in Neo4j with schema index

I have a graph that has only Schema indexes and not legacy indexes as Neo4j documentation recommends. I want to search for nodes like in this example described under the legacy indexing section (exact match, start queries etc). I am wondering if this is possible with schema indexes and if...

Editing /etc/hosts for MPI cluster

I struggled trying to set up an MPI cluster, following Setting Up an MPICH2 Cluster in Ubuntu tutorial. However, I tangled things up and it did not work, so I undid all the changes (except of the passphrase in step 7, which I have no clue how to undo) and...

HornetQ using JNDI / remoting when used with WildFly 8.1 cluster

Previously 2 JBoss 6 servers (master & slave) were started with 2 different IP addresses ( and and those 2 server instances were run in domain mode. The instances used HornetQ to communicate index jobs for Hibernate Search. I'm trying to create a similar configuration using Wildfly instead of...

Lucene how to index in Database (Cassandra)

I am just experimenting with Lucene and want to indexing objects in Database(Cassandra) as a table. But, I didnt realized out, how the indexing does work on Cassandra. Especially searching... When i take a simple Example Indexing in Lucene: Document doc = new Document(); doc.add(new TextField("id", "Hotel-1345", Field.Store.YES)); doc.add(new TextField("description",...

Special Characters that can't be indexed using lucene

Can some one tell me which are the special characters that cannot be indexed using Apache lucene ?

Elasticsearch boost per field with function score

I have a query with different query data for different fields and ORed results. I also want to favor hits with certain fields. Ideally this would only increase ranking but would not cause results that did not contain some of the terms in the other fields. This would skew results...

Solr custom UpdateRequestProcessorFactory fails with “Error Instantiating UpdateRequestProcessorFactory”

I have a custom class extending UpdateRequestProcessorFactory doing some work on a document when it gets added to the index. This was working fine in v4.10.3 in standalone Solr. I moved to SolrCloud v5.2 and it throws this error when adding the Collection (node): ERROR - 2015-06-14 12:25:11.071; [ docs_shard1_replica1]...

Using FrenchAnalyzer with Neo4J

I am trying to use the Lucene FrenchAnalyzer with Neo4J: final GraphDatabaseService graphDatabaseService = new GraphDatabaseFactory().newEmbeddedDatabase("..."); final IndexManager index = graphDatabaseService.index(); final Index<Node> frenchIndex = index.forNodes("Entry", stringMap(IndexManager.PROVIDER, "lucene", "type", "fulltext", "to_lower_case", "true", "analyzer","" )); but this throws java.lang.NoSuchMethodException:...

Solrcloud multicore configuration

I have a standalone Solr instance with 4 different cores working fine using the embedded Jetty server. I configured the cores for v4.10.3 but since I moved to v5.1 and all seems to work fine without any changes. Before going into production, I need to set it up as a...

Predis: Pros and Cons of the two cluster strategies

I am using Predis to connect to a Redis 3.0.2 cluster. So, I have an option to use either predis or redis cluster strategy. I am wondering if there are any significant pros and cons of choosing one of them over the other? Based on my wandering through documentation, I...

How can I query lucene based on a lucene search result?

Here's the problem I'm trying to solve: I have multiple lucene indices, each containing a subset of the same data structure (they have the same fields, but the fields may or may not be present in a document in a certain index) There is a global identifier that is shared...

solrException. XML parser doesn't support XInclude option

After configuring solr4.7.2 with tomcat 7, got the error in solrAdmin page stating SolrCore Initialization Failures fran92:org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: XML parser doesn't support XInclude option My solr.xml file contains one core <?xml version="1.0" encoding="UTF-8" ?> <solr persistent="true"> <cores host="${host:}" adminPath="/admin/cores" hostContext="${hostContext:solr}"> <core config="solrconfig.xml" name="fran92" instanceDir="generic" schema="schema.xml"...

How to resolve NoSuchFieldError exception when testing Lucene 4.0

I want to test my own Analyzer. Following is test code from Lucene in Action 2nd Edition, Code List 4.2, page 121. public class AnalyzerUtils { public static void displayTokens(Analyzer analyzer, String text) throws IOException { TokenStream tokenStream = analyzer.tokenStream("contents", new StringReader(text)); displayTokens(tokenStream); } public static void displayTokens(TokenStream stream) throws...