mongodb,heroku,express,elasticsearch,bonsai-elasticsearch , Bonsai automatic indexing


Bonsai automatic indexing

Question:

Tag: mongodb,heroku,express,elasticsearch,bonsai-elasticsearch

I use MongoDb for storage with an express, Heroku-hosted app.

I would like to provide search capabilities to some MongoDb collections, using ElasticSearch via the Bonsai Heroku add-on.

However, according to the Bonsai faq, Elasticsearch River plugins are not supported.

How can I automatically re-index my MongoDb collections (when a MongoDb document is created/updated/deleted) on Bonsai, using custom mapping, if I cannot use elasticsearch-river-mongodb?


Answer:

Here is the answer by Bonsai support:

- You could always set up a script with a curl command to index the MongoDB collections, then use cron to run the script at regular intervals.
- You could also use an Elasticsearch client to index collections in some cases.

So I made a reindex script inspired by Elaster, which can:


Related:


paging subdocument in mongodb subdocument


mongodb,mongodb-query
I want to paging my data in Mongodb. I use slice operator but can not paging my data. I wish to bring my row but can not paging in this row. I want to return only 2 rows of data source. How can resolve it My Query : db.getCollection('forms').find({ "_id":...

Brand new Luminus app giving Error: Could not find or load main class clojure.main


heroku,clojure,procfile,luminus
I just created a Luminus app by running: lein new luminus foobar and when I try to run it with foreman like this: foreman start the way the docs describe, I get this error: Error: Could not find or load main class clojure.main which is also the same error I...

Why the query scans 8x times the documents it finds - MongoDB


mongodb
I've got 50 documents (for trial purposes) more or less like this one: "_id" : ObjectId("5461c8f0426f727f16010000"), "f" : [ { "t" : "ry", "v" : 1972 }, { "t" : "g", "v" : [ "Crime", "Drama" ] }, { "t" : "ml", "v" : "English" }, { "t" : "k",...

How could I get the matched nested items in array


mongodb
How could I get the matched nested items in array I want to return the matched items in nested array. For example, I want to filter out the records that contain "A428 ","A429 " in their items How could I get it ? Query pipeline_work = [ { '$match': 'records.items':...

using MongoDB aggregate count subdata


mongodb,aggregate
here is my data in mongodb: { "data": { "order_goods": [{ "category": 235 }, { "category": 666 }] } }, { "data": { order_goods: [{ "category": 235 }] } } here is my expected output: {"category":235, "total":2} {"category":666, "total":1} I have try many ways about aggregate such as $group, but...

How to overwrite object Id's in Mongo db while creating an App in Sails


mongodb,sails-mongo
I am new to Sails and Mongo Db. Currently I am trying to implement a CRUD Function using Sails where I want to save user details in Mongo db.In the model I have the following attributes "id":{ type:'Integer', min:100, autoincrement:true }, attributes: { name:{ type:'String', required:true, unique:true }, email_id:{ type:'EMAIL',...

How to check if multiple documents exist


mongodb,mongodb-query
Is there such a query that gets multiple fields, and returns which of these exists in the collection? For example, if the collection has only: {id : 1} {id : 2} And I want to know which of [{id : 1} , {id : 3}] exists in it, then the...

AngularJS factory dependencies


javascript,html,angularjs,mongodb
I'm using AngularJS to extract information stored in mongodb. I'm trying to use a factory to retrieve that information using $http . I read so much information about how to do it, and no one works for me. Also I'm using node + express, the routes works fine. The problem...

mongodb populate method not working


node.js,mongodb,model,populate,auto-populate
Here is my code for models var postSchema = new mongoose.Schema({ created_by: {type: Schema.ObjectId, ref:'User', autopopulate: true }, //should be changed to ObjectId, ref "User" created_at: {type: Date, default: Date.now}, text: String }); var userSchema = new mongoose.Schema({ username: String, password: String, //hash created from password created_at: {type: Date, default:...

How to download files from git repository?


git,heroku
So I have few .mp4 files in my heroku repository. And I want to download them to my local repositroy. I'm doing it like this: git pull heroku master But it doesn't download the .mp4 files. How can I get those files?...

How to get node.js to connect to mongolab using mongoose


database,node.js,mongodb,mongoose,mongolab
I've been trying to use mongoose (module for node.js and mongodb). And tried to get a connection with mongolab up and running. I tried the following at the top of my app.js file, but I couldn't seem to enter the db.on function. global.mongoose = require('mongoose'); var uri = 'mongodb://username:[email protected]:#####/db'; global.db...

Cassandra data model to store embedded documents


mongodb,database-design,cassandra
In mongodb we can able to store embedded documents into a collection.Then, How do we store embedded documents into cassandra??? For this sample JSON representation??? UserProfile = { name: "user profile", Dave Jones: { email: {name: "email", value: "[email protected]", timestamp: 125555555}, userName: {name: "userName", value: "Dave", timestamp: 125555555} }, Paul...

Are the changes done by db command in MongoDB permanent and how?


mongodb,mongodb-query
Are the changes done using the shell in mongodb permanent ? And how MongoDb detects the config file? I am stuck in setting Slow query logs in MongoDb. We have a command db.setProfilingLevel() to set profiling level. But how can we set profiling for all databases and how to set...

Deploying a tar file from CI to heroku


heroku,playframework
I have a play application for my website. Currently, codeship builds it, and heroku deploys it once the build succeeds. The problem is that since the project has become too big, heroku is not able to build it within 15 minutes while codeship can still build it in less than...

Not able to create staging instance on heroku [duplicate]


ruby-on-rails,git,heroku,sqlite3
This question already has an answer here: Errors of pushing rails app to Heroku error occurred while installing sqlite3, and Bundler cannot continue 3 answers I am trying to create a staging instance of my app on heroku. While I am doing git push staging master I am getting...

mongodb aggregate unwind array and no array


arrays,mongodb,aggregation-framework
I'm trying to write an aggregate query using $unwind no matter the element is an array or not. I know $unwind does not work on no array elements, but I wonder if there is a way to make it work, like converting this element into an array. I have a...

What are some patterns I can look at for database implementations in JavaScript?


javascript,node.js,mongodb
I'm fairly new to JavaScript, and I'm busy playing around with a node test app and MongoDB. I'm at a point where I'd like to start with the db side of the app, but I'm not sure what patterns are most commonly used in such a stack, and more importantly,...

Use JSON file to insert data in database


javascript,json,mongodb,meteor,data
I'm using my JSON file like this to insert data in my collection : var content = JSON.parse(Assets.getText('test.json')); console.log('inserting...'); Profiles.insert({ user: id, data:content }; But I would like to have a "data's tree" like that : [ user: "rtegert23423131", firstname:"test", surname:"test2", // ... ] Not like that : [ user:...

Async await usage for MongoDB repository


c#,mongodb,asynchronous,parallel-processing,async-await
I have a MongoDB repository class as you see below: public class MongoDbRepository<TEntity> : IRepository<TEntity> where TEntity : EntityBase { private IMongoClient client; private IMongoDatabase database; private IMongoCollection<TEntity> collection; public MongoDbRepository() { client = new MongoClient(); database = client.GetDatabase("Test"); collection = database.GetCollection<TEntity>(typeof(TEntity).Name); } public async Task Insert(TEntity entity) { if...

Incorrect response to mapReduce query in mongo-db


mongodb,mapreduce
I have 1000 user records in collecton, in which 459 document has gender male and remaining as female //document structure > db.user_details.find().pretty() { "_id" : ObjectId("557e610d626754910f0974a4"), "id" : 0, "name" : "Leanne Flinn", "email" : "[email protected]", "work" : "Unilogic", "dob" : "Fri Jun 11 1965 20:50:58 GMT+0530 (IST)", "age" :...

After deploying to heroky scripts and css not available


node.js,heroku
I am new in heroku. Locally project works fine but after deploying vendor scripts files are not available. Their paths are redirected to main page. Here is the deployed version with errors https://salty-woodland-8424.herokuapp.com/. In nodeJs file i use var express = require('express'); var path = require('path'); var stylus = require('stylus');...

Rails, DNSimple, Heroku and SSL - do I need a certificate?


ruby-on-rails,ssl,heroku,dnsimple
So I'm currently deploying my app via Heroku. I noticed that in my-app-name.herokuapp.com has HTTPS, so if I do config.force_ssl = true in my environments/production.rb it seems like I have wildcare SSL, right? Now I'm using DNSimple to get my actual name - call it my-app-name.com. Which currently resolves to...

How to add a subdocument in a mongoose schema


javascript,node.js,mongodb,mongoose
I am trying to create a subdocument in a mongoose schema from node.js/Express. I have two schemas: Member and Address Member.js // app/models/member.js // load mongoose since we need it to define a model var mongoose = require('mongoose'), Schema = mongoose.Schema var Address = require('./address'); var MemberSchema = Schema({ FName...

Find Mongoid geospacial circles that contain a point


mongodb,mongoid,geospatial
I have Mongoid documents that represent services offered by local merchants. Each has one or more locations (lat/lng point) with a service area (radius). Given a customer location (lat/lng point), how can I find all the documents where the customer location falls within the service area? class Service include Mongoid::Document...

Get json values from key-value pair without passing keys in meteor


javascript,json,mongodb,meteor
i am using keys to get the values from json in meteor. My json is in this format. { "_id" : "SXTJBs7QLXoyMFGpK", "Brand" : { "value" : "Nike" }, "Material" : { "value" : "Cooton" }, "Price" : { "value" : "67484" }, "ComboId" : { "value" : "y23" },...

Redirect Loop on Heroku with Rails 4 App, but not on local machine


ruby-on-rails,ruby-on-rails-4,heroku,dns
App running Ruby 2.2.2, Rails 4.2.1 When I run my app locally everything is fine. It was running fine on heroku until I made some updates. Now all I get in a browser is "Err too many redirects" from Chrome, or "Too many redirects..." in Safari. Even after I rolled...

Is there a built-in function to get all unique values in an array field, across all records?


arrays,node.js,mongodb,mongoose,schema
My schema looks like this: var ArticleSchema = new Schema({ ... category: [{ type: String, default: ['general'] }], ... }); I want to parse through all records and find all unique values for this field across all records. This will be sent to the front-end via being called by service...

Heroku rake db:migrate failing - uninitialized constant


ruby-on-rails,ruby,heroku
My app is working fine locally and my push to Heroku was successful. But, when I run heroku run rake db:migrate, I get the following error: NameError: uninitialized constant AddWeightToExercises Here is the failed migration: class AddWeightToExercise < ActiveRecord::Migration def change add_column :exercises, :weight, :float end end edit: Thanks for...

Getting failed to load c++ bson extension error using Mongodb and Node.js


javascript,node.js,mongodb
I am getting the following error while trying to run server using Node.ja with Mongodb. Error: { [Error: Cannot find module '../build/Release/bson'] code: 'MODULE_NOT_FOUND' } js-bson: Failed to load c++ bson extension, using pure JS version { [Error: Cannot find module '../build/Release/bson'] code: 'MODULE_NOT_FOUND' } js-bson: Failed to load c++...

How to use a variable as an Object Key [MongoDB] [duplicate]


node.js,mongodb
This question already has an answer here: How to use a variable as a field name in mongodb-native findOne()? 1 answer I'm trying to use variable name given to me by my function, but its actually setting the object key equal to 'name' function addlist(name, item, id){ // Add...

Inserting a variable in MongoDB specifying _id field


python,mongodb,pymongo
I want to insert a variable, say, a = {1:2,3:4} into my database with a particular id "56". It is very clear from the docs that I can do the following: db.testcol.insert({"_id": "56", 1:2, 3:4}) However, I cannot figure out any way to insert "a" itself, specifying an id. In...

Heroku RAM not increasing with upgraded dynos


ruby-on-rails,ruby,ruby-on-rails-3,memory,heroku
I have a massive function i have been calling manually through the heroku rails console. I have been receiving the error rapid fire in my logs: 2015-06-22T14:56:42.940517+00:00 heroku[run.9877]: Process running mem=575M(112.4%) 2015-06-22T14:56:42.940517+00:00 heroku[run.9877]: Error R14 (Memory quota exceeded) A 1X dyno is suppose to have 512 MB of RAM. I...

Compare Array with Collection-Array containing Objects


node.js,mongodb,mongoose,schema
This is my collection schema: var objectSchema = new Schema({ members: [{ user_id: ObjectId, settings: { type: Boolean } }], title: String }); And now I'm trying to search for objects with specific members (identified by their "user_id", for example ["asdf123lkd", "asdf1223"]). Is there any way to search for these...

Query with filter builder on nested array using MongoDB C# driver


c#,mongodb,mongodb-query,mongodb-csharp,mongodb-csharp-2.0
Consider the following object structure stored as documents: public class Foo { public string Id { get; set; } public ICollection<FooBar> Bars { get; set; } // ... } public class FooBar { public string BarId { get; set; } // ... } Using a LINQ-style query with the driver...

Use $or operator in @Query annotation in spring data mongodb repository


java,spring,mongodb,spring-data-mongodb
I am using spring-data-mongodb. I want to use $or operator in my repository. This is my query: @Query("{'type':?0}") List<Doc> findByType(String type, Pageable pageable); How do use $or in @Query so as it can match either type or name and fetch me a document. Please help....

Mongoose population in instance methods


node.js,mongodb,mongoose,mongoose-populate
I have a model with reference to other documents. I would like to have a method in that model that can process data used in the referenced models. 'use strict'; var mongoose = require('mongoose') , Schema = mongoose.Schema , deepPopulate = require('mongoose-deep-populate'); var MainSchema = new Schema({ childs: [{type:Schema.ObjectId, ref:...

Do you get the same performance using index prefixes?


performance,mongodb,indexing
Say I have a collection containing documents like the one below: { _id: ObjectId(), myValue: 123, otherValue: 456 } I then create like below: {myValue: 1, otherValue: 1} If I execute the following query: db.myCollection.find({myValue: 123}) will I get the same performance with my index as I would if I...

Data not visible in mongoDB when inserted from PyMongo


python,mongodb,pymongo
I am trying to insert data in a mongodb collection from python but the data is not being logged. This is how I am doing it: from pymongo import MongoClient import time class data2db: def __init__(self): pass def enter_data(self,data): client = MongoClient('127.0.0.1', 27017) db = client.db coll=db.Temperature1 post = {"auth":...

How to add new items to an array in MongoDB


arrays,node.js,mongodb
I'm trying to add a new item to whichever name that was passed in under whichever id. My first problem is that it seems like its not grabbing the values from any of my variables (name, item, id), instead just using them as object keys. My next issue is that...

MongoJS - No Error On Unique Index


mongodb,mongojs
So I'm messing around with MongoDB using MongoJS and can't seem to get an error to be returned when a unique ID exists. So this is the code I am using: server.post('/register', function (req, res, next) { var details = req.params; details.password = md5(details.password).toString(); db.users.insert(details, function (err, test1, test2) {...

Creating index while updating the documents


mongodb,indexing
I have a collection I am updating adding a new field. The document looks like: {"A": "P145", "B":"adf", "C":[{"df":"14", "color":"blue"},{"df":17}], "_id":ObjectID(....), "Synonyms":{"Synonym1": "value1", "Synonym2": ["value1", "value2"]}} In the update I am adding new elements to C I want to create a index on the field A and B. A and...

Get matching elements in mongodb array


php,mongodb,aggregation,pipeline
I want to use aggregation to get this array only with those tickets, which have start field after 2015-06-16. Can someone help me with the pipeline? { "name" : "array", "tickets" : [ { "id" : 1, "sort" : true, "start" : ISODate("2015-06-15T22:00:00.000Z") }, { "id" : 2, "sort" :...

Devise emails are not being sent using sendgrid while other emails are being sent well


ruby-on-rails-4,heroku,devise,sendgrid
I am using sendgrid to send emails in production my all other emails are being sent as expected but all the devise emails are not being sent neither a crash occur my in my production.rb I have config.action_mailer.raise_delivery_errors = false config.action_mailer.default_url_options = { :host => 'myapp.herokuapp.com' } config.action_mailer.delivery_method = :smtp...

Meteor/MongoDB limiting the result


mongodb,meteor
I am trying to find all documents and publish at most 5 from the results. Following this section of the MongoDB doc, I am trying to do this: Meteor.publish('teams', function () { return Teams.find().limit(5); }); Yet, in the server console, I get an exception: Exception from sub teams id Pm6jKL8Sv3FSDSTfM...

sqlite issue with heroku


ruby-on-rails,sqlite,heroku
I was using sqlite gem for my rails project, and since I had to put the app on Heroku, and Heroku doesn't support sqlite, I added pg gem to my Gemfile. gem 'pg', '0.17.1' But now I am not able to run the project locally on my machine. Specified 'sqlite3'...

push item in sub document


mongodb,mongodb-query
I have simple collection. I want to push item in question, but it gives an exception.How can I resolve it ? /* 1 */ my collection data { "_id" : ObjectId("557e8c93a6df1a22041e0879"), "QuestionCount" : 2.0000000000000000, "Questions" : [ { "_id" : ObjectId("557e8c9ba6df1a22041e087a"), "DataSource" : [], "DataSourceItemCount" : NumberLong(0) }, { "_id"...

Mongoose : update collection with previous data


node.js,mongodb,mongoose
I am learning NodeJs and for a demo project i need to use MonGoDB with Mongoose. I created a collection for a photo album. The first row is : albumName and the second row is : pictures. I don't undersand how i can save all my pictures in the row...

MongoDB Java Driver 3.0 MapReduce


mongodb,mongodb-java
This is the code I'm using to run map reduce on sourceCollectionName and to get the output to targetCollectionName. But the targetCollectionName is never created. new MongoClient("localhost").getDatabase(dbName).getCollection(sourceCollectionName) .mapReduce(map, reduce) .action(MapReduceAction.REPLACE) .databaseName(dbName) .collectionName(targetCollectionName) .sharded(false); Although I'm able to get output as MapReduceIterable and when I iterate this the result is dumped...

Query not working - MongoDB


mongodb
Given this kind of documents: "_id" : ObjectId("5461c8f0426f727f16010000"), "f" : [ { "t" : "ry", "v" : 1972 }, { "t" : "g", "v" : [ "Crime", "Drama" ] }, { "t" : "ml", "v" : "English" }, { "t" : "k", "v" : "movie" }, { "t" : "ai",...

Query an array of embedded documents in mongodb


mongodb,embedded-documents
I'm having a little trouble writing a query that needs to compare a given value against a certain field in all embedded documents within an array. I will give an example to make the issue less abstract. Let's say I want to use MongoDB to store the last queries that...