FAQ Database Discussion Community


Data not visible in mongoDB when inserted from PyMongo

python,mongodb,pymongo
I am trying to insert data in a mongodb collection from python but the data is not being logged. This is how I am doing it: from pymongo import MongoClient import time class data2db: def __init__(self): pass def enter_data(self,data): client = MongoClient('127.0.0.1', 27017) db = client.db coll=db.Temperature1 post = {"auth":...

how do I write a function for connecting to mongodb using pymongo

python,mongodb,pymongo
I'm trying to write a function to connect to mongodb and return a collection object. I have the following: def getCollection(dbname,collection): client = MongoClient() data_base = client.dbname collObject = data_base.collection return collObject When I run: collection = getCollection(client, "hkpr_restore", "agents") print collection I get: Collection(Database(MongoClient('localhost', 27017), u'dbname'), u'collection') What am...

how to query using isodate in pymongo

python,pymongo
I have a mongodb query that works in the mongo shell: db.collection.find({ created_at: { $gte : ISODate("2015-03-01T00:00:00.000Z"), $lt : ISODate("2015-03-30T00:00:00.00Z") } }) I'm trying the following pymongo: test_range = agents.coll.find({ "created_at" : { "$gte" : "ISODate('2015-03-01T00:00:00.000Z')", "$lt" : "ISODate('2015-03-30T00:00:00.00Z')" } }) and I'm getting SyntaxError: invalid syntax What the correct...

Performing sentiment analysis on a mongodb collection containing JSON elements (tweets) in Python

python-2.7,pymongo,textblob
Hi I have created a python script using tweepy to stream tweets based on a keyword array into a mongodb collection based on the name of the element in the array that it was filtered by via pymongo ie (apple tweets saved to an apple collection). This script saves them...

How to specify a unsafe/safe write with pymongo's collection.update_one or update_many

python,mongodb,pymongo
I have the connection defaulted to w=0 but for collection.update_one or collection.update_many, I want to set the write_concern per operation by setting the parameter w=0. Instead I'm getting this error: update_one() got an unexpected keyword argument 'w' What is the right way to do this? I saw that insert accepts...

ValueError: dictionary update sequence element #0 has length 3; 2 is required

json,mongodb,python-3.x,pymongo
I am trying to insert values in MongoDB, but I am getting this error: ValueError: dictionary update sequence element #0 has length 3; 2 is required. From pymongo import MongoClient client = MongoClient() db = client.abc_database keys = [] values = [] key = input("enter keys:").split(",") keys.append(key) print(keys) print(keys[0][1]) value...

Error in mongo query execution through python when queries and collection name are varying

python,mongodb,pymongo
I was trying to connect to Mongo collection where each time collection and query is different.I'm trying to write a function where user will pass collection name and query string and user will get cursor as a result. But this function is not connecting to collection passed in variable but...

Get data of Referenced objects in MongoEngine query, not just id

python,mongodb,pymongo,mongoengine
I have a set of models as below: import mongoengine as mongo class Post(mongo.DynamicDocument): text = mongo.StringField() class User(mongo.DynamicDocument): name = mongo.StringField(required=True) posts = mongo.ListField(mongo.ReferenceField(Post)) When I query for a User, say id = 1 user = User.objects.get(id=1) print(user.to_json()) I get the payload as such: { '_id': {'$oid': "1"}, 'name':...

how to get full list of keys from a collection in mongodb

python,mongodb,pymongo
How do I get a list of all the keys used in a mongodb collection? I'm trying to create a csv file in python using the following: userAgentProp_list = [] # userAgentProp is a set of user ids # userColl is a mongodb collection for user in userAgentProp: userAgentProp_list.append(userColl.find_one({"_id":user})) with...

Using the $cond statement with $project and aggregate in PyMongo

python,mongodb,pymongo
I want to project a new field based on a conditional logic statement, using pymongo. The value should equal 1 if the 'status' field is either 'successful_ended' or 'successful_ongoing'. I've tried implementing this by using $in within a $cond statement. A simplified version of my aggregate statement is like so:...

PyMongo: Should I use single or multiple clients?

mongodb,pymongo
The question is simple: should I keep all operations to a single MongoClient? Is single client or multiple clients better than the other?

Abstract classes and PyMongo; can't instantiate abstract class

python,python-3.x,abstract-class,pymongo
I created the empty abstract class AbstractStorage and inherited the Storage class from it: import abc import pymongo as mongo host = mongo.MongoClient() print(host.alive()) # True class AbstractStorage(metaclass=abc.ABCMeta): pass class Storage(AbstractStorage): dbh = host def __init__(self): print('__init__') Storage() I expected the output to be True __init__ however, the one I'm...

How to get class of pymongo database suitable for isinstance()?

python,mongodb,pymongo
I have a test where I'm trying to identify if I've successfully gotten an instance of a database using pymongo, and would like to use isinstance(obj, class) in an assertion. However I cant figure out how to get a class (not instance) of the database I've tried several approaches. This...

Writing Data Into CSV Format File

python,mongodb,csv,pymongo,export-to-csv
I'm trying to write some data into CSV format using fetched data from MongoDB, with Mongopy. Currently, the headers are currently set, and the problem is that the actual data is not being inserted into the CSV file. here is the code snippet : from pymongo import MongoClient import csv...

Finding number of inserted documents in a bulk insert with duplicate keys

mongodb,pymongo,tornado-motor
I'm doing a bulk-insert into a mongodb database. I know that 99% of the records inserted will fail because of a duplicate key error. I would like to print after the insert how many new records were inserted into the database. All this is being done in python through the...

Dynamically add lists as embedded document in Python pymongo mongodb

python,mongodb,pymongo
Let’s say this represents a complex document I need to insert: customer = { 'first_name' : 'Arnold', 'last_name' : 'Pettibone', 'addresses': [ 'home' : { 'street' : '1234 fake street', 'city' : 'Anytown', 'state' : 'OH', 'zip' : '12345' }, 'work' : { 'street' : '742 Evergreen Terrace', 'city' :...

using the mongodb entry while updating

mongodb,pymongo
I have a mongdb structure which looks like this: {'_id':1, 'aggregate_sum': 12, 'total':10, 'mean':1.2} I want to update this database's mean. num =12 value = db.find_one({'_id':1}) new_mean = (value['mean']*value['total'] + num )/(value['total']+1) db.update({'_id':1, {'$inc':{'aggregate_sum':num, 'total':1},{"$set" :{'mean':new_mean}) Can I prevent the first call to mongo and instead update the mean on...

How to remove duplicate values inside a list in mongodb

mongodb,mongodb-query,pymongo,aggregation-framework
I have a mongodb collection . When I do. db.bill.find({}) I get, { "_id" : ObjectId("55695ea145e8a960bef8b87a"), "name" : "ABC. Net", "code" : "1-98tfv", "abbreviation" : "ABC", "bill_codes" : [ 190215, 44124, 190215, 147708 ], "customer_name" : "abc" } I need an operation to remove the duplicate values from the bill_codes....

Saving MongoDB cursor between HTTP requests in Flask

mongodb,python-2.7,flask,pymongo
I am writing a webapp to display tweets from a MongoDB database using pymongo (regular pymongo) and flask. The problem that I have is that when accessing paginated results, the search is being performed again and again, when all I really need to do is iterate over cursor. Page 1:...

pymongo update_one syntax error

python,mongodb,pymongo
have a collection in mongodb called 'collection' and I need to do and upsert using pymongo. collection.update_one({'_id': workflowID}, { '$set': {'repop': repop}, {'data': workflow__list()__} }, upsert = True, multi = False) but I get 'SyntaxError: invalid syntax' with the little up arrow pointing under the 'o' in the repop object...

Why am I getting TypeError: 'module' object is not callable in python?

python,mongodb,python-2.7,pymongo
I'm creating a string that I an then use in a method that queries a mongodb collection. Eventually the dates will be from user input. Here's the relevant code and string: import pymongo from pymongo import MongoClient from datetime import datetime import time import datetime start_yr = 2015 start_mnth =...

Update all the values of the json document to default value using pymongo

python,json,mongodb,pymongo
I have a JSON document in the following format, { "_id": ObjectId("54a2462820fb5b6068b45b05"), "Ref": 1, "a": { "b": "value1", "c": "value2", "d": { "e": "value3" } } } I need to update values of all the keys even nested (Ref,b,c,e) except _id to some default value say 'na' at once. I...

How to check that mongo ObjectID is valid in python?

python,mongodb,pymongo
I want to verify that the objectID is a valid mongoID string. Currently I have: import bson try: bson.objectid.ObjectId(id) except: pass # do something I wanted to make my exception more specific, and it looks like there is a solution, but except bson.objectid.InvalidId ends up in TypeError: id must be...

Does MongoClient work on pymongo2.3?

python,mongodb,pymongo
I have a very straight forward question. I am trying to connect to MongoDB using pymongo 2.3. I get the following error: ImportError: cannot import name MongoClient However import pymongo works fine and pymongo.version yields '2.3'. And python version is 2.7.3....

Updating an object inside an array with PyMongo

python,mongodb,pymongo
I am wondering how do you update a nested array with PyMongo/MongoDB by selecting a document(row) and then going into the nested array and selecting a specific object. { "_id" : "12345", "name" : "John Doe, "mylist" : [ { "nested_id" : "1", "data1" : "lorem ipsum", "data2" : "stackoverflow",...

Getting rid of _id in mongodb collection

mongodb,mongodb-query,pymongo
I know it is not possible to remove the _id field in a mongodb collection. However, the size of my collections is large, that the index on the _id field prevents me from loading the other indices in the RAM. My machine has 125GB of RAM and my collection stats...

Logical operators in mongodb queries with python

python,mongodb,python-2.7,pymongo
I am trying to query my mongo db for a item in python2.7 with output = collection.find_one({ $and : [{'name' : data['name']},{'phone_1' : data['phone_1']}]}) when I try to run the script python tell me File "./test.py", line 113 output = collection.find_one({ $and : [{'name' : data['name']},{'phone_1' : data['phone_1']}]}) ^ SyntaxError:...

Does long or int in pymongo queries have to match the number type in mongodb?

mongodb,python-3.x,pymongo
I can't find much of an indication in neither the pymongo documentation or the mongodb documentation. I'm currently working with python3, and I have one program inserting data and another program that queries it. Basically, if one program inserts data as a BSON long type (64 bit), can another program...

Inserting a variable in MongoDB specifying _id field

python,mongodb,pymongo
I want to insert a variable, say, a = {1:2,3:4} into my database with a particular id "56". It is very clear from the docs that I can do the following: db.testcol.insert({"_id": "56", 1:2, 3:4}) However, I cannot figure out any way to insert "a" itself, specifying an id. In...

Do file size requirements change when importing a CSV file to MongoDB?

python,mongodb,csv,python-3.x,pymongo
Background: I'm attempting to follow a tutorial in which I'm importing a CSV file that's approximately 324MB to MongoLab's sandbox plan (capped at 500MB), via pymongo in Python 3.4. The file holds ~ 770,000 records, and after inserting ~ 164,000 I hit my quota and received: raise OperationFailure(error.get("errmsg"), error.get("code"), error)...

update mongodb on fly

python,mongodb,pymongo
I have a mongodb database which stores values like this: {'_id':1, "summary":{"tags": {"Good":1, "bad": 4}}} I want to make it such that I can enter any new tag and update the count of existing tags. so like: db = pymongo.MongoClient().collection.summary tags = ['Good', 'Enjoy'] id_tags =db.find_one({'_id':1})[summary][tags].keys() for i in tags:...

Pymongo issue in a python project

python,pymongo
I have installed pymongo, and when I start a new project and I write import pymongo Python shows me the next error. Python version: 3.4.3 Error code Traceback (most recent call last): File "TestMongoDB.py", line 3, in <module> import pymongo ImportError: No module named 'pymongo' Pymongo If I put this...

Mongo Bounding Box Query with Limit

python,mongodb,pymongo
We are using mongo (via pymongo) to store a database of points in our system. This data is returned via our api using bounding box queries ($geoWithin). We want to limit the number of results returned to 200 sorted from the center. I'm trying to figure out the best way...

pymongo sorting by date

python,mongodb,pymongo
I want to get newest posts first, and try the following: db.posts.find({"date": {"$lt": tomorrow, "$gte": today}}).sort({'date':pymongo.DESCENDING}) (without sort, I get the oldest posts first fine) I am getting this error TypeError: if no direction is specified, key_or_list must be an instance of list What is going on here? Is not...

How to kill dead cursors in MongoDB

mongodb,cursor,timeout,pymongo
When querying a MongoDB collection, if the read will take a while, the cursor times out. There's a timeout=False option, which makes the cursor not time out, and then it must be closed manually. When a script crashes with an open cursor, that cursor presumably just stays alive in the...

MongoReplicaSetClient,which uri should I connect to?

mongodb,pymongo,database-replication
Let's say I have a replcaset of 3 machines(M1, M2, M3), and serveral other machines for hosting servers(S1, S2, ..Sn). I don't care much about real-time so ReadPreference will be set to NEAREST. There's one thing that bothers me: As the doc says, the only required parameter of MongoReplicaSetClient is...

Store images in MongoDB

python,mongodb,pymongo,gridfs
I'm using PyMongo to connect to my database, when a user uploads an image I want to store it using GridFS within the users specific document in the collection. I'm doing it as so: class uploadHandler(BaseHandler): @tornado.web.authenticated def get(self): self.render("upload.html",user=self.current_user) def post(self): db = pymongo.Connection('mongodb://heroku_app.mongolab.com:/heroku_app').heroku_appxxxx user = db.userInfo.find_one({'Username':self.current_user}) file1 =...

Pymongo 3.0.2 slower than 2.8.1

python,performance,mongodb,pymongo
I am running a simple Django app on top of MongoDB, and recently upgraded to pymongo 3.0.2 -- but this runs incredibly slowly. If I downgrade to pymongo 2.8.1 or 2.7.2, it speeds back up again. This happens with both MongoDB 3 and 2.6, so I'm thinking that something fundamental...

How to get distinct values from PyMongo

mongodb,python-2.7,pymongo
In MongoDB I have a dataset of store data. Using PyMongo I am finding all of the distinct/unique values in a collection for testy in collection.distinct('stores'): print(testy) I can also find a subset of badly spelt stores I'm interested in for testy in collection.find({'stores': {'$in': ['Aldi','ALDI','aldi']}}): What I want to...

Return certain fields in mongodb

python,mongodb,pymongo
I am trying to return just the arrays that meet my criteria. Here is what i have: { "_id": 1, "awardAmount": 20000, "url": "www.url.com", "numAwards": 2, "award": "Faculty Research Grant", "Type": "faculty", "Applicants": [ { "preAwards": "NO1", "Name": "Omar1", "School": "SCSU1", "citizenship": "YES1", "budget": 1, "Advisor": "Dr. DaPonte1", "Major": "CSC1",...

Unable to serialize JSON object from MongoDB in python

python,json,mongodb,pymongo
I have the following JSON stored in my MongoDB: { "_id" : ObjectId("54fed786265e7f01d66ca778"), "id" : "http://some.site.somewhere/entry-schema#", "schema" : "http://json-schema.org/draft-04/schema#", "description" : "schema for an fstab entry", "type" : "object", "required" : [ "storage" ], "properties" : { "storage" : { "type" : "object", "oneOf" : [ DBRef("#/definitions/diskDevice", 1), DBRef("#/definitions/diskUUID", 2),...

Import collection to mongodb server from gmaps library in python

python,mongodb,google-maps-api-3,pymongo,mongolab
I'm using python and i'm try to import a JSON like i'm recieving from this code: from gmaps import Geocoding api = Geocoding(api_key='<my key>') api.geocode("calle tigre 129 cusco") to mongodb server, like I created a collection with this code: from pymongo import MongoClient client = MongoClient('mongodb://<user>:<pass>@ds049219.mongolab.com:49219/<__>') db = client.<__> posts...

Counting elements inside an array in mongodb

python,mongodb,pymongo
I want to count the number of elements in an array using python and pymongo. Here is the data. { "_id": 5, "type": "Student", "Applicates": [ { "appId": 100, "School": "dfgdfgd", "Name": "tony", "URL": "www.url.com", "Time": "5/5/5-6/6/6", "Research": "dfgdfg", "Budge": 5000, "citizenship": "us", "Major": "csc", "preAwards": "None", "Advisor": "dfgdfg", "Evaluators":...

Can't upsert a record using update_one

python,mongodb,pymongo
I want to add a record to the collection if the key doesn't already exist. I understand MongoDB offers the upsertfor this so I did a db.collection.update({"_id":"key1"},{"_id":"key1"},True) This seems to work. However in the Python API it says that update is deprecated and use to update_one(). But: db.collection.update_one({"_id":"key1"},{"_id":"key1"},True) Gives: raise...

from pymongo.objectid import ObjectId ImportError: No module named objectid

python,mongodb,twitter,pymongo
I made an python code that fetch tweets from Mongo collection called Tweets. I wan't to fetch only the object text and add an additional object called Sentiment. When i loop through the Tweets and parse the json object to a string i get the error : from pymongo.objectid import...

pymongo query by datetime

python,python-2.7,pymongo
I got one problem which made me headache for almost one day. The python scripts are: # coding:utf-8 import pymongo import datetime, time def query_data(IP,datefrom,dateto): conn = pymongo.MongoClient(IP) db = conn.mymongo startdate = time.mktime(time.strptime(datefrom,'%Y-%m-%d')) enddate = time.mktime(time.strptime(dateto,'%Y-%m-%d')) # all above are correct definitely. # # but I got no result...

How to get all the keys of a Json document using pymongo?

python,json,pymongo
I have a document in the below format, { "glossary": { "title": "example glossary", "GlossDiv": { "title": "S", "GlossList": { "GlossEntry": { "ID": "SGML", "SortAs": "SGML", "GlossTerm": "Standard Generalized Markup Language", "Acronym": "SGML", "Abbrev": "ISO 8879:1986", "GlossDef": { "para": "A meta-markup language, used to create markup languages such as DocBook.",...

mongoengine user with readWrite can not create indexes

python,mongodb,authentication,pymongo,mongoengine
I'm running with python 2.7 and mongodb 2.6.5. I can't get my mongoengine to connect and make request but i can log on mongo with user and make the same request. I've created 3 users in mongodb like this (password are not blank but i've removed them there): $ mongo...

When to ensure_index usage in mongodb

mongodb,pymongo
I have a mongo db collection that is updated (a new record is added) once a second. Will it suffice to call ensure_index ONCE on the index attribute or do I have to call it repeatedly - for example after each document is added? I understand that ensure_index looks at...

How to display ImageGridFSProxy in HTML?

html,mongodb,flask,pymongo,gridfs
I have several images stored within MongoDB and I would like to display them now in a web page. I'm using Flask for my application. The record has a binary field named "payload" class BinaryFile(mongo.Document): created_at = mongo.DateTimeField(default=datetime.datetime.now, required=True) file_name = mongo.StringField(max_length=255, required=True) payload = mongo.ImageField(required=False) I want to diplay...

mongo - find items who at least match an array of values

python,mongodb,pymongo
database entries: {"item1": ["a", "b"], "item2":"etc"} {"item1": ["a", "b", "c"], "item2":"etc"} {"item1": ["a", "b", "c", "d", "e"], "item2":"etc"} I want to return all entries, where a, b and c are in the item1 array. It can have additional values. The {"item1": {"$in" :["a", "b", "c"]}} query returns if any of...

Multiple query in pymongo

python,mongodb,pymongo
I have mongoDB with items of following structure: { "some_field": { "date": 123, "text": "secondary text" }, "text": "primary text" } I want to write queries in pymongo that check for text field of top level and apply conditional query at the "date" field of second level. I'm getting filters...

PyMongo: Access fields returned from document within array resulting from find() query

python,mongodb,pymongo
I am executing a query like this on a MongoDB collection: cursor = collection.find({"activityArray":{"$elemMatch":{"sport":0}}},{"activityArray.sport" : 1, "activityArray\|here is result object .id":1, "endo" : 1}) |20166249 for result_object in cursor[0:1]: |here is result object print "here is result object" |20166249 |here is result object print result_object["endo"] |20166249 # print result_object["activityArray.sport"] #...

Issue with multiple records of the same subdocument returned for MongoDb and Python

python,mongodb,tornado,pymongo
I'm getting this strange issue when getting back the results of a object with subdocuments from a Mongo database using Python and pymongo. I have a document with a list of sub documents, e.g. User: { "_id": .... hats: [{"colour": "blue" }] } I query this using find_one(). It returns...

Collection object is not callable error with PyMongo

python,mongodb,pymongo
Following along the PyMongo tutorial and am getting an error when calling the insert_one method on a collection. In [1]: import pymongo In [2]: from pymongo import MongoClient In [3]: client = MongoClient() In [4]: db = client.new_db In [5]: db Out[5]: Database(MongoClient('localhost', 27017), u'new_db') In [6]: posts = db.posts...

Tailable cursor in pymongo seems to have stopped working

python,mongodb,pymongo
I have successfully been using tailable cursors in Pymongo for 2 years without a hitch, but all of a sudden, today, my identical code is throwing an "unexpected keyword" error: I upgraded to 3.0 Mongo a few weeks ago and it was still working fine, but perhaps a new pymongo...

How to use the same database in diferent computers

mongodb,pymongo
Me and a friend are developing a database, but we just decided to migrate from sql to mongo due to big irregularities within our data. However, when we were working with sqlite3 for python, all we had to do was to to put the .db generated at github and pretty...

Efficient way to aggregate in Mongodb

python,mongodb,pymongo,aggregation-framework
I have a collection { "name" : "foo" "clicked" : {"0":6723,"1": 1415,"2":1122} } { "name" : "bar" "clicked" : {"8":1423,"9": 1415,"10":1122} } { "name" : "xyz" "clicked" : {"22":6723,"23": 1415,"2":1234} } clicked is basically {"position of item-clicked in the list" : "id of the item"} The final output which I...

how to write a mongodb query in pymongo

python,mongodb,pymongo
The following query works in mongodb shell: db.user.count( {$and: [ {"agent_id":{$exists:true}}, {"is_agent":{$ne:true}} ] } ) When I try it in python I get a different answer. Here's the python code: import pymongo from pymongo import MongoClient def getCollection(cient,dbname,collection): """Return a colleciton based on client, db and collection""" data_base = getattr(client,...

Mongodb, osm street maps, unique users

mongodb,pymongo
I imported a json file into my database, and the db itself works. If use print db.points.find_one() my output looks like : {u'id': u'342902', u'_id': ObjectId('555af76a029d3b1b0ff9a4be'), u'type': u'node', u'pos': [48.9979746, 8.3719741], u'created': {u'changeset': u'7105928', u'version': u'4', u'uid': u'163673', u'timestamp': u'2011-01-27T18:05:54Z', u'user': u'Free_Jan'}} which is exactly what I want it to...

Speeding up $or query in pymongo

mongodb,nosql,mongodb-query,pymongo
I have a collection of 1.8 billion records stored in mongodb, where each record looks like this: { "_id" : ObjectId("54c1a013715faf2cc0047c77"), "service_type" : "JE", "receiver_id" : NumberLong("865438083645"), "time" : ISODate("2012-12-05T23:07:36Z"), "duration" : 24, "service_description" : "NQ", "receiver_cell_id" : null, "location_id" : "658_55525", "caller_id" : NumberLong("475035504705") } I need to get...

addToSet and set in same update using PyMongo and MongoDB

python,mongodb,dictionary,pymongo
In MongoDb is it possible to perform an update on everything except a field that has already been updated for example with addToSet? I'm storing information in a python dictionary which is updated by the user. I then want to use this dictionary to update the entry in MongoDb. One...

Fetch data from Mongodb using Pymongo and serve as a WS

mongodb,pymongo
I have some data in my mongodb and i want to pull selective data from it (using query) and then serve it over http as a webservice. Here is my code so far, from flask import Flask from pymongo import MongoClient conn = MongoClient('172.16.1.195', 27017) db = conn.Views_AllTables collection =...

Using $each with $push on all fields in python

python,mongodb,python-2.7,pymongo
I have a python dictionary representing updates to be made to a mongo document: { '1': [{'x': 21, 'y': 37}, {'x': 12, 'y': 41}], '2': [{'x': 17, 'y': 19}, {'x': 82, 'y': 31}], ... ... 'r2': [{'x': 21, 'y': 37}, {'x': 12, 'y': 41}] ... ... } Fields can be...

Why does PyMongo throw AutoReconnect?

python,mongodb,pymongo
While researching some strange issues with my Python web application (in particular, issues regarding MongoDB connectivity), I noticed something on the official PyMongo documentation page. My web application uses Flask, but this shouldn't influence the issue I'm facing. The PyMongo driver does connection pooling, but it also throws an exception...

How to convert a string to BSON?

python,mongodb,pymongo,bson
I have a string like this: document = '{ time : 14/09/19 16:00:00, global : { full:190 , defects: 7 , btp: 6 , total: 202 } , domains : [ { domain : "A" , full:7 , defects: 2 , btp: 0 , total: 9 }, { domain :...

How to find an element in one of several MongoDBs subdocuments

mongodb,pymongo
I have a mongodb collection that contains documents that contain a subdocument with the following form: 'Store': { 'cupboard1': { 'Cheese': 21, 'Humous': 25, 'Natchos': 10, 'Olives': 10, 'stockItems': 66}, 'cupboard2': { 'Cheese': 11, 'Humous': 9, 'Olives': 2, 'Sausage': 3, 'stockItems': 25}, 'whole': { 'Chris': 32, 'Olives': 11, 'Sausage': 3,...

Multiple field Indexing in pymongo

mongodb,indexing,pymongo
I am trying to apply indexing for my mongodb collection through pymongo. I am using db[collection_name].ensure_index([("field_name" , "text"),("unique", 1), ("dropDups" , 1)]) and it works. But now how can apply it for multiple fields? Something like this db[collection_name].ensure_index([("field_name1" , "text"),("field_name2", "text"),("field_name3", "text"),("unique", 1), ("dropDups" , 1)]) I know we can...

MongoDB string index that is not text?

mongodb,twitter,pymongo
How does MongoDB index a string that is not identified as text? For example, tweets have many fields that are text, and I create an index for any of them. In my application, I have created an index for when the tweet was written, who wrote it, and the text...

Best way to learn PyMongo [closed]

python,pymongo
I am working on Python and want to learn PyMongo. I have already gone through the video tutorial explianing basics of how to create, delete collections. I want to learn about indexing and advanced level. Where can I get tutorials on these

PyMongo, handling fields with time relative to the database time

python,mongodb,pymongo
Say I have a document field name "creation_time". I would like to have its value initialized with the database current time, instead using the client machine time with: datetime.datetime.utcnow() How can I achieve this with PyMongo? Moreover is there any way to do operations during the field initialization, for example:...

PyMongo sending “authenticate” for every query

mongodb,pymongo,mongoengine,python-rq
I am getting an authentication to MongoDb for every query I run using PyMongo MongoClient. This seems expensive / unnecessary: 2015-02-13T09:38:08.091-0800 [conn375243] authenticate db: { authenticate: 1, user: "", nonce: "xxx", key: "xxx" } 2015-02-13T09:38:08.876-0800 [conn375243] end connection xxx (15 connections now open) 2015-02-13T09:38:08.962-0800 [initandlisten] connection accepted from xxx:42554 #375244...

Dictionary style access returning None in MongoDB

mongodb,python-3.x,pymongo
I have been trying to access a database in mongodb and then modifying the particular fetched data into key-value pair for further issue. Although, my code works fine with other collections. But, It's giving me some errors with this. 1: Here is the snippet of my code which was giving...

How to properly write this query

mongodb,python-2.7,pymongo
Using pymongo v2.6, what it a proper way to write a find query with these conditions: select * from tasks where processed = 0 AND (process_lock_date is null or process_lock_date < now - 10 minutes) order by date_added asc limit 50 Basically need to write the query to select 50...

What is the correct use of $and operator in pymongo?

python,mongodb,pymongo
I have this structure: >>>test_3.find_one({"humsavar.Disease": {"$exists": True}}, {"humsavar":True, "_id":False}) {u'humsavar': [{u'Association': u'Polymorphism', u'Disease': u'-', u'Gene names': u'DTWD1', u'Mutate aa': u'Pro', u'Position aa': 9, u'Reference aa': u'Leu', u'Substitution': u'Leu9Pro', u'SwissVarID': u'VAR_036757', u'Uniprot': u'Q8N5C7', u'dbSNP': u'rs11539522'}, {u'Association': u'Polymorphism', u'Disease': u'Pyruvate dehydrogenase lipoic acid synthetase deficiency', u'Gene names': u'DTWD1', u'Mutate aa': u'Lys',...

How do I write a query to get values based on two dependent input variables in MongoDB?

json,node.js,mongodb,mongodb-query,pymongo
I have a mongoDB collection that contains JSON documents in the following format. This is just a sample not the full document. { "_id": ObjectId("555ba8a6ae96b63b98969192"), "toptags": { "@attr": { "artist": "Rihanna" }, "tag": [ { "count": "100", "name": "pop", "url": "http://www.last.fm/tag/pop" }, { "count": "89", "name": "rnb", "url": "http://www.last.fm/tag/rnb" },...

How to convert a pymongo.cursor.Cursor into a dict?

python,dictionary,cursor,pymongo,square
I am using pymongo to query for all items in a region (actually it is to query for all venues in a region on a map). I used db.command(SON()) before to search in a spherical region, which can return me a dictionary and in the dictionary there is a key...

Pymongo How to I extract the data

python,mongodb,pymongo
My (python)MongoDB has ISODate("2014-10-11T10:51:00.000Z") format: { "_id" : 2360836, "created_at" : ISODate("2014-10-11T10:51:00.000Z"), } I want to get data between 2014-09-01 to 2014-12-09. How can I do get the records from the MongoDB? I prefer use as the mysql....

can I have a mongodb collection as a class attribute in python

python,mongodb,pymongo
I have the following class: import sys import os import pymongo from pymongo import MongoClient class Collection(): client = MongoClient() def __init__(self, db, collection_name): self.db = db self.collection_name = collection_name def getCollection(self): data_base = getattr(self.client, self.db) collObject = getattr(data_base, self.collection_name) return collObject def getIdFromEmail(self, email): collection = self.getCollection() id =...

Inserting string as regular string in mongodb

pymongo
The pymongo documentation says that BSON strings are UTF-8 encoded so PyMongo must ensure that any strings it stores contain only valid UTF-8 data. Unicode strings (<type ‘unicode’>) are encoded UTF-8 first. The reason our example string is represented in the Python shell as u’Mike’ instead of ‘Mike’ is that...

Views in Django with arbitrary number of url parameters

python,django,mongodb,pymongo
I want to write universal View with Django, in this function i want to handle several situations: first when i have url like vkusers3/11122233/1/2/ and also i want it working when 2 or third arguments is missing in url, like: vkusers3/11122233/ or vkusers3/11122233/1/ I cannot find it tutorials how to...

Python: Handle pyMongo's ObjectId with Tornado's get_argument

jquery,python,mongodb,tornado,pymongo
I'm building a REST api with python, tornadoweb and MongoDB. I'm looking for a simple way to handle client-server document exchanges with the lowest operations and the simplest API as possible. I'm facing a problem to serialize and deserialize MongoDB's ObjectId to JSON to send/receive data to and from the...

MongoDB, Python and PyMongo: Document size too large with BSONObj size is invalid

python,mongodb,pymongo
I am getting this error when writing to Mongo: OperationalFailure caught 10334 {u'connectionId': 2365, u'code': 10334, u'ok': 1.0, u'err': u'BSONObj size: 17254820 (0xA4490701) is invalid. Size must be between 0 and 16793600(16MB) First element: 0: This is a normal document full of strings and integers, constructed in Python, yet it's...

MongoDB file path as unique index

python,mongodb,indexing,pymongo
How should I organize my collection for documents like this: { "path" : "\\192.168.77.1\user\1.wav", // unique text index "sex" : "male", "age" : 28 // some fields } I use this scheme in Python (pymongo): client = MongoClient(self.addr) db = self.client['some'] db.files.ensure_index([('path', TEXT)], unique=True) data = [ {"path": r'\\192.168.77.5\1.wav', "base":...

Null set returned by mongodb aggregate Cursor in PHP

php,mongodb,mongodb-query,pymongo
MongoDB aggregate query returning empty set .Below is the query i am using in a php script to retrieve data from mongoDB .Please let me know where i am going wrong. $result = $collection->aggregateCursor([[ '$match'=> [ 'date'=> [ '$gte'=>ISODate("2015-06-01T00:00:00Z"), '$lte'=>ISODate("2015-06-03T00:00:00Z")] ] ],[ '$group'=> [ '_id'=> '$date', 'count'=> [ '$sum'=>1 ]...

Slow MongoDB/pymongo query

python,mongodb,pymongo
I am submitting a pretty simple query to MongoDB (version 2.6) using the pymongo library for python: query = {"type": "prime"} logging.info("Querying the DB") docs = usaspending.get_records_from_db(query) logging.info("Done querying. Sorting the results") docs.sort("timestamp", pymongo.ASCENDING) logging.info("Done sorting the results, getting count") count = docs.count(True) logging.info("Done counting: %s records", count) pprint(docs[0]) raise...

mongodb result from commandline different then from pymongo

python,regex,mongodb,pymongo
I am trying to create a new collection based on a result from a find. From the mongodb(robomongo) commandline if I do this db.liCollection.find({current_companies : { $regex: /^DIKW/i }}) I get nice restult of 11 documents out of 2.6 million. Now if I try to use pymongo like this: from...