django,postgresql,partitioning , Postgresql Table Partitioning Django Project


Postgresql Table Partitioning Django Project

Question:

Tag: django,postgresql,partitioning

I have a Django 1.7 project that uses Postgres 9.3. I have a table that will have rather high volume. The table will have anywhere from 13million to 40million new rows a month.

I would like to know what the best way to incorporate Postgres table partitioning with Django?


Answer:

As long as you use inheritance, and then only connect the parent table to your Django model, the partitions should be entirely transparent to Django. That is, a SELECT on the parent table will cascade down to the partitions, unless the ONLY keyword is explicitly used (where applicable).

Note that partitioning does add complexity in terms of needing to implement a programmatic method of determining when new partitions need to be created, and then creating them -- or doing this manually at certain intervals. Depending on your exact data and business logic, it is quite likely you may also need to implement triggers and rules to determine which partition to, say, INSERT something into (since you wouldn't want to INSERT into the parent table). These, too, should be abstracted from Django, however.

I have found that, depending on the exact circumstances, this may need to be done with your main apps shutdown, lest the new partition creation cause a deadlock.

Also worth considering is whether you need true partitions which get created over time, or if an inheritance model of, say, tables foo and foo_archive would suffice, where foo_archive inherits from foo, and periodically something (e.g. a script) moves older data into foo_archive to keep foo smaller.


Related:


Django add an attribute class form in __init__.py


python,django,forms,django-forms
My form class from django import forms class Form(forms.Form): your_name = forms.CharField(label='Your name ', max_length=100) My app file __init__.py from django import forms from my_app.forms import Form from captcha.fields import ReCaptchaField def register(form): form.captcha=CaptchaField() register(Form) Code in __init__.py add an attribute captcha but it is not on the page. I...

Django test RequestFactory vs Client


django,unit-testing,django-views,django-rest-framework,django-testing
I am trying to decide whether I should use Django's Client or RequestFactory to test my views. I am creating my server using DjangoRESTFramework and it's really simple, so far: class SimpleModelList(generics.ListCreateAPIView): """ Retrieve list of all route_areas or create a new one. """ queryset = SimpleModel.objects.all() serializer_class = SimpleModelSerializer...

sorl-thumbnail: Creating a thumbnail inside a for loop in a template


django,sorl-thumbnail
I am trying to create a thumbnail with the size of 100x100 px from an image. I am using a for loop, and the for loops works and the image is rendered but for some reason when I am trying to create a thumbnail it doesn't work. Here is my...

Callable in model field not called upon adding new object through Django admin


django,django-models,django-admin
In https://docs.djangoproject.com/en/1.8/ref/models/fields/#django.db.models.Field.default we read that the callable of a default field value is called, every time a new object is created. In my model I have: when_first_upload = models.DateTimeField(default=datetime.datetime.now()) When in the admin page I create a new object I always get the same datetime, as if the callable is...

Django listview clone selected record


django
From an existing LISTVIEW is there a simple way to create a new record based on existing previos record? 1) select the record 2) call the CREATEVIEW 3) set as initail the data from point 1) would this be possible and which is best? A) Get data from 1) by...

Django: html without CSS and the right text


python,html,css,django,url
First of all, this website that I'm trying to build is my first, so take it easy. Thanks. Anyway, I have my home page, home.html, that extends from base.html, and joke.html, that also extends base.html. The home page works just fine, but not the joke page. Here are some parts...

django and python manage.py runserver execution error


python,django,manage.py
When I execute python manage.py runserver command for my django_test projects I get following error: System check identified no issues (0 silenced). June 14, 2015 - 20:43:03 Django version 1.8.2, using settings 'django_test.settings' Starting development server at http://127.0.0.1:8000/ Quit the server with CONTROL-C. Traceback (most recent call last): File "manage.py",...

Redirect if ActiveRecord::RecordNotUnique error exists


ruby-on-rails,postgresql,activerecord,error-handling
I have the next code to save: Transaction.create(:status => params[:st], :transaction_id => params[:tx], :purchased_at => Time.now).save! But how can I redirect to main root page if this ActiveRecord::RecordNotUnique error appears? Can I catch this error?...

How can I fill the table? (Python, Django)


python,django
I faced with a problem: I have 4 lists with data. And I need to fill the table with this data. It is the example of table: So, as you already understood, at first column must be data from a first list, at second - second list, ... I tried...

Save a hex-string to PostgreSQL column character varying


postgresql,hex
I want to save hex-string ('A2-5A-47-00-10-00-00-00') into a PostgeSQL database. My column is character varying(30). How can I do this?...

need help specifying potentially reserved words as strings in postgres query


postgresql
I've been using mysql for close to 10 years, but have recently jumped into a project that's exposed me to postgres for the first time. Most of the differences aren't a bit deal, but I have been running into some small issues along the way that are throwing me off....

Permission denied Setuptools


python,django,curl,setuptools
I'm trying install setuptools in my Mac, but when I run command curl https://bootstrap.pypa.io/ez_setup.py -o - | python show a message telling: Processing setuptools-17.1.1-py3.4.egg Removing /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/setuptools-17.1.1-py3.4.egg Copying setuptools-17.1.1-py3.4.egg to /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages Adding setuptools 17.1.1 to easy-install.pth file error: [Errno 13] Permission denied:...

Upload to absolute path in Django


django
I am trying to get upload_to of FileField to be an absolute path rather than relative to MEDIA_ROOT. If I make the path absolute I get a 400 error on post of file. If no leading / it stores under MEDIA_ROOT. The uploaded document needs to be held securely and...

Error while trying to insert data using plpgsql


postgresql,timestamp,plpgsql
I have the following plpgsql function: CREATE OR REPLACE FUNCTION test_func(OUT pid bigint) RETURNS bigint AS $BODY$ DECLARE current_time timestamp with time zone = now(); BEGIN INSERT INTO "TEST"( created) VALUES (current_time) RETURNING id INTO pid; END $BODY$ LANGUAGE plpgsql; select * from test_func(); The above gives an error: column...

Is there a way to install django with pip to point to a specific version of python in virtualenv


python,django,pip,virtualenv
I have a system with CentOS installed. It currently runs python2.6, but python2.7 is also installed. I want to run django 1.7, which is also currently installed. If I run django outside of a virtualenv, it is using python2.6 by default. I didn't install it myself. What I assume is...

Return integer value of age(date) function in Postgres


postgresql
I would like to determine the number of days that an account has been open. Ideally, I would like to compare the return value to an integer(days). i.e I would like to see if age(open_date) > 14 If anyone has any better ideas... Thanks...

How to create a SELECT query FROM “TABLE1 AND TABLE2”


sql,postgresql,select,join
I have a PostgreSQL database, with only SELECT permissions. In this DB there are two tables with the same structure (the same columns). I need to write several query in each table and join the results. There is a way for writing a query like this one? SELECT field1, field2,...

Is there a better way to write this query involving a self select?


sql,postgresql,join,aggregate-functions
I am looking for a "better" way to perform a query in which I want to show a single player who he has played previously and the associated win-loss record for each such opponent. Here are the tables involved stripped down to essentials: create table player (player_id int, username text);...

postgresql complex group by in query


sql,postgresql
I have a query that gives: itemid deadlineneeded delievrydate quantity 200 15/07/15 14/07/15 5 200 15/07/15 14/07/15 10 200 15/07/15 13/07/15 25 201 15/07/15 14/07/15 30 200 14/07/15 10/07/15 3 201 15/07/15 15/07/15 100 It gives the information from multiple tables. Basically it means When items arrive to warehouse (delievrydate)...

How to use template within Django template?


python,html,django,templates,django-1.4
I have the django template like below: <a href="https://example.com/url{{ mylist.0.id }}" target="_blank"><h1 class="title">{{ mylist.0.title }}</h1></a> <p> {{ mylist.0.text|truncatewords:50 }}<br> ... (the actual template is quite big) It should be used 10 times on the same page, but 'external' html elements are different: <div class="row"> <div class="col-md-12 col-lg-12 block block-color-1"> *django...

DjangoCMS 3 Filter Available Plugins


django,django-cms
In the DjangoCMS3 documentation it says you can configure DjangoCMS behavior using CMS_PLACEHOLDER_CONF in your settings. For instance: CMS_PLACEHOLDER_CONF = { 'right-column': { 'plugins': ['TextPlugin', 'PicturePlugin'], ... This would make TextPlugin and PicturePlugin to be the only two plugins available inside any placeholder called "right-column". It works, but what if...

Django MySQLClient pip compile failure on Linux


python,linux,django,gcc,pip
Django documentation as of v1.8 recommends using mysqlclient connector for the framework. I'm attempting to pip install the package on Ubuntu 14.04 with Python 3.4 and running into a GCC error that I'm unable to find reference to. I'm not an expert on compiling software, so was hoping somebody can...

Use django to expose python functions on the web


python,django
I have not worked with Django seriously and my only experience is the tutorials on their site. I am trying to write my own application now, and what I want is to have some sort of API. My idea is that I will later be able to use it with...

django 1.8 CreateView Error


python,django
I'm using django 1.8 to create a login form. But the template containing registration form does not render properly. my views.py class SignUpView(CreateView): RegistrationForm = UserCreationForm fields = ['username','password1','password2'] model = User template_name = 'accounts/signup.html' my forms.py from django.contrib.auth.forms import UserCreationForm from django.contrib.auth.forms import AuthenticationForm from crispy_forms.helper import FormHelper from...

PostgreSQL conditional statement


sql,postgresql,if-statement
I need to implement a conditional SELECT statement based on the result of another statement. I don't want to create a function for this purpose, but simply using conditional SELECT. My unsuccessful approach looks as below: DO $do$ BEGIN IF SELECT count(*) FROM table1 < 1 THEN SELECT * FROM...

PostgreSQL: trigger to call function with parameters


postgresql
In PostgreSQL, I'm trying to create a trigger that passes an argument to a function, but it appears that this operation looks for a function whose signature has zero arguments: ERROR: function create_rec_if_needed() does not exist: CREATE TRIGGER after_update_winks AFTER UPDATE ON winks FOR EACH ROW WHEN (NEW.counter > 3)...

How can i hook into django migrations for django 1.8


python,django,migration
I am using django 1.8. Now i need to add some custom fields based permisions so i have created a YML file from python models like this description: permissions: ['ADMIN'] award: permissions: ['USER'] Its working fine but my issue if some chnages the field names or reomves some field then...

How to order SQL query result on condition?


sql,postgresql,order,condition
i have the this SQL query: SELECT DISTINCT category FROM merchant ORDER BY category ASC that gives this output: accommodation education food general health money shopping sport transport How to put the row that contains "general" at the start (or the end) of the result?...

How can I resolve my variable's unexpected output?


django,python-2.7
I have a variable in django named optional_message. If I debug the variable then it says Swenskt but when I try to print the variable on my page the following comes out: (u'Swenskt',) and the variable can't be tested for its length etc. What should I do if I only...

Retrieve updated rows in AFTER UPDATE trigger Postgresql


postgresql,triggers,plpgsql
I'm writing a AFTER UPDATE trigger in postgresql. Actualy I need to get at least one row after update in STATEMENT LEVEL trigger, but there is no OLD or NEW variable there. In FOR EACH ROW trigger I didn't manage to find a sort of batch_last param. But to do...

upload_to dynamically generated url to callable


python,django,models
I've seen a lot of post about this problem without really understanding how to solve it. I have this model: class Project(TimeStampedModel): name = models.TextField(max_length=100, default='no name') logo = models.ImageField() I'd like to have my image saved to media root following this template: <name>/logo/<filename> At first glance, I would like...

Prepared statements: Using unnamed and unnumbered question mark style positional placeholders


postgresql
In SQL systems other than Postgres, such as MySQL for instance, prepared statements can use question marks ? as a placeholder for data in prepared statements. INSERT INTO foo (id, name) VALUES (?, ?), (?, ?); However, in Postgres the only available placeholders seem to be the numbered placeholders, so...

Why is Django widgets for TimeInput not showing


django,django-forms,django-templates,django-views
I'm trying to create a TimeInput field in a form and noticed that the widget isn't showing correctly. But when I check the localhost:8000/admin, I see the widget showing up correctly. My code is as follows. For models.py, class TimeLimit(models.Model): before = models.TimeField(blank=True, default=time(7, 0)) # 7AM after = models.TimeField(blank=True,...

Create angular page in Django to consume data from JSON


angularjs,django,django-templates
In an angular controller I have a list of items: app.controller('MainController', ['$scope', function($scope) { $scope.items = [ {"foo":"bar 1"}, {"foo":"bar 2"}, {"foo":"bar n"} ] }]); The following html page, based on angular, displays a list of item: <!DOCTYPE html> <html> <head> <title>list</title> <script src= "http://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js"></script> </head> <body data-ng-app="myApp"> <div data-ng-controller="MainController">...

How to check what constraint has been violated?


java,sql,postgresql,exception
I'm used to check if the user try to store a valid data with cathcing the ConsratinViolationException, like that: try { //persisitng to a db } catch (ConstraintViolationException e){ //Print message } I'm using PostgreSQL and now I'm under the isssue that the persisitng can violate more than one different...

Using .update with nested Serializer to post Image


django,rest,django-models,django-rest-framework,imagefield
I have an ImageField. When I update it with the .update command, it does not properly save. It validates, returns a successful save, and says it is good. However, the image is never saved (I don't see it in my /media like I do my other pictures), and when it...

Postgresql Update JDBC


java,postgresql,jdbc
I want to update table: id integer NOT NULL, "first" character varying(255), "last" character varying(255), age integer, CONSTRAINT registration_pkey PRIMARY KEY (id) using method: void updateTable(String tableName, String columnName, String value, String columnName2, String value2) { try { String sql = "UPDATE " + tableName + " SET " +...

Django file upload took forever with Phusion Passenger


django,apache,file-upload,passenger
I have a Django app deployed using Passenger (I did not choose mod_wsgi because mod_passenger is already there and being used). When I uploaded an MP3 file (900 kB), Google Chrome displays upload % which reached 100% pretty fast but then it took forever for the resulting page. The database...

Translation of interval


postgresql,datetime,translation,intervals,postgresql-8.4
There is maybe a way to translate automatically interval as it's shown by postgresql (e.g. "1330 days 10:54:54.266684") to other languages? For instance using locales, or other settings or I have to use replace, e.g. regexp_replace((now()-t.another_date)::text, 'day','dzień')?

DRF Update Existing Objects


django,django-rest-framework
I'm new to DRF and python so go easy on me... I can successfully get the RoadSegment objects, but I cannot figure out how to update an existing object I have the following model: class RoadSegment(models.Model): location = models.CharField(max_length=100) entryline = models.CharField(unique=True, max_length=100) trafficstate = models.CharField(max_length=100) With the following serializer:...

Django: Handling several page parameters


python,django,list,parameters,httprequest
I have several possible parameter to process in a page. Assume x0, x1, x2,..., x1000. It seems awkward to get and process them one by one by request.GET.get('x0'), request.GET.get('x1'), ... Any idea to put them in a list, so that they can be processed in a loop....

Django: show newlines from admin site?


python,html,django,newline,textfield
This is probably very simple. My database objects have a TextField. Now, when I add the contents of the TextFields to an html paragraph, there are no new lines. How can I make Django show those newlines? Thank you!

Show message when there's no excerpt - Django templates


python,django,django-templates,django-template-filters
I have this field on a Django template: <p class="border_dotted_bottom"> {{ expert.description|slice:":300" }} <a href="{% url 'profile' expert.username %}">{% trans "read more" %}</a>.... </p> If this object (user) has no 'decription' (a text field) it shows the word 'None', I need to get rid of that, maybe if he doesn't...

Display django runserver output from Vagrant guest VM in host Mac notifications?


python,django,osx,notifications,vagrant
I can think of many ways to skin this cat, and Googling hasn't shown an elegant solution either. Does anyone have an easy / elegant way to forward the output from Django's runserver command in a terminal connected to a guest Vagrant VM to the host to display to Mac's...

Django block inclusion not working. What did I miss?


django,django-templates
In my base.html file I have this: {% block menu %}{% endblock menu %} and in base_menu.html I have this: {% extends "base.html" %} {% block menu %} <nav class="navbar navbar-inverse navbar-fixed-top"> ... stuff ... </div> {% endblock menu %} I would expect the menu html to show where the...

Django REST tutorial DEBUG=FALSE error


python,django,django-rest-framework
I'm trying to learn to use the django REST framework via the tutorial. I've gotten to the section "Testing our first attempt at a Web API". When I start the server, I get: System check identified no issues (0 silenced). June 15, 2015 - 00:34:49 Django version 1.8, using settings...

How to work with django-rest-framework in the templates


json,django,django-templates,django-rest-framework
Please bear with me. I am just learning django-rest-framework. And I really can't seem to grab it. model: class Day(models.Model): date = models.DateField(default=date.today) class ToDo(models.Model): date = models.ForeignKey(Day) name = models.CharField(max_length=100) very_important = models.BooleanField(default=False) finished = models.BooleanField(default=False) normal view: def home(request): days = Day.objects.all() return render(request, 'test.html', { 'days':days })...

How to get second row in PostgreSQL?


sql,postgresql
I have next table: COLUMN_NAME DATA_TYPE PK NULLABLE inc_id bigint YES NO dt_cr timestamp NO NO email varchar(255) NO YES email column is not unique, so I should use GROUP BY email. Question: How can I get from this table minimal date, next row after minimal date and email related...

Django template not found in main project directory error


python,django,templates
I am getting a 'template not found' error, although I've set up a correct template hierarchy (or so I thought) . ├── manage.py ├── twinja │   ├── admin.py │   ├── admin.pyc │   ├── __init__.py │   ├── __init__.pyc │   ├── migrations │   │   ├── __init__.py │   │   └── __init__.pyc │   ├──...

In PostgreSQL is it possible to join between table and function?


sql,postgresql
I have a table named Cars and a function named PartsPerCar there is no direct link between carid and partid. the link is though 3 tables in the middle. I do have a function called PartsPerCar which gets carid (it goves via the tables in the middle) and give back...